Skip to content

[SWIFT-TESTING] Feature: Add test verification lab#3

Open
LeeDayday wants to merge 2 commits into
mainfrom
swift-testing/sammy/verification
Open

[SWIFT-TESTING] Feature: Add test verification lab#3
LeeDayday wants to merge 2 commits into
mainfrom
swift-testing/sammy/verification

Conversation

@LeeDayday

Copy link
Copy Markdown

Summary

This PR adds VerificationTestingLab, a Swift Package focused on verifying whether tests actually catch bugs when using Swift Testing.

The lab explores common sources of false confidence in tests, including weak assertions, false positives, flaky randomness, shared mutable state, insufficient coverage, and mutation validation.

What Changed

  • Added a new Swift Package under VerificationTestingLab/testing-lab-verification
  • Implemented small production examples:
    • ThreeSixNineGame
    • LoginValidator
    • RandomIDGenerator
    • SharedCounter
  • Added Swift Testing suites for six experiments:
    • Weak assertions
    • False positives
    • Flaky tests
    • Shared mutable state
    • Parameterized tests
    • Test validation checklist
  • Added disabled tests that demonstrate intentionally unreliable or order-dependent test cases without breaking the normal verification run
  • Added documentation:
    • VerificationTestingLab/README.md
    • Docs/verification-tests.md
  • Added a package-level .gitignore to exclude SwiftPM/Xcode build artifacts

Methodology

The lab is organized around test verification risks rather than file structure.

Each experiment asks:

  • What can make a test misleading?
  • Would this test fail if production code were intentionally broken?
  • How can the test be improved to verify behavior more directly?

The documentation includes a Mermaid diagram that maps production subjects to verification risks and stronger testing strategies.

Verification

Ran the Swift Package test suite:

cd VerificationTestingLab/testing-lab-verification
swift test

Result:

19 tests in 6 suites passed
3 disabled failure examples skipped

The disabled tests are intentional examples for flaky and order-dependent behavior.

Notes for Reviewers

  • .build, .swiftpm, and .DS_Store files are intentionally excluded from the package commit.
  • The package is not an iOS app target. It is a Swift Package designed for focused Swift Testing experiments.
  • Failure examples are represented as disabled tests so reviewers can inspect or re-enable them without making the default test run unstable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant