Generated projects flaky tests

Flaky tests are tests that produce different results (pass or fail) when run multiple times with the same code. They erode trust in your test suite and waste developer time investigating false failures. Tuist automatically detects flaky tests and helps you track them over time.

How flaky detection works#

Tuist detects flaky tests in two ways:

Test retries#

When you run tests with retry functionality, Tuist analyzes the results of each attempt. If a test fails on some attempts but passes on others, it's marked as flaky.

Pass -retry-tests-on-failure or -test-iterations through tuist test:

Cross-run detection#

Even without test retries, Tuist can detect flaky tests by comparing results across different CI runs on the same commit. If a test passes in one CI run but fails in another run for the same commit, both runs are marked as flaky.

This is particularly useful for catching flaky tests that don't fail consistently enough to be caught by retries, but still cause intermittent CI failures.

Managing flaky tests#

Automatic clearing#

Tuist automatically clears the flaky flag from tests that haven't been flaky for 14 days. This ensures that tests that have been fixed don't remain marked as flaky indefinitely.

Manual management#

You can also manually mark or unmark tests as flaky from the test case detail page. This is useful when:

• You want to acknowledge a known flaky test while working on a fix
• A test was incorrectly flagged due to infrastructure issues

Quarantining flaky tests#

Quarantining isolates a flaky test so it doesn't block CI while you fix it. A quarantined test is in one of two modes:

• Muted: the test still runs, but tuist test masks the failure. Failures still feed the flaky-tests detector, so you can keep watching the test without breaking the build. Pick this for a test you're actively investigating.
• Skipped: xcodebuild receives -skip-testing <identifier>, so the test never starts. It produces no new results and drops off the flaky-tests dashboard until you re-enable it. Pick this when the test is broken, slow, or so persistently flaky that running it is just wasted CI minutes.

Running tests#

tuist test honours both modes automatically:

Skipped tests are passed to xcodebuild as -skip-testing and never start. Muted tests run normally; if they fail, the failure is masked in the resulting build status.

To bypass quarantine entirely and run everything, including muted and skipped tests:

Slack notifications#

Get notified instantly when a test becomes flaky by setting up flaky test alerts in your Slack integration.