Skip to content

Rust flaky tests

Tuist can detect inconsistent Cargo test results and help you investigate them in Tests → Flaky Tests.

Requirements

How flaky detection works#

Tuist compares CI runs on the same commit. A test that passes in one run and fails in another can be marked as flaky.

Report each run through the usual command:

bash
tuist cargo test --workspace

Cargo does not automatically retry failed tests. Re-running a CI job on the same commit provides another result for cross-run detection.

Managing flaky tests#

Automatic clearing#

Detection and clearing run through an automation alert on the project. Every project gets a default "Flaky test detection" automation whose trigger marks a test as flaky and whose recovery clears the flag once the test has gone the configured recovery window (default 14 days) without re-triggering. Edit it under Settings → Automations to change the recovery window, change the recovery actions (for example, also release the test from quarantine), or disable recovery entirely so tests stay marked flaky until you clear them by hand.

Manual management#

You can also manually mark or unmark tests as flaky from the test case detail page. This is useful when:

  • You want to acknowledge a known flaky test while working on a fix
  • A test was incorrectly flagged due to infrastructure issues

Quarantining flaky tests#

Mute or skip a test from its detail page, or configure quarantine in Settings → Automations. The next tuist cargo test fetches the project's current quarantine state:

  • Muted tests still run and report their actual results, but their failures do not fail the command. Other test failures, compilation errors, crashes, and incomplete results still fail normally.
  • Skipped tests are excluded from execution and reported as quarantined skips. Matching includes the package, target, Rust module path, and test name, so similarly named tests remain independent.

Muting preserves the selected workspace packages and features, and runs all selected targets even after a muted target fails. Skip quarantine requires one selected package: use tuist cargo test -p store rather than a multi-package workspace command. Tuist rejects multi-package skipping because narrowing the command can change Cargo's unified dependency features.

Quarantine also applies to sharded Cargo runs. If Tuist cannot fetch the state, it warns and runs without quarantine; failures are not suppressed. To deliberately run without quarantine:

bash
TUIST_TEST_SKIP_QUARANTINE=1 tuist cargo test --workspace

Limitations#

  • Enforcement requires standard libtest harnesses and captured default output. Commands with quiet, verbose, alternate-format, or uncaptured output fail when quarantine needs to be applied.
  • Select an exact workspace package with -p <package> if target attribution is ambiguous. Custom harnesses, package/target-name globs, and unresolved package selections are unsupported.
  • Doctests can be muted or skipped too, but Rustdoc's test filtering cannot distinguish every possible pair of example names. If skipping an example would also skip an enabled example, or a partial doctest exclusion uses --exact, Tuist fails with a diagnostic rather than executing the skipped example or silently omitting enabled tests. Use the override above while investigating.

Stress-testing new tests#

Tuist does not stress-test new Cargo tests. Use a Cargo test filter to repeat a failing test while investigating it:

bash
tuist cargo test -p store rejects_invalid_token

Slack notifications#

Get notified when a test becomes flaky by setting up flaky test alerts in your Slack integration.

Querying flaky and quarantined state#

CLI#

The tuist test case command tree exposes everything Tuist tracks about a test case:

bash
tuist test case list --flaky # only flaky test cases
tuist test case list --quarantined # only muted or skipped test cases
tuist test case show <test_case_id> # detail: flakiness rate, last status, run counts
tuist test case events <test_case_id> # audit log: marked_flaky, muted, skipped, ...
tuist test case run list <test_case_id> # run history with status and duration
tuist test case run show <test_case_run_id> # single run, including failure breakdown

All of these accept --json for scripting.

REST API#

The same data is available over HTTP. See the Test Cases endpoints in the API reference for the full list of routes, filters, and response fields. To change a test case's state (mark or unmark flaky, mute, skip), send PATCH /api/projects/{account_handle}/{project_handle}/tests/test-cases/{test_case_id} with is_flaky or state. The same change is available as tuist test case update and as the update_test_case MCP tool.