Bazel flaky tests

Flaky tests are tests that produce different results (pass or fail) when run multiple times with the same code. They erode trust in your test suite and waste developer time investigating false failures. Tuist automatically detects flaky tests and helps you track them over time.

How flaky detection works#

Tuist uses the individual cases from Bazel's test.xml reports to detect inconsistent results across retries and across runs of the same commit and target selection in continuous integration. Local runs remain visible but do not automatically flag test cases as flaky.

Test retries#

Retries are optional and independent of quarantine. --flaky_test_attempts=3 enables up to three total Bazel attempts per test, not three additional retries. Keep your existing retry configuration in .bazelrc or pass it explicitly:

Retries must actually run for Tuist to observe their results. A Bazel target's flaky summary alone does not identify which individual case was flaky.

Cross-run detection#

Comparing runs needs commit and branch metadata. Provide it when your build environment does not already publish it:

Cases are identified by target label, class name (falling back to the suite name), and test name, as described in Bazel test insights.

Managing flaky tests#

Automatic clearing#

Detection and clearing run through an automation alert on the project. Every project gets a default "Flaky test detection" automation whose trigger marks a test as flaky and whose recovery clears the flag once the test has gone the configured recovery window (default 14 days) without re-triggering. Edit it under Settings → Automations to change the recovery window, change the recovery actions (for example, also release the test from quarantine), or disable recovery entirely so tests stay marked flaky until you clear them by hand.

Manual management#

You can also manually mark or unmark tests as flaky from the test case detail page. This is useful when:

• You want to acknowledge a known flaky test while working on a fix
• A test was incorrectly flagged due to infrastructure issues

Quarantining flaky tests#

Quarantining isolates a flaky test so it doesn't block CI while you fix it. By default it's a manual action: you quarantine a test, release it, and switch modes from the test case detail page in the dashboard. You can also wire it into an automation alert under Settings → Automations so a test is automatically muted (or skipped) when it crosses a flakiness threshold, and released when it recovers. Every transition, manual or automated, is recorded on the test case's audit log.

By default, recovery waits until the trigger condition no longer holds, so a test stays quarantined until its old flaky runs leave the trigger window. To recover sooner, set Recover when to No new flaky runs: the test recovers once the recovery window (days or runs) passes without a new flaky run, every new flaky run restarts that wait, and after recovering only a new flaky run quarantines it again.

tuist bazel test fetches the quarantine policy before each invocation and applies it to the underlying bazel test run. Ordinary bazel test still reports test results, but does not apply quarantine:

Arguments after -- are forwarded to bazel test. Use --path to select the project and working directory, and --bazel to select a different executable such as bazelisk. A named Bazel configuration such as --config=ci is optional and works the same way.

Mute#

Use Mute as the primary quarantine workflow. Muted cases continue to run, their failures remain visible, and healthy sibling cases still affect the result. Tuist turns Bazel's test-failure exit into success only when the reports attribute every failure in each failed target to a muted case. Build errors, timeouts, interrupted runs, missing or incomplete reports, and failures outside the muted set keep Bazel's failure status.

Skip#

Skipping works per Bazel target: skipping one case excludes its whole target, including healthy cases, because Bazel can't filter individual cases for every test framework. Prefer Mute unless you intend to exclude the entire target. The command prints every excluded target label so build logs show what the quarantine policy left out.

If a skipped target no longer exists, Tuist names it in a warning and runs without that exclusion. Return stale cases to Enabled to remove their policies permanently. If no test targets remain, Bazel keeps its normal "no tests found" exit status.

Why quarantined tests can appear as passing#

• Muted tests still execute. A muted test that fails is recorded as failed on the test case run and flagged as flaky; the per-test status is not rewritten. What gets overridden is the overall test run: if every failing test case in the run is muted, the run as a whole is reported as passed, so muted failures don't break CI.
• Skipped tests don't run at all, so the dashboard keeps showing the status from the test's last actual execution. That snapshot can be weeks old.

Running quarantined tests again#

To investigate or validate a fix, bypass the quarantine policy:

Return the case to Enabled to include its target in subsequent normal runs. Skipped cases produce no new executions until explicitly included, so an automation based on successful new runs needs a validation run before it can recover them.

Limitations#

• An explicit --build_event_json_file or --target_pattern_file cannot be combined with active quarantine. Pass target patterns directly.
• If Tuist cannot fetch the quarantine policy, the command stops before running tests. It never applies a partial policy. Use --no-quarantine when you intentionally want to run without it.
• A failure that matches a mute recorded under an older suite-based identity stays blocking, and the command names the policy to reapply.

See Bazel's test filters, command options, and build-event descriptions for the underlying behavior.

Slack notifications#

Get notified when a test becomes flaky by setting up flaky test alerts in your Slack integration.

Querying flaky and quarantined state#

CLI#

The tuist test case command tree exposes everything Tuist tracks about a test case:

All of these accept --json for scripting.

REST API#

The same data is available over HTTP. See the Test Cases endpoints in the API reference for the full list of routes, filters, and response fields. To change a test case's state (mark or unmark flaky, mute, skip), send PATCH /api/projects/{account_handle}/{project_handle}/tests/test-cases/{test_case_id} with is_flaky or state. The same change is available as tuist test case update and as the update_test_case MCP tool.