Elixir test sharding

The Tuist Hex package includes built-in support for sharding the tests of an Elixir project. It uses the Tuist server to create balanced shard plans based on historical timing data, and it compiles your project once so every runner can go straight to running tests.

How it works#

Test sharding follows a two-phase workflow:

1. Build phase: Tuist compiles your project for testing, enumerates your test files, and creates a shard plan on the server. The server uses historical test timing data from the last 30 days to distribute tests across shards so each shard takes roughly the same amount of time. The build phase uploads the build and outputs a shard matrix that your CI system uses to spawn parallel runners.
2. Test phase: Each CI runner receives a shard index, downloads the build, and executes only the tests assigned to that shard.

Tests are distributed by file: every test in a file runs on the same shard. Tests that Tuist has not seen yet get an estimate, so the first plans of a project are less balanced than later ones.

Build phase#

Prepare test shards using the tuist.test.build task:

Compile through the alias installed by the shared setup before invoking the sharding task. On a cold checkout, Mix can compile dependencies while discovering tuist.test.build, before Tuist's dependency cache has a chance to restore them. The explicit compile runs in the same test environment as the sharding task.

This task:

1. Compiles the project in the test environment
2. Creates a shard plan on the Tuist server using historical timing data
3. Uploads the build so the shards don't compile again
4. Outputs a shard matrix for your CI system

Build options#

Every other argument is forwarded to mix compile.

The shard reference is automatically derived from CI environment variables (GITHUB_RUN_ID, CI_PIPELINE_ID, CIRCLE_WORKFLOW_ID, BUILDKITE_BUILD_ID) or can be set explicitly via the TUIST_SHARD_REFERENCE environment variable. Outside those providers you have to set it, to a value that the build phase and the shards of one pipeline run share. On GitHub Actions, references also include the run attempt: when rerunning only failed jobs, pass the original build job's reference rather than deriving a new one in the shard jobs.

Test phase#

When TUIST_SHARD_INDEX is set, the package fetches the shard assignment from the server, downloads the build, and runs only the assigned test files without compiling. On a cold runner, bootstrap the SDK and load its paths before invoking the task:

Compiling the SDK in Mix's worker processes does not load it in the parent process. The explicit deps.loadpaths step prevents task discovery from compiling the remaining dependencies before downloading the build. This single-command form preserves your project's test preparation alias. If its setup commands trigger compilation, use the separate setup sequence below.

Any test files or directories you pass narrow the shard further: only the files that are both in the shard and in your selection run. A shard that ends up with no tests exits successfully.

The results of all the shards arrive on the dashboard as a single test run, and its Shards section shows how long each one took.

If setup must run before the tests, download the build first without starting the application:

Run your setup in a separate process with compilation disabled. For example, an Ecto project can create and migrate its databases using:

Keep any other project-specific preparation in this phase. Your compile alias must also honor --no-compile before doing bootstrap work: Mix only forwards flags to the last task of a list alias, so an earlier unconditional deps.compile still runs. See the guarded alias in manual compile integration.

Then load the restored SDK without compilation before executing the shard. This also prevents task discovery from compiling a path dependency when its sources are newer than the downloaded build:

Both commands need the same shard reference. --prepare-only requires an uploaded build, and --no-download fails if the prepared application artifact is missing. --no-download explicitly skips your project's test preparation alias, including on retries, and prints a notice when it skips setup steps. Ordinary runs keep their aliases, even when you forward --no-compile --no-deps-check. Keeping database migrations in a separate process also avoids module-redefinition warnings when migration regression tests load those modules themselves.

When testing different environments, pass a stable --scheme label, such as --scheme clickhouse-current or --scheme clickhouse-floor. Results with different labels are not treated as flaky reruns of the same configuration.

What the shards need#

The uploaded build is the _build/test directory. Each shard runner still needs:

• The same checkout as the build phase
• The same Elixir and Erlang/OTP versions
• The dependency sources, from mix deps.get

Sharding from the root of an umbrella project is not supported yet. Run both phases inside one of its applications.

Continuous integration#

GitHub Actions#

On GitHub Actions the build phase writes the shard indexes to the matrix output of its step. Use a matrix strategy to run shards in parallel:

Other providers#

Outside GitHub Actions the build phase writes the plan to .tuist-shard-matrix.json:

Use shard_count to set up your parallel jobs, and give each job its TUIST_SHARD_INDEX, from 0 to shard_count - 1. On a provider that Tuist does not derive the reference from, also give every job the same TUIST_SHARD_REFERENCE.