Elixir test sharding

The Tuist Hex package includes built-in support for sharding the tests of an Elixir project. It uses the Tuist server to create balanced shard plans based on historical timing data, and it compiles your project once so every runner can go straight to running tests.

REQUIREMENTS

Test insights must be configured

How it works#

Test sharding follows a two-phase workflow:

Build phase: Tuist compiles your project for testing, enumerates your test files, and creates a shard plan on the server. The server uses historical test timing data from the last 30 days to distribute tests across shards so each shard takes roughly the same amount of time. The build phase uploads the build and outputs a shard matrix that your CI system uses to spawn parallel runners.
Test phase: Each CI runner receives a shard index, downloads the build, and executes only the tests assigned to that shard.

Tests are distributed by file: every test in a file runs on the same shard. Tests that Tuist has not seen yet get an estimate, so the first plans of a project are less balanced than later ones.

Build phase#

Prepare test shards using the tuist.test.build task:

bash

mix tuist.test.build --shard-max 5

This task:

Compiles the project in the test environment
Creates a shard plan on the Tuist server using historical timing data
Uploads the build so the shards don't compile again
Outputs a shard matrix for your CI system

Build options#

Option	Description
`--shard-max <N>`	Maximum number of shards (default: 2)
`--shard-min <N>`	Minimum number of shards
`--shard-total <N>`	Exact number of shards, instead of a range
`--shard-max-duration <MS>`	Target maximum duration per shard in milliseconds
`--shard-reference <REF>`	The name the shards find the plan by
`--no-upload`	Plan the shards without uploading the build; each shard then compiles for itself

Every other argument is forwarded to mix compile.

The shard reference is automatically derived from CI environment variables (GITHUB_RUN_ID, CI_PIPELINE_ID, CIRCLE_WORKFLOW_ID, BUILDKITE_BUILD_ID) or can be set explicitly via the TUIST_SHARD_REFERENCE environment variable. Outside those providers you have to set it, to a value that the build phase and the shards of one pipeline run share.

Test phase#

Each shard runner executes its assigned tests using mix tuist.test, or mix test if you aliased it. When TUIST_SHARD_INDEX is set, the package fetches the shard assignment from the server, downloads the build, and runs only the assigned test files without compiling.

bash

TUIST_SHARD_INDEX=0 mix test

Any test files or directories you pass narrow the shard further: only the files that are both in the shard and in your selection run. A shard that ends up with no tests exits successfully.

The results of all the shards arrive on the dashboard as a single test run, and its Shards section shows how long each one took.

What the shards need#

The uploaded build is the _build/test directory. Each shard runner still needs:

The same checkout as the build phase
The same Elixir and Erlang/OTP versions
The dependency sources, from mix deps.get

Sharding from the root of an umbrella project is not supported yet. Run both phases inside one of its applications.

Continuous integration#

GitHub Actions#

On GitHub Actions the build phase writes the shard indexes to the matrix output of its step. Use a matrix strategy to run shards in parallel:

yaml

name: Tests
on: [pull_request]

env:
  MIX_ENV: test
  TUIST_TOKEN: ${{ secrets.TUIST_TOKEN }}

jobs:
  build:
    name: Build test shards
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.build.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.18'
          otp-version: '27'
      - run: mix deps.get
      - id: build
        run: mix tuist.test.build --shard-max 5

  test:
    name: "Shard #${{ matrix.shard }}"
    needs: build
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: ${{ fromJson(needs.build.outputs.matrix).shard }}
    env:
      TUIST_SHARD_INDEX: ${{ matrix.shard }}
    steps:
      - uses: actions/checkout@v4
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.18'
          otp-version: '27'
      - run: mix deps.get
      - run: mix tuist.test

Other providers#

Outside GitHub Actions the build phase writes the plan to .tuist-shard-matrix.json:

json

{
  "reference": "gitlab-1234",
  "shard_count": 2,
  "shards": [
    {
      "index": 0,
      "test_targets": ["MyApp.AccountsTest", "MyApp.CheckoutTest"],
      "estimated_duration_ms": 41000
    },
    {
      "index": 1,
      "test_targets": ["MyAppWeb.PageControllerTest"],
      "estimated_duration_ms": 39500
    }
  ]
}

Use shard_count to set up your parallel jobs, and give each job its TUIST_SHARD_INDEX, from 0 to shard_count - 1. On a provider that Tuist does not derive the reference from, also give every job the same TUIST_SHARD_REFERENCE.