Worker Rebalancing ControlsΒΆ

By default, Solano CI will automatically batch and distribute tasks across workers to run the commands and file-pattern-matched tests for your build. Part of this workload distribution is an automatic rebalancing of work so that each worker finishes in approximately the same time, leading to shorter builds.

This dynamic rebalancing can sometimes change the grouping and order in which tests run, sometimes even if no code has changed. Unfortunately, many tests have implicit dependencies on order because it’s hard to ensure that all tests must pass in every possible order. That is, if you have tests A, B, and C, then they must pass when run as ABC, ACB, BAC, etc.

To reduce intermittent failures due to test reordering, we support a mechanism to freeze the balance of tests across workers, for all branches of a repo. As tests are added to the repo, the optimal balancing across workers will degrade. In order to return to the optimal balance, when you have the time to deal with new test failures, we support a a workflow that lets you rebalance on a branch, fix any new order dependent failures and then promote to new balance to be used re-wide after merging.

The UI that exposes this workflow is in limited release.

When Working Rebalancing is enabled, organization admins will see a new widget at the bottom of each report page, titled “Worker Rebalancing Controls”. The recommended workflow is:

  1. Make a new branch. Push it or use solano run to start a build for it.
  2. Click “Use” or “Rebalance” on the branch. A new session will start using a recomputed balance. (“Rebalance” will be shown if there’s an existing worker balance. If there is none, “Use” will create a new one from the current session.)
  3. Work in the branch until it’s green, merge fixes upstream, and then click “Promote” in the branch to apply the balance repo-wide.
  4. We suggest that you announce the update to the group before promoting the schedule so that any changes from (3) can be merged upstream.