Automatic Rerun for Flaky Tests

Note

Use of this feature for builds started using solano run requires a recent version of the solano gem.

Flaky tests lead to a huge amount of wasted developer effort. Of course, in the ideal world, all tests would be 100% predictable and deterministic. However, with browser-based integration tests, or tests that rely on live results from third-party API sandboxes (like Youtube or Facebook), it’s not uncommon for a useful test to sometimes fail unnecessarily.

Solano CI now supports automatic rerun of tests that have been specifically marked as flaky.

If a test fails and is rerun, output from each run is captured and preserved.

The final status of the test is what counts towards the build result. So, if a test fails on its first run and passes on its second run, it will be counted as passed.

The number of tests rerun and the number of reruns per test are both displayed in the report page for a build.

Example

An example configuration will illustrate how to enable and use rerun.

test_pattern:
- spec/requests/**_spec.rb

rerun_list:
- spec/requests/flaky_spec.rb
- spec/requests/really_flaky_integration_spec.rb: 10
- spec/requests/long_flaky_spec.rb: 1
- spec/requests/flaky_requests/*_spec.rb: 1

This configuration selects 3 tests as rerun-candidates. If any of them fails, it will be rerun until it passes, up to a rerun limit.

By default (as in for flaky_spec.rb), a selected test will be run at most 4 times. If you want to set a different limit, specify it in the configuration, as shown for long_flaky_spec and really_flaky_integration_spec.

Keep in mind: if you have a single test script that takes 5 minutes to complete, and you indicate that it should rerun 5 times, your build time could increase by 20 minutes. We recommend a lower rerun limit for longer tests. You could also split the flaky testcase out of the longer test script.

Configuration

There can be at most 20 entries in the rerun_list.

The rerun_list selectors can be one of:

  • a repo-relative path of a test file (as in the above example)
  • a repo-relative path of a test file with glob patterns using the same syntax as test_pattern (as in the above example)
  • a regular expression, starting and ending with / (see example below)

If you use a regular expression pattern, it can apply to tests specified by test_pattern AND to test commands itemized in the tests block of solano.yml.

An example configuration allowing rerun of a test command using a regular expression selector:

tests:
- ./some-really-solid-test.sh
- ./a-really-flaky-test.sh

rerun_list:
- '/flaky/': 2

This example will rerun the a-really-flaky-test.sh command up to 2 additional times if the first runs fail.