Automatic Rerun for Flaky Tests¶
Use of this feature for builds started using
solano run requires a recent version of the solano gem.
Flaky tests lead to a huge amount of wasted developer effort. Of course, in the ideal world, all tests would be 100% predictable and deterministic. However, with browser-based integration tests, or tests that rely on live results from third-party API sandboxes (like Youtube or Facebook), it’s not uncommon for a useful test to sometimes fail unnecessarily.
Solano CI now supports automatic rerun of tests that have been specifically marked as flaky.
If a test fails and is rerun, output from each run is captured and preserved.
The final status of the test is what counts towards the build result. So, if a test fails on its first run and passes on its second run, it will be counted as passed.
The number of tests rerun and the number of reruns per test are both displayed in the report page for a build.
An example configuration will illustrate how to enable and use rerun.
test_pattern: - spec/requests/**_spec.rb rerun_list: - spec/requests/flaky_spec.rb - spec/requests/really_flaky_integration_spec.rb: 10 - spec/requests/long_flaky_spec.rb: 1 - spec/requests/flaky_requests/*_spec.rb: 1
This configuration selects 3 tests as rerun-candidates. If any of them fails, it will be rerun until it passes, up to a rerun limit.
By default (as in for
flaky_spec.rb), a selected test will be run
at most 4 times. If you want to set a different limit, specify
it in the configuration, as shown for
Keep in mind: if you have a single test script that takes 5 minutes to complete, and you indicate that it should rerun 5 times, your build time could increase by 20 minutes. We recommend a lower rerun limit for longer tests. You could also split the flaky testcase out of the longer test script.
There can be at most 20 entries in the
The rerun_list selectors can be one of:
- a repo-relative path of a test file (as in the above example)
- a repo-relative path of a test file with glob patterns using the same syntax as
test_pattern(as in the above example)
- a regular expression, starting and ending with
/(see example below)
If you use a regular expression pattern, it can apply to tests specified by
test_pattern AND to test commands itemized in the
tests block of
An example configuration allowing rerun of a test command using a regular expression selector:
tests: - ./some-really-solid-test.sh - ./a-really-flaky-test.sh rerun_list: - '/flaky/': 2
This example will rerun the
a-really-flaky-test.sh command up to 2
additional times if the first runs fail.