Parallelism And Intermittent Failures

Sometimes tests fail intermittently in Solano CI. There are two main reasons: Solano CI’s parallelism, and your test’s interaction with asynchronous behavior.

Other Causes of Intermittent Failures

Timeouts

Integration/browser testing tools like Capybara, Selenium, Capybara-Webkit, or Culerity rely on timeouts to start dependencies and to handle browser behavior.

Xvfb

You should be sure not to start a copy of X11 or Xvfb yourself as Solano CI manages the display for you. One common method for manually starting the display is to use the headless gem.

Selenium and Asynchronous Behavior

Selenium and Capybara-WebKit both can’t handle asynchronous JavaScript events in a completely deterministic way, by the very nature of the testing that needs to happen.

Capybara relies on waits and timeouts to synchronize with events on the page being tested. A test may take longer to execute some behavior in Solano CI, and trigger a Capybara failure timeout.

Disabling Garbage Collection

Large test suites often disable the Ruby garbage collector in an effort to improve local test performance. While disabling the garbage collector may make sense and work well in your environment, it often leads to hard to diagnose and intermittent failures in Solano CI. If you need to disable the collector locally, make it conditional on ENV[‘TDDIUM’] not set.