Browser-Based Integration Tests

Integration and acceptance tests that use real browsers or simulated browsers are great for really exercising your app. The power they provide comes with some complexity. We’ve collected information on common problems with Capybara, Selenium, and other related tools.

Capybara: Subdomains

One approach is to set Capybara.app_host. Many folks edit /etc/hosts locally. We suggest using lvh.tddium.com or lvh.me which have wildcard A records in DNS that point .lvh.me and to 127.0.0.1. You can read more about this approach here.

Capybara: Blank Pages, SystemExit and 500 Server Error

Three common, frustrating errors with Capybara are server timeouts that resulting in Capybara raising the SystemExit exception, failed redirections, and other errors that result in an HTTP 500. By default the stack traces for these types of errors get eaten, making them hard to diagnose.

One way to get a stack trace to help debug these sorts of problems is to wrap your application in a stub that logs the error. You can add the following snippet to your Capybara configuration, making sure to change MyApp to the name of your application.

You can download this example: https://gist.github.com/1371143

### Show stack trace for 500 errors
### (adapted from https://gist.github.com/1079020)

# Given an application, yield to a block to handle exceptions
class ExceptionRaiserApp
  def initialize(app)
    @app = app
  end

  def call(env)
    @app.call(env)
  rescue => e
    backtrace = [ "#{'*'*20} #{e.to_s} #{'*'*20}",
                  '',
                  e.message,
                  '',
                  Rails.backtrace_cleaner.clean(e.backtrace, :silent),
                  '',
                  '*'*80,
                  '' ].join("\n\t")

    # log the backtrace
    Rails.logger.error backtrace

    # re-raise so capybara gets a 500 and knows what's up
    raise e
  end
end

Capybara.app = ExceptionRaiserApp.new(MyApp::Application)

Capybara and Selenium tests are often timing dependent and the timing of tests often changes when the test suite is run in parallel in Solano CI. The first step in debugging these test failures is to make sure you have made the changes necessary to run your tests in parallel. There is a separate discussion of parallelism-related Selenium failures in handling failures.

Capybara Timeouts and Server Exits

If you are seeing Selenium/Webkit tests fail in Solano CI with SystemExit or Timeout::Error exceptions or if you see “Rack application timed out during boot” there are three likely causes:

  1. You’re explicitly setting the Capybara.server_port. Recent versions of Capybara (after 1.1.0), will automatically pick a server port in a robust way.
  2. You’re using an old version of Capybara (earlier than 1.1.0) that is less intelligent at automatic port picking.
  3. You have started your own copy of X11/Xvfb, possibly using the headless gem. Solano CI manages the X11 display for you, so you should not start your own headless display when ENV[‘TDDIUM’] is true.

If you can upgrade Capybara, we recommend that you do so. If not, we’ve tested the following patches:

If you are using Capybara 0.4.0 or later with Selenium, then add the following to your features/support/env.rb:

def find_available_port
  server = TCPServer.new('127.0.0.1', 0)
  server.addr[1]
ensure
  server.close if server
end

if ENV['TDDIUM'] then
  Capybara.server_port = find_available_port
end

If you are using Capybara 0.3.9 or earlier, there is no attribute to set the server_port, so a little monkey-patch is needed. Add the following to your features/support/env.rb:

require 'socket'
class Capybara::Server
  private
    def find_available_port
      server = TCPServer.new('127.0.0.1', 0)
      @port = server.addr[1]
    ensure
      server.close if server
    end
end

Capybara hangs when using capybara-webkit

Problems with webkit-server hangs have been reported when using some versions of the thin webserver with some versions of libqt, the toolkit used to implement much of webkit. Users have reported success switching to Webrick or mongrel for their capybara app-under-test.

Here’s an example configuration to switch to Webrick:

# See: https://github.com/thoughtbot/capybara-webkit/issues/331
Capybara.server do |app, port|
  require 'rack/handler/webrick'
  Rack::Handler::WEBrick.run(app, :Port => port, :AccessLog => [], :Logger => WEBrick::Log::new(nil, 0))
end

Failure to start capybara-webkit

When changing either the capybara-webkit gem or the version of Qt library specified in a configuration file, the dependency cache may need to be dropped and rebuilt. The default dependency cache settings will automatically do so when Gemfile[.lock] files are changed.

If a webkit_server failed to start after 15 seconds error occurs, you can increase the timeout value with:

Capybara::Webkit.configure do |config|
  config.class.parent::Server.send(:remove_const, :WEBKIT_SERVER_START_TIMEOUT)
  config.class.parent::Server::WEBKIT_SERVER_START_TIMEOUT = 30 # default 15
end