Solano CI tries to automatically detect your project’s Ruby version, from your current environment if you’re using the solano command.


If you need to configure a specific ruby version, the best way is to specify it using a solano.yml

A reasonably complete configuration file for a ruby app might look like this:

ruby_version: ruby-1.9.3-p194        
bundler_version: 1.3.5
  worker_setup: bundle exec rake db:setup    # Runs once for each worker to do, e.g., database setup
  post_setup: bundle exec rake assets:precompile    # Runs once before tests and after worker_setup
  post_build: bundle exec rake heroku:deploy # Runs once after entire build completes
  - spec/**/*_spec.rb                       
  - bundle exec rake jasmine:ci

Also it is possible to customize test command name using name key:

- name: jasmine_tests
  command: bundle exec rake jasmine:ci

Installing Gems in Solano CI

Keeping your Bundle Up-to-date

The first time you push a repository to Solano CI, Solano CI examines your Gemfile.lock to determine the set of gems to install. Solano CI runs a bundle install in deployment mode to install gems and caches the result for future runs. Your Gemfile and Gemfile.lock must be up to date in order for the bundle install to succeed. Solano CI will invalidate the cached bundle if the checked-in copy of the Gemfile.lock changes.

Platform-Specific Gems

Solano CI ignores the tddium_ignore group in your Gemfile when installing gems. Platform-specific gems that should not be installed in the Solano CI environment should be placed in this group and this group only. For instance, if you write your code on a Mac, any Mac-specific gems should go in the development group if they can not be built under other flavors of Unix. You can read more about the tddium_ignore group here.

Bundle Install Time Limit

Solano CI caps the time alloted to the initial bundle install step in order to prevent runaway sessions. The most common cause of a bundle install timeout is a failure to pull a git repo specified in your Gemfile, for instance because you have not authorized the public key used by Solano CI.

Private Gems (and “private” git URLs)

Git(hub) has 3 forms of URL:

  1. HTTPS (public):
  2. SSH (private): ssh:// (or
  3. Git (public): git://

If you have listed a public git repo in your Gemfile using an SSH-style URL, your bundle install may fail in Solano CI with a “public key error”.

Conversely, if you have listed a private git repo, you’ll have to use an SSH-style URL. HTTPS URLs to private repos will produce a password prompt in the Solano Worker process, and that will cause your build to hang until it is timed out.

See our documentation on installing private gems for more information.

Supported Versions and Patch Levels

Solano CI supports most major versions of the Ruby platform:

  • REE (Ruby Enterprise Edition) 1.8.7
  • Ruby (MRI) 1.8.7
  • Ruby (MRI) 1.9.2
  • Ruby (MRI) 1.9.3
  • Ruby (MRI) 2.0.0
  • Ruby (MRI) 2.1.0 - 2.1.10
  • Ruby (MRI) 2.2.0 - 2.2.10
  • Ruby (MRI) 2.3.0 - 2.3.7
  • Ruby (MRI) 2.4.0 - 2.4.4
  • Ruby (MRI) 2.5.0 - 2.5.1
  • JRuby 1.6.7, 1.7.3, 1.7.4, 1.7.8, 1.7.12,
  • JRuby (aka: 9K),,,,,

For each major Ruby version, we support all major patchlevels.

For MRI, the ruby_version specified in solano.yml should be of the form ruby-MAJOR-pPATCH. For instance, ruby-1.9.3-p484. For the 2.1.x series, omit the patch version; for instance ruby-2.1.6.

For JRuby, Solano CI uses a custom specification syntax. For JRuby 9000, simply use jruby-

For earlier versions of JRuby, the syntax is of the form jruby-VERSION-mode-MODE where version is a JRuby version such as 1.7.8 and MODE is currently 1.8 or 1.9. The reason is that you must specify enough information to select both a JRuby implementation version and a language version. The system will not validate MODE against VERSION.

Database Setup for Rails Applications

Make sure that you can:

  1. run bundle exec rake -T without a database set up
  2. run migrations without a database set up (scopes in models often get evaluated eagerly and interefere with migrations).

To specify a particular version of a database subsystem, you will need to add an entry to your Solano CI configuration in solano.yml.

Solano CI will then write a config/database.yml relative to your repository root. The configuration file is YAML with embedded ERB references to environment variables that Solano CI will set when running rake tasks and your tests. Many common database interface gems will automatically handle these YAML-ERB configurations, including Rails (ActiveRecord) and Mongoid.

The list of config files installed:

  • config/database.yml
  • config/mongo.yml
  • config/mongoid.yml
  • config/redis.yml
  • config/resque.yml
  • config/ripple.yml

Once databases are created and config files are written, the system will run appropriate db_hook.

Each of the above configuration files has the following environment sections with identical contents:

  • defaults
  • production
  • test
  • staging
  • development
  • cucumber.

Test Frameworks

Solano CI natively supports the standard test frameworks, including but not limited to the following:

  • RSpec 1, 2, and 3
  • Cucumber
  • Spinach
  • Turnip – currently with RSpec 2 and 3
  • Test::Unit
  • Spork
  • Selenium RC 1.0.8 and Selenium 2
  • Jasmine Webkit
  • Capybara Webkit
  • Poltergeist

Solano CI also supports many common Javascript testing frameworks such as PhantomJS. If you don’t see a tool here, it may already be on our language-agnostic compatibility list.

Solano CI uses the first-level directory to guess how to run a given Ruby test script:

Test Framework Mappings

spec/* RSpec (or Jasmine)
spec/features/* Turnip
fast_specs/* RSpec
features/* Cucumber (or Spinach)
test/* Test::Unit

Solano CI will run tests under the features directory with cucumber if you have Cucumber installed and with spinach if you have Spinach installed.

Cucumber Profiles

Solano CI will process config/cucumber.yml or a cucumber.yml in your repository root with ERB. If there is a hash key in the YAML file called tddium, it will use the value as the profile name for running Cucumber tests, otherwise it will use the default profile. Since the cucumber configuration is parsed as ERB, it is possible to use environment variables to control the cucumber configuration on a branch or suite basis. The TDDIUM environment variable will be set inside Solano CI; environment variables can be specified via config/solano.yml or config variables.

Note that to enable strict mode for Cucumber you will need to add –strict to your profile and also add the following to solano.yml:

    strict: true

RSpec Tags

Solano CI will process the .rspec file at the root of your repository with ERB. Solano CI respects only the tag specification options (–tag and its synonym -t); all other options are ignored. Since the rspec options file is parsed as ERB, it is possible to use environment variables to control the rspec configuration on a branch or suite basis. As in the case of Cucumber, the TDDIUM environment variable will be set inside Solano CI; we recommend setting environment other variables either via config/solano.yml or Solano CI config variables.

Fast Specs

If you have some rspec tests that do not need to load Rails, we recommend that you use a solano.yml file to configure a command to run them (e.g. ruby spec/fast.rb).

One common idiom for this latter case is described by Les Hill.

Gem Blacklist

Gem Version Description
autotest-fsevent   MacOS X specific gem
growl_notify   MacOS X specific gem
rb-fsevent   MacOS X specific gem
rb-appscript   MacOS X specific gem
Capybara < 1.1.0 Chooses Rack server port incorrectly
Cucumber < 0.9.0 Versions before 0.9.0 not supported
Cucumber 1.0.4 Malformed gemspec; use newer version
Selenium-Webdriver < 2.12.0 In general, use most recent version
Bson and bson_ext 1.4.{0,1} 1.4.0 and 1.4.1 were pulled by maintainer

Rake And Rails Environments

By default, rake tasks such as database migrations and setup hooks run in the development Rails environment. In practice, we have found that running migrations and hooks in the development environment offers the widest range of compatibility. Tests, on the other hand, are often best run in the test environment.

To change the environment used for running migrations and hooks to the Solano CI environment, you should explicitly set the RAILS_ENV environment variable in your :ref:`setup hook <setup-hooks>`_ shell command.

You can similarly set the rails environment for tests by setting the RAILS_ENV environment variable. For instance, add the following to config/solano.yml to configur tests to use the Solano CI environment:

  'RAILS_ENV': 'solano'

Javascript in Ruby

Jasmine CI, Guard-Jasmine, and Konacha

Solano CI has preliminary support for running the Jasmine:CI rake task and Guard-Jasmine. Using either one requires a recent version of the solano gem and a little bit of manual configuration. See our guide for Running Commands as Build Steps for more information.

  1. Edit your suite configuration to not select any JS tests:

    $ solano suite --edit

    When you’re prompted for the test pattern, omit

  2. Create a config/solano.yml, and add a “tests” stanza to it:

    # ... existing stuff ...
     - bundle exec rake jasmine:ci       # Replace with whatever command you want to run
     # you can have multiple entries listed here.  They will each be run and displayed separately.

    You can also copy from this gist:

    The tests: key should contain a YAML array of hashes, each hash with keys [type, command, invocation, output].

    The only key you can vary is command, which can be whatever you want. You can have as many custom commands as you’d like.

#. If you are using a library to stub out network connections such as the webmock library in Ruby, we recommend disabling auto-loading it. For Ruby projects with bundler this means setting :require => false in your Gemfile.

If you use the standard Jasmine runner, set the command bundle exec rake jasmine:ci.

If you prefer guard-jasmine, use: bundle exec guard-jasmine --server-timeout=90 -t 30000.

If you use Konacha, use: bundle exec rake konacha:run.

(Really, you can enter whatever command you want, and its results will be automatically integrated into your Solano CI build. Contact us for help with other configurations.)

Jasmine & WebMock

If you are using webmock, you may need to define a new task tddium:jasmine and invoke it instead of invoing jasmine:ci directly. This is due to a bug in webmock that causes it to block connections by default.

namespace :tddium do
  task :jasmine do
    require 'webmock'

Jasmine Headless Webkit (Deprecated)


jasmine-headless-webkit used to be our recommended method for running jasmine tests, as it allowed us to easily separate output from different test scripts. Unfortunately, many of our users have experienced stability and compatibility issues using jasmine-headless-webkit with recent versions of Rails, so we now recommend directly executing e.g. rake jasmine:ci as described above.

Solano CI can run javascript unit tests using Jasmine Headless Webkit. To get started with jasmine and Solano CI, you will need to make sure that the jasmine-headless-webkit gem is in the test group in your Gemfile. When you add the gem, you will need to re-run bundle install locally and check in an updated Gemfile and Gemfile.lock.

Solano CI looks for Jasmine javascript specs under the spec/javascripts directory in your repository and reads the configuration in spec/javascripts/support/jasmine.yml. If you are using a pre-sprockets/asset pipline version of Rails, Solano CI will intersect the set of Jasmine tests selected by your test pattern with the set matching the src_files glob in ;jasmine.yml and run only those that match both. With more recent versions of rails, Solano CI delegates asset pipeline compilation to Rails and runs any Jasmine tests that match your test pattern.

For more information on Jasmine, see the Jasmine Headless Webkit home page. For more information on debugging Jasmine test failures in Solano CI, see here.


Solano CI supports Evergreen. Solano CI will automatically use Evergreen to run specs in spec/javascripts if the Evergreen gem is installed.

Testing Ruby Gems

Solano CI can be used to run the test suites for Gems, too. It is a common misperception that you can not or should not specify dependencies in the Gemfile or check in the Gemfile.lock for gems. The dependencies for gems are specified in the gemspec and not the Gemfile. You can put development dependencies in either place and with the handy gemspec directive in the Gemfile you don’t have to repeat yourself.

If you absolutely cannot add Gemfile.lock to your repo, then we recommend at least adding the following to solano.yml to invalidate when the gemspec changes. Of course you will want to replace gemname.gemspec with the actual filename of your gemspec.

    - Gemfile
    - gemname.gemspec

Solano CI performs best if you specify a Gemfile.lock since it can then cache the bundle and invalidate it when necessary. The best practice is to keep both the Gemfile and Gemfile.lock in your repository. Whenever you update the gemspec, re-run bundle install to update your Gemfile.lock and check it in. A sample Gemfile as used by the solano gem is shown below:

source ""

# Specify your gem's dependencies in tddium.gemspec

group :development, :test do
  gem 'aruba', '0.4.6'
  gem 'pickle'
  gem 'mimic'
  gem 'daemons'
  gem 'httparty'
  gem 'antilles'

  gem 'rspec'
  gem 'cucumber'

Asset Caching

If you have large compiled assets, it may make sense to cache pre-compiled assets out-of-band. Solano CI does not yet have built-in support for this work flow, but it can be implemented with a post_setup hook such as the one shown below, which caches assets in S3.

# coding: UTF-8

namespace :tddium do
  desc 'tddium post hook'
  task :post_hook do
    # Cache precompiled assets in S3 and use turbo-sprockets
    # to avoid recompiling things that have not changed.
    WebMock.allow_net_connect! if defined?(WebMock)
    branch = `git rev-parse --abbrev-ref HEAD`
    s3_conf = YAML.load( File.join(Rails.root, 'config', 'amazon_s3.yml'))['development']
    bucket =[s3_conf['bucket']]
    branch_tarball = bucket.objects["tddium-precompiled-assets/#{branch}.tar.gz"]
    default_tarball = bucket.objects["tddium-precompiled-assets/_default.tar.gz"]
    tarball_destination = File.join(Rails.root, "tmp", "assets.tar.gz")
    tarball_to_download = if branch_tarball.exists?
      puts "Downloading precompiled assets from S3 to #{tarball_destination}"
    elsif default_tarball.exists?
      puts "Downloading default precompiled assets from S3 to #{tarball_destination}"
      puts "No precompiled assets available, will compile from scratch"
    if tarball_to_download, "wb") do |f| do |chunk|
          f.write chunk
      `tar xvfz #{tarball_destination}`
      File.delete tarball_destination
    `tar cvfz #{tarball_destination} public/assets`
    puts "Uploading precompiled assets to S3"
    branch_tarball.write(file: tarball_destination)

Rails Engines and Secondary Bundles

Occasionally a single repository will have multiple Gemfiles and associated bundles. Generally it is best to split these into separate repositories and test them separately, but this is not always possible. One common example where multiple bundles are put into a single repository is the case of Rails engines. The complication that engines introduce is that bundler maintains global state in the filesystem by default and this can cause unexpected failures.

One way to run engines in Solano CI is to use explicit test commands. For instance if you have three engines: account, admin, and search each in the engines directory, you could run them in parallel as follows. Add the run_engine.rb script to the bin directory in the root of your repository and add the following configuration to solano.yml (this example assumes rspec):

ruby_version: ruby-2.1.1      # choose appropriate version
bundler_version: 1.6.2       # to run unmodified 1.6.x or later
  pre_setup: npm install     # optional; only if you use nodejs
  worker_setup: bundle exec rake db:setup         # for main app
  post_setup: bundle exec rake assets:precompile  # for main app
  - ruby bin/run_engine.rb run account
  - ruby bin/run_engine.rb run admin
  - ruby bin/run_engine.rb run search

To split a single engine up into multiple, parallel batches run_engine.rb can take an optional third parameter. The third parameter to run_engine.rb is a glob relative to the engine’s root directory that specifies a subset of tests to run.

Rubocop Configuration

If you use Rubocop, the following configuration may be used to exclude Solano CI-specific code.

- db/**
- config/boot.rb
- rgloader/loader.rb
- !ruby/regexp /.*solano.*\.rb$/

Capybara Screenshots

By default, Solano CI will enable screenshot and HTML capture for rspec and cucumber failures if you have installed the capybara-screenshot gem. This is implemented by dropping a file wrapping the gem into spec/support and features/support. This wrapper is responsible for attaching the generated screenshot to a specific test script so that it can be displayed as part of the build report. Images attached to a test script will appear in the expanded detail view of the relevant test script.

Your test suite must then automatically load the newly added files (by convention most test suites using rspec and cucumber load all files in these directories). If you have difficulty with the default extension, we recommend starting a debug console and examining the extension to ensure it is compatible with your test suite.

You can disable the addition of the extension with the following configuration:

    rspec: false               # or cucumber/spinach

If you want to install your own hook or you are using a variant of test-unit, we recommend using the capybara-screenshot gem. You can then attach the file to the build, or better yet the failing test script, using the build agent interface.