Build And Setup Failures

Occasionally, a build fails in Solano CI during the setup phase before the first test runs. This phase includes building and installing the bundle based on the gems specified in the Gemfile and running rake setup tasks that install configuration files and run migrations. There are a few common causes of errors during the setup phase:

This article explains how you can address many of these cases.

Git Clone Failures

If your build encounters an error that looks like this:

Git clone failed.

Make sure the following public key is authorized:

That means there was a problem authenticating via SSH with your git server (e.g., GitHub, Bitbucket, Gitlab, Atlassian Stash or your self-hosted git repo).

The most likely reason is that the SSH keypair used to initiate the build isn’t authorized for this repo.

For example, to check a Github repo’s configuration:

  1. Open the repo’s settings page (find the tools icon on the main repo page).
  2. Select “Deploy Keys”.
  3. See if the key is already listed by fingerprint.

If it’s not listed, add it by clicking on “Add deploy key”.

If the key is already configured, and you still receive this error, contact Solano Labs support.

Missing Binary Dependencies

Solano CI’s test workers come standard with a large set of common binary dependencies. If your application’s bundle install fails to compile because of a missing binary dependency, contact us at support@solanolabs.com.

Database Setup Failures

Failed Migrations

Some times migrations, schema dumps, or application configuration (e.g. routes.rb) inadvertently depend upon the existing contents of the database. This means that is impossible for Solano CI (or you!) to reconstruct the application state from scratch. The best way to resolve these sorts of problems is to create a fresh workspace (or if necessary drop the database with rake db:drop) and then run:

  1. rake db:create
  2. rake db:migrate or rake db:schema:load as appropriate
  3. rake routes – dump Rails routes

Note that if you need to run both db:schema:load AND db:migrate that db:migrate modifies schema.rb and therefore it is NOT safe to run multiple copies in parallel and worker hooks do run in parallel.

Referencing the DB before it exists

Does your Rails app build fail in running the tddium:db_hook, for example with an error like this?

unable to run rake task tddium:db_hook

rake aborted!
Mysql2::Error: Table 'u999_0.model' doesn't exist: SELECT `model`.* FROM `model`
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract_adapter.rb:202:in `rescue in log'
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract_adapter.rb:194:in `log'
/home/u999/bundle/ruby/1.9.1/gems/mysql2-0.2.6/lib/active_record/connection_adapters/mysql2_adapter.rb:314:in `execute'
/home/u999/bundle/ruby/1.9.1/gems/mysql2-0.2.6/lib/active_record/connection_adapters/mysql2_adapter.rb:635:in `select'
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all'
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract/query_cache.rb:56:in `select_all'
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/base.rb:467:in `find_by_sql'
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/relation.rb:64:in `to_a'
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/relation/finder_methods.rb:143:in `all'
/home/u999/bundle/ruby/1.9.1/gems/activerecord-3.0.3/lib/active_record/base.rb:439:in `all'
/home/u999/src/repo/myproject/app/models/records.rb:95:in `models'

This is usually due to code in a model class that references the database when the class is loaded. Something like:

class Record < ActiveRecord::Base
   validate :model_type, :inclusion=>Model.all.collect(&:type)
end

When Solano CI tries to run migrations, it has to load your app as part of the rails environment. Since migrations haven’t finished, this happens before the database has been created, so the call to Model.all will fail.

To break the dependency loop, wrap the database-dependent code in a Proc - that way, it runs when it’s evaluated, and not at class load time.

class Record < ActiveRecord::Base
   validate :model_type, Proc.new{{:inclusion=>Model.all.collect(&:type)}}
end

You can easily identify circular database references in your code by running migrations in an environment without a database. Note that by default Solano CI runs migrations in the development environment and tests in the test environment. This behavior is configurable. See Databases and Search.

$ RAILS_ENV=development rake db:drop
$ RAILS_ENV=development rake db:setup

Missing Postgres Extensions

By default, Solano CI does not install any of the Postgres contrib extensions. The recommended way to install either the standard extensions or your own custom extensions is to use the tddium:db_hook. Read more about Postgres extensions in our Postgres Extension Guide.

Missing or Private Gems

Solano CI installs gems from RubyGems or via git URLs. If Solano CI can not contact RubyGems or the remote git server, it may be unable to complete a bundle install. It is also possible that a gem or a particular version of a gem has been removed from RubyGems since you last ran bundle install and checked in your Gemfile.lock. If this is the case, you will need to re-run bundle install locally and check in an updated Gemfile.lock. Similarly, a revision or branch may be removed from a remote Git repository, preventing the bundle from being built. In this case you will need to update the revision specified in your Gemfile (if any) and re-run bundle install locally to update and check in Gemfile.lock. Typical output looks something like this:

Bundle install failed

For more information on Solano CI and bundle install, see:
    http://docs.solanolabs.com/Setup/installing-gems-in-tddium
To ignore platform-specific gems (e.g. OS X or Windows) see:
    http://docs.solanolabs.com/TroubleshootingGuide/

Bundle Install Output:
Fetching gem metadata from http://rubygems.org/......
Fetching gem metadata from http://rubygems.org/..
Fetching git://github.com/tddium/gem-with-missing-branch.git
fatal: Could not parse object 'bbe05f3665c33d6d3a6b2c81198c8de14fababd6'.
Git error: command `git reset --hard bbe05f3665c33d6d3a6b2c81198c8de14fababd6` in

Remember that Solano CI tries to cache the results of bundle install. When in doubt, run bundle to change your Gemfile.lock so that Solano CI invalidates its cached bundle. If you are having trouble with specific gems, see the Gem Blacklist.

Platform-incompatible Gems

Solano CI automatically blacklists gems that we know to be incompatible. See the Gem Blacklist for more information. If you have gems that are not compatible with Solano CI’s standard Linux environment (for example, Mac-specific gems), you can put them in a group in your Gemfile called :tddium_ignore. For example:

# Gemfile
group :tddium_ignore do
  gem 'growl_notify'
  gem 'rb-fsevent'
end

We’ll soon enable functionality to let you configure the Gemfile groups for Solano CI to ignore. Until then, please use this approach.

We considered automatically ignoring the :development group, but that doesn’t work well. Most developers who use Bundler treat :development and :test as interchangeable, and bundler will only skip a gem if it appears solely in the excluded group(s).

Git Gem URLs

Solano CI caps the time alloted to the initial bundle install step in order to prevent runaway sessions. The most common cause of a bundle install timeout is a failure to pull a git repo specified in your Gemfile.

Git URLs in your Gemfile must either refer to a public git repository or be self-authenticating. If you are using a public git repository on github, use a git URI (git://github.com/user/repo) instead of an SSH URI (git@github.com:user/repo).

Copying/Generating application configuration files

Using solano pre_setup hooks, you can easily perform tasks to generate or copy configuration files that are needed by your tests.

Say you use a billing provider, and your application reads keys from config/billing_provider.yml. You don’t want these keys in git, but you’ve saved sample keys in config/billing_provider.example.yml so your developers can get up and running quickly.

You can write a pre_setup hook task like the following to use your example:

namespace :solano do
  desc "Solano environment pre-run setup"
  task :pre_setup_hook do
    system("cd config && ln -s billing_provider.example.yml billing_provider.yml")
  end
end

If you’re not using ruby, you can also write the same command as a pre_setup hook in your solano.yml

solano:
  hooks:
    pre_setup: `cd config && ln -s billing_provider.example.yml billing_provider.yml`