Caching Dependencies

Solano CI intelligently caches directories in your build environment, making large or expensive-to-produce artifacts persist from build to build.

Most often, you’ll want to cache directories containing project dependencies, like packages managed by bundler, pip and npm. These package installs often take several minutes – time that you don’t want to be wasting on every single build.

Defaults: bundler, pip

By default, Solano CI will automatically perform dependency installation for ruby and python projects based on the presence of package manifest files (Gemfile.lock or requirements.txt).

By default, our caching is keyed on the contents of the following files used by the common dependency managers:

  • Gemfile (bundler)
  • Gemfile.lock (bundler)
  • requirements.txt (pip)
  • package.json (npm)

Solano CI will keep its cache until one or more of these manifest files in your repo changes.

When a manifest file changes, the cache will be invalidated and a new install will happen automatically for bundler and pip. For Node projects, you’ll have to run npm install from a pre_setup hook. More automation is on the way.

Configuration

For most Ruby projects, you won’t need to do much configuration, as our defaults will work out of the box.

If you want to customize the files Solano CI checks to invalidate its cache, add a cache section to your solano.yml file. For example, to key on a different pip requirements file:

python:
  requirements_file: requirements-ci.txt
cache:
  key_paths:
    - requirements-ci.txt

You can also control the paths to be saved in the cache using save_paths in the configuration file. For example, to use a custom virtualenv, you could configure as follows:

hooks:
  pre_setup: "virtualenv env && pip install server/requirements.txt"
cache:
  key_paths:
    - server/requirements-ci.txt
  save_paths:
    - env

The format of save_paths is a list of path specifiers:

  • Paths can be directories or filenames.
  • By default, paths are relative to the repo root
  • To make the path relative to the home directory, prefix it with HOME/. You can also explicitly use prefix REPO/ to indicate a repo-relative path.
  • Paths cannot contain .. parent-directory references.

A path specifier is either a string literal conforming the description above or a hash.

If a path specifier is a hash, it must contain a key named path that has the string specifier as its value. In this case, it may also have the optional key overwrite which must be a boolean (default: false). If the overwrite option is true, then if the path exists in the repository, it will be recursively removed before restoring the cache. We recommend against relying on the overwrite option but it can be useful in cases such as directories with .gitkeep files.

If you don’t specify save_paths, Solano CI will cache the following:

  • HOME/bundle (where Solano CI’s built-in bundle install writes to)
  • HOME/python-env (where Solano CI’s built-in pip install writes to)
  • HOME/.gem
  • HOME/usercache (put your own artifacts here!)
  • REPO/node_modules

How It Works

When your build starts, after Solano CI pulls your repo from your git server, our system will compute a cache key based on the configured key files (or our defaults).

You will see a message displayed in the build report page indicating the files checked and the keys we computed.

If a cached artifact is available, it will be downloaded by build workers, put into place, and our default bundle install or pip install will be skipped. The cache is restored before any user code runs.

If a cached artifact isn’t available, dependency install will happen from scratch.

When a session completes, including any post-build hooks, the contents of the saved directories will be written into off-worker storage and tracked for the next build.

The cache object is updated only when the cache key is changed. To invalidate the cache manually, you can change one of the key files and push. You can also use the “Build Now (Drop Cache)” menu item from a report page, which will also trigger a new build.

We are preparing a UI and a CLI to let you invalidate without making a commit or triggering a build.

Disable Caching

When building gems or other libraries, it can be useful to disable caching to ensure your code is using the latest versions of its dependent libraries. This can be useful in exposing conflicts when changes are made to external libraries.

Please note that disabling Solano CI’s cache will often negatively affect build times, sometimes substantially.

To disable caching, use the following as the cache section in your solano.yml file:

cache:
  nocache: true

CLI Compatibility

To take advantage of dependency caching for CLI builds, you will need to have a recent version of the solano gem installed. Earlier versions of the CLI gem will perform dependency install on every run.

Cache modes

Drop

The cache object is updated only when the cache key is changed. To invalidate the cache manually, you can change one of the key files and push. You can also use the “Build Now (Drop Cache)” menu item from a report page, which will also trigger a new build.

Ignore

Normally a cache object will always be used if one is found. To test a build without a cache uou can use the “Build Now (Ignore Cache)” menu item from a report page, which will also trigger a new build. This build will not use a cache object. However the option does not invalidate the cache so it will still be able to be used in future builds.