Most users get the greatest value from Solano CI by setting it up to run automatically when commits are pushed. See Continuous Integration to set this up.
You can also run tests from your individual working copy, at the command line. See Use Solano CI from the CLI.
Either way, you’ll need to add a
solano.yml configuration file,
You configure your tests and their environment in a file called
solano.yml. This configuration file is in YAML and looks like this example:
tests: - nosetests python: python_version: 2.7 pip_requirements_file: requirements.txt
solano.yml file should be in the root of your repository or
config/solano.yml relative to the root.
Also it is possible to customize test command name using
tests: - name: Python test examples command: nosetests
Your Test Command¶
tests key contains a list of tests. The simplest and
most general way to configure a test is as a shell command. For
tests: - python setup.py test # one test - python test/slowtests.py # another test
If an item in
tests is a string, it will be passed to the shell for
expansion and execution. The command’s Unix exit status marks success
or failure of the test, and its output is captured and shown on the
Note that punctuation in complicated commands may be meaningful to YAML.
You can test how your YAML parses here or with
print yaml.load(foo) in Python.
For test suites that run with nosetests or py.test, Solano CI can automatically parallelize test files and give detailed results broken down by file.
Before using the nosetests or py.test support, you should first
ensure that the rest of the environment is set up correctly by just
py.test as a simple command. If that works,
try enabling parallel mode using one of these config snippets:
tests: - type: nosetests mode: parallel output: exit-status files: - **/*_test.py
tests: - type: pytest mode: parallel output: exit-status files: - **/*_test.py
You have to provide one or more patterns to find test files. The patterns are extended shell-style globs, where a double-star matches any number of path components.
You’ll also have to be sure to list
pytest in your
requirements.txt file. See the section below on Library
Note that test suites based on Python’s built-in unittest module can be run by either of these test runners with no modifications.
Limitations: Parallel mode is currently not supported for Mercurial repos.
Solano CI supports all current versions of all major implementations of
Python: the standard implementation (also known as CPython), Jython, and PyPy.
Choose a version by setting
python_version within the
python: python_version: '3.4'
(Make sure the version is in quotes so it’s not interpreted as a number.)
A version number with two parts like 3.4 means “the latest version of Python 3.4”, which is currently 3.4.0. You can also write:
python: python_version: '3.4.0'
to pin to that specific version. It is an error if the specific version is not supported.
The current list of available Python versions is:
Please note that pip does not currently support 3.0.x or 3.1.x (see Github Issue regarding 3.1.x).
To ensure your tests run in a known environment, Solano CI installs all
and only the third-party libraries you specify. If your application
uses any libraries from outside of the Python standard library, you can list them in the standard
Python dependency-tracking format, a Pip requirements file, aka
requirements.txt. For example, a Django application using MySQLdb
and tested with
nosetests might have a
the following contents:
Django MySQL-python nose
Supply the path to your requirements file (relative to the repo root) like so:
python: pip_requirements_file: requirements.txt
We’ve had a number of questions about setup.py vs requirements.txt and several users have found this blog post to be a useful discussion of the distinction.
Solano CI with Python supports the full range of databases, search engines, and other background services (Background Services) that Solano CI supports generally. However, it doesn’t yet automatically detect what services you need (as Solano CI ordinarily does), so you’ll need to configure them directly. For example, to automatically start MySQL:
mysql: version: '5.5'
See Databases and Search for details and to configure any database.
Behind the Scenes¶
To help get a mental model of what’s going on, here’s a brief overview of what happens to run a typical Python test suite.
In the setup phase of execution, a virtualenv will be created at
~/python-env using your selected Python version. If you have
specified a requirements file,
pip install -r <requirements file>
will be run. At this point, we’ll also use pip to install our nosetests
The test environment will be configured as if the virtualenv was
activated. That means that
~/python-env/bin will be at the front of
VIRTUAL_ENV will be set to
When using plain command mode, the given commands are then run as-is. If those commands use nosetests or py.test or another test runner, that test runner will be responsible for enumerating test files.
When using nosetests or py.test parallel mode, Solano CI enumerates your test files by matching the given file patterns against the contents of your repo. The test files are then intelligently partitioned into batches, based on run time history and other metrics. For each batch, Solano CI runs a command that looks roughly like this:
py.test --junitxml=<output> <test> ...
nosetests --with-tddium-output --tddium-output-file=<output> <test> ...
Each batch gets a separate output file, and all outputs are merged by the system to generate a unified test result view.
For py.test, we use use the built-in JUnit XML output formatter.
For nosetests, the built-in XUnit output formatter leaves out some useful information, so for completeness, we install our own output plugin that writes results in a simple JSON format.