Python

Solano CI supports testing Python applications. See the Sphinx Python document generator as one of our Example Repositories. Contact support@solanolabs.com with any questions about Python support.

Running Tests

Most users get the greatest value from Solano CI by setting it up to run automatically when commits are pushed. See Continuous Integration to set this up.

You can also run tests from your individual working copy, at the command line. See Use Solano CI from the CLI.

Either way, you’ll need to add a solano.yml configuration file, described below.

Configuration

You configure your tests and their environment in a file called solano.yml. This configuration file is in YAML and looks like this example:

tests:
  - nosetests
python:
  python_version: 2.7
  pip_requirements_file: requirements.txt

The solano.yml file should be in the root of your repository or located at config/solano.yml relative to the root.

Also it is possible to customize test command name using name key:

tests:
  - name: Python test examples
    command: nosetests

Your Test Command

The tests key contains a list of tests. The simplest and most general way to configure a test is as a shell command. For example:

tests:
  - python setup.py test       # one test
  - python test/slowtests.py   # another test

If an item in tests is a string, it will be passed to the shell for expansion and execution. The command’s Unix exit status marks success or failure of the test, and its output is captured and shown on the report page.

Note that punctuation in complicated commands may be meaningful to YAML. You can test how your YAML parses here or with import yaml; print yaml.load(foo) in Python.

Parallelization

For test suites that run with nosetests or py.test, Solano CI can automatically parallelize test files and give detailed results broken down by file.

Before using the nosetests or py.test support, you should first ensure that the rest of the environment is set up correctly by just setting nosetests or py.test as a simple command. If that works, try enabling parallel mode using one of these config snippets:

For nosetests:

tests:
  - type: nosetests
    mode: parallel
    output: exit-status
    files:
      - **/*_test.py

For py.test:

tests:
  - type: pytest
    mode: parallel
    output: exit-status
    files:
      - **/*_test.py

You have to provide one or more patterns to find test files. The patterns are extended shell-style globs, where a double-star matches any number of path components.

You’ll also have to be sure to list nose or pytest in your requirements.txt file. See the section below on Library Dependencies.

Note that test suites based on Python’s built-in unittest module can be run by either of these test runners with no modifications.

Limitations: Parallel mode is currently not supported for Mercurial repos.

Python Versions

Solano CI supports all current versions of all major implementations of Python: the standard implementation (also known as CPython), Jython, and PyPy. Choose a version by setting python_version within the python section:

python:
  python_version: '3.4'

(Make sure the version is in quotes so it’s not interpreted as a number.)

A version number with two parts like 3.4 means “the latest version of Python 3.4”, which is currently 3.4.0. You can also write:

python:
  python_version: '3.4.0'

to pin to that specific version. It is an error if the specific version is not supported.

The current list of available Python versions is:

  • 2.6.9
  • 2.7.6
  • 3.0.1
  • 3.1.5
  • 3.2.5
  • 3.3.3
  • 3.4.0
  • 3.5.0
  • jython-2.5.3
  • pypy-1.9
  • pypy-2.2.1
  • pypy-2.4.0
  • pypy3-2.4.0

Please note that pip does not currently support 3.0.x or 3.1.x (see Github Issue regarding 3.1.x).

Library Dependencies

To ensure your tests run in a known environment, Solano CI installs all and only the third-party libraries you specify. If your application uses any libraries from outside of the Python standard library, you can list them in the standard Python dependency-tracking format, a Pip requirements file, aka requirements.txt. For example, a Django application using MySQLdb and tested with nosetests might have a requirements.txt with the following contents:

Django
MySQL-python
nose

Each library is identified by the name it uses on the Python Package Index. For more on Pip requirements files, see the official documentation.

Supply the path to your requirements file (relative to the repo root) like so:

python:
  pip_requirements_file: requirements.txt

We’ve had a number of questions about setup.py vs requirements.txt and several users have found this blog post to be a useful discussion of the distinction.

Background Services

Solano CI with Python supports the full range of databases, search engines, and other background services (Background Services) that Solano CI supports generally. However, it doesn’t yet automatically detect what services you need (as Solano CI ordinarily does), so you’ll need to configure them directly. For example, to automatically start MySQL:

mysql:
  version: '5.5'

See Databases and Search for details and to configure any database.

Behind the Scenes

To help get a mental model of what’s going on, here’s a brief overview of what happens to run a typical Python test suite.

In the setup phase of execution, a virtualenv will be created at ~/python-env using your selected Python version. If you have specified a requirements file, pip install -r <requirements file> will be run. At this point, we’ll also use pip to install our nosetests output plugin.

The test environment will be configured as if the virtualenv was activated. That means that ~/python-env/bin will be at the front of the PATH, and VIRTUAL_ENV will be set to ~/python-env.

When using plain command mode, the given commands are then run as-is. If those commands use nosetests or py.test or another test runner, that test runner will be responsible for enumerating test files.

When using nosetests or py.test parallel mode, Solano CI enumerates your test files by matching the given file patterns against the contents of your repo. The test files are then intelligently partitioned into batches, based on run time history and other metrics. For each batch, Solano CI runs a command that looks roughly like this:

py.test: py.test --junitxml=<output> <test> ...

nosetests: nosetests --with-tddium-output --tddium-output-file=<output> <test> ...

Each batch gets a separate output file, and all outputs are merged by the system to generate a unified test result view.

For py.test, we use use the built-in JUnit XML output formatter.

For nosetests, the built-in XUnit output formatter leaves out some useful information, so for completeness, we install our own output plugin that writes results in a simple JSON format.

Contact

If you find any of this documentation unclear or have any questions about using Solano CI with Python, please don’t hesitate to ask us at support@solanolabs.com.