Build Profiles and Plans

Availability

Support for multiple configuration profiles per repo and chaining together profiles in a build plan is in public Beta.

Usage Scenarios

Solano CI offers many configuration options to control how your build environment is set up and how to run your build and tests.

In a simple project, each repository you configure with Solano CI will have a single configuration for its build, including language runtime versions, dependencies, and the commands to run and the tests execute in order to complete the build.

More complex projects often need more flexibility in configuration to support situations like:

  • Running the same build process with different environment variables set
  • Running tests against different runtime versions or database versions
  • A build plan that runs a subset of the tests first (e.g. smoke tests) and only if they pass, running a larger suite of integration tests
  • Periodically (e.g. nightly) running a much larger or more expensive test suite

Concepts

Profile
A set of parameters to customize, overriding the top-level configuration in solano.yml. Configuration of profile-related behavior belongs in the profile_config sub-hash of a profile configuration. This configuration can, for instance, be used to control how a plan should treat the failure of a profile step.
Plan
A sequence of Steps, each identifying a Profile to run by default when the repo is pushed to, a user clicks “Start Build”, or with solano run
Step
The name of a Profile to run in sequence. Each Step runs as a separate, Subordinate Session.
Parent Session
The Build/Session representing the Plan, where you can see the progress of each Plan Step.
Subordinate Session
A Session representing a Step in a Plan.
Plan Variable
A step in the plan can bind a value to a variable for use later in the plan

Example: Splitting Smoke Tests and Integration Tests

The following configuration file runs the default toplevel profile first in a subordinate build. If that build passes, it runs the integration profile. It also defines an integration_chrome profile that runs the integration tests using chrome.

plan:
  - default
  - flaky_integration
  - integration
profiles:
  default:
    tests:
      - npm test
    test_pattern:
      - spec/unit/**_spec.rb
  integration:
    environment:
      INTEGRATION: true
    test_pattern:
      - spec/integration/**_spec.rb
    test_exclude_pattern:
      - spec/integration/chrome/**_spec.rb
      - spec/integration/flaky/**_spec.rb
  flaky_integration:
    environment:
      INTEGRATION: true
    profile_config:
      advance_on: 'always'
    test_pattern:
      - spec/integration/flaky/**_spec.rb
  integration_chrome:
    environment:
      INTEGRATION: true
      BROWSER: chrome
    test_pattern:
      - spec/integration/**_spec.rb
    test_exclude_pattern:
      - spec/integration/firefox/**_spec.rb
      - spec/integration/flaky/**_spec.rb

Example: Dynamically Determining the Next Plan Step

It may be useful to dynamically determine the next plan step to run, for example if you want to choose which subset of tests to continue with.

The following configuration example uses a “Plan Variable”, set by the compute-next-to-run script (see below) for the profile name of the second step. compute-next-to-run will be executed as a post-build hook. When the first step in the plan completes, Solano CI will substitute the variable to determine the name of the profile to execute in the next step.

Note that the first step includes the sets: next_profile line, which indicates that the variable next_profile is usable elsewhere in the plan.

Note

The SOLANO_PROFILE_NAME and TDDIUM_PROFILE_NAME environment variables will be set in the build worker with the name of the currently running profile.

Solano CI will consistency check plans to ensure that plan variables are declared before they are used.

plan:
- profile: default
  sets: next_profile
- profile: "{{ next_profile }}"

hooks:
  post_build: "./compute-next-to-run"

profiles:
  default:
    tests:
    - "/bin/true"
  first:
    environment:
      MY_PROFILE: "first"
    tests:
    - '[ "$MY_PROFILE" == "first" ]'
  second:
    environment:
      MY_PROFILE: "second"
    tests:
    - '[ "$MY_PROFILE" == "second" ]'
  third:
    environment:
      MY_PROFILE: "third"
    tests:
    - '[ "$MY_PROFILE" == "third" ]'

The compute-next-to-run script can be any executable that generates a solano-plan-variables.json file with a hash of variable names to values.

The solano-plan-variables.json file will be collected just after the post_build hook runs. Technically, any process that runs in the course of the build can generate it.

Here’s a contrived example that chooses the value of next_profile at random:

require 'json'

# pick at random for the sake of example.
# replace this with your own logic
next_profile = %w(first second third).sample

puts "choosing #{next_profile}"

File.open("solano-plan-variables.json", "w") do |f|
  f.write(JSON.dump({"next_profile" => next_profile}))
end

Of course, you should implement more meaningful logic based on your needs.

Example: Continue On Failure

In some cases you may want a build plan to continue even after a build step fails. You can control how the failure of a build step is treated via the profile_config block in a profile definition.

Two values are currently permitted: passed and always; the default is passed. When advance_on is evaluates to passed, the build plan will advance only if the current profile passes all of its tests. If it is set to always, the profile will always advance to the next step, if any.

The overall build will only be marked as a pass if all profile steps pass all of their tests.

The advance_on keyword can be used to specify an explicit value or a list of branch patterns to match. The system will evaluate the list of branch patterns to determine the value of advance_on to use.

tool_config:
  git:
    version: '2.0.3'
plan:
  - smoke
  - flaky_integration
  - integration
profiles:
  smoke:
    tests:
      - /bin/true
  flaky_integration:
    profile_config:
      advance_on:                   # can also just be string value
        - branch: master
          advance_on: "passed"
        - branch: "*"
          advance_on: "always"
    tests:
      - /bin/false
  integration:
    tests:
      - test `git --version | awk '{ print $3 }'` == '2.0.3'

Running A Specific Profile

From the Command Line

To run a specific profile from the command line, use the --profile=name option. For example, if you have a crossbrowser profile, you could run:

$ solano run --profile=crossbrowser

From a Web Hook

If you have a crossbrowser that you want to trigger via a webhook API call to Solano CI, you can include the profile_name field in the JSON payload. For example, with curl:

curl -H 'Content-type: application/json' \
     -d '{"source": "solano", "branch": "unstable", \
          "force": true, "profile_name": "crossbrowser"}' \
     https://hooks.solanolabs.com/1/builds/2390812089347102894710897241089274

See Build Trigger Webhooks (Incoming) for more information on where to find the webhook URL for your repo.

Limitations

  • Plan variables are only evaluated in the profile key of a step