Custom Enumeration

Availability

Please contact support@solanolabs.com regarding availability of this feature.

Usage Scenarios

The primary use case for this feature is to allow more control over which tests are run by a build.

Configuration

This feature allows you to run a custom script that generates the list of tests to run. This script must write a test_list.json file in either the $HOME or repository’s root directory. To enable this feature you must specify an executable script as the enumeration setup hook in your repo’s configuration file.

hooks:
  enumeration: "./enumeration.sh"

The above configuration specifies that a enumeration.sh script in the repo’s root directory will determine the set of tests to run. A trivial example of an implementation might be the following (more advanced examples included below):

echo '{"tests":["features/test_stuff.feature","spec/faked_spec.rb","spec/forking_spec.rb","test/faked_test.rb"]}' > test_list.json

The enumeration hook script must create a test_list.json file in either the home directory or the repo root. This output file must contain a json blob of the form {"tests":["a","b","c"],"commands":["d"]}, having a, b, and c be file paths from the repo root for tests to run, and d is a command.

The generated test_list.json file will automatically be added to the session artifacts. The enumeration hook log file will be added as enumeration.log.

Empty tests

If your enumeration script determines no appropriate tests should be run, it should still create a test_list.json file. Depending on if this behavior should cause the build to pass or fail, one of the following can be used:

{"commands":["echo 'No tests to run'"]}                # pass
{"commands":["echo 'FAIL: No tests found' && exit 1"]} # fail

Simple commands

Custom enumeration supports basic command line tests. The following in a solano.yml configuration file:

tests:
  - /bin/true
  - name: Custom Enumeration
    command: /bin/true

Is equivalent to having the following in test_list.json:

{"commands":["/bin/true",{"name":"Custom Enumeration","command":"/bin/true"}]}

Parallel commands

Parallel command mode allows tests that are run with a command to be automatically parallelized. See language specific doc page for examples of parallel test runners, and parallel Junit for creating your own custom parallel test runner.

The following in a solano.yml configuration file:

tests:
- type: phpunit
  mode: parallel
  output: exit-status
  command: vendor/bin/solano-phpunit
  config: custom-phpunit.xml    # Defaults : phpunit.xml, phpunit.xml.dist
  files:
    - tests/event/**Test.php

Is equivalent to having the following in test_list.json (note that files_expanded is an array of file paths that matched the files pattern from above):

{
  "commands": [
    {
      "type": "phpunit",
      "mode": "parallel",
      "output": "exit-status",
      "command": "vendor/bin/solano-phpunit",
      "config": "phpunit.xml",
      "files": "tests/event/.*Test.php",
      "files_expanded": [
        "tests/event/aTest.php",
        "tests/event/bTest.php",
        "tests/event/cTest.php"
      ]
    }
  ]
}

Helpful ENV VARS

The following variables are avalible in this hook, and can be used to select tests by profile or branch.

$SOLANO_PROFILE_NAME
$TDDIUM_CURRENT_BRANCH
$TDDIUM_LAST_BRANCH_PASSED_COMMIT
$TDDIUM_CURRENT_COMMIT
$TDDIUM_SESSION_ID

More Examples

BASH
Python
PHP
RUBY

BASH

Basic

The following example finds all files in subdirectories that match a "\(spec\|test\)\/.*\.rb" pattern:

#!/bin/sh
set -e


file_list=$(find . -type f -not -path '*/\.*' | grep -o "\(spec\|test\)\/.*\.rb")
echo "\n FILE LIST"
echo $file_list
formated=$(echo "$file_list" | awk ' BEGIN { ORS = ""; print "{@tests@:["; } { print "@"$0"@"; } END { print "]}"; }' )
echo "\n START OF FORMAT"
echo $formated
json=$(echo $formated | sed -e "s/@@/@\, @/g" -e "s/@/\"/g")
echo "\n JSON"
echo $json
echo $json > test_list.json

View on Github

Python

Basic

The following example finds all files in subdirectories that match a pattern that is defined by the current profile.

import fnmatch, os, json, glob, re

jsonarray = {}
files = []

patterns = {
  "default"    : "^tests\/test_.*\.py$"
  }

#get profile name
profile = os.environ["SOLANO_PROFILE_NAME"]
print "selected profile: " + profile

#get the repo root
repo_root = os.path.abspath(os.environ["TDDIUM_REPO_ROOT"])
# test_dir = os.path.join(repo_root, 'test')
test_dir = repo_root

print "repo_root is set as: " + repo_root
print "test_dir is set as: " + test_dir

#figure out what pattern to use
if profile in patterns:
  profile_pattern = patterns[profile]
else:
  print "profile not found in patterns falling back to default"
  profile_pattern = patterns['default']
  profile = 'default'

regex=re.compile(profile_pattern)

print "selected pattern: " + profile_pattern

os.chdir(test_dir)
files = []
for root, dirnames, filenames in os.walk(test_dir):
  for filename in filenames:
    #paths need to be freom the TDDIUM_REPO_ROOT
    name = os.path.join(re.sub(repo_root+"\/", '', root), filename)
    if regex.search(name):
      files.append(name)

#printing so they appear in enumeration.log
print files

#set up parallel command mode
#should be a list of dictionaries
commands = [{
      "type"            : "nosetests",
      "mode"            : "parallel",
      "files"           :  [profile_pattern],
      "files_expanded"  : files
      }]

print commands

#adds a test that is always true
#if we dont find any tests we need this
if len(files) == 0:
  commands = ["/bin/true"]

# #build test_list.json
jsonarray['commands'] = commands

os.chdir(repo_root)
f = open('test_list.json', 'w')
f.write(json.dumps(jsonarray))
f.close()

View on Github

PHP

Basic

The following example finds all files in subdirectories that match a pattern that is defined by the current profile. An example repo is available at solanolabs/guzzle.

<?php
$json = array();
$commands = array();

$patterns = array(
  "default"     =>   "tests/\w*Test.php",
  "subscriber"  =>   "tests/Subscriber/.*Test.php",
  "event"       =>   "tests/event/.*Test.php"
  );

//get profile name
$profile = getenv("SOLANO_PROFILE_NAME");
echo "selected profile: $profile" . PHP_EOL;

//get the repo root
$root = realpath(getenv("TDDIUM_REPO_ROOT"));
$test_dir = realpath($root . DIRECTORY_SEPARATOR . 'tests');
echo "root is set as: $root" . PHP_EOL;
echo "test_dir is set as: $test_dir" . PHP_EOL;

//figure out what pattern to use
if (array_key_exists($profile, $patterns)) {
  $profile_pattern = $patterns[$profile];
} else {
  echo "profile not found in patterns falling back to default" . PHP_EOL;
  $profile_pattern = $patterns['default'];
}

echo "selected pattern: $profile_pattern" . PHP_EOL;

//grab files in repo root
$directory = new RecursiveDirectoryIterator($test_dir);
$flattened = new RecursiveIteratorIterator($directory);

//paths should be from repo_root
$clean_files=new RegexIterator($flattened,'#(' . preg_quote($root) . DIRECTORY_SEPARATOR .')(.*)#',  RegexIterator::REPLACE);
$clean_files->replacement='$2';

$files = new RegexIterator($clean_files, '#' . $profile_pattern . '$#Di');


//printing so they appear in enumeration.log
foreach($files  as $file) {
    echo $file . PHP_EOL;
}

//set up parallel command mode
$commands[] = array(
      "type"            => "phpunit",
      "mode"            => "parallel",
      "output"          => "exit-status",
      "command"         => "vendor/bin/solano-phpunit",
      "config"          => "phpunit.xml",
      "files"           => $profile_pattern,
      "files_expanded"  => array_values(iterator_to_array($files ))
      );

//adds a test that is always true
//if we dont find any tests we need this
$commands[] = "/bin/true";

//build test_list.json
$json['commands'] = $commands;

$fp = fopen('test_list.json', 'w');
fwrite($fp, json_encode($json));
fclose($fp);

View on Github

RUBY

Base

The below ruby examples presume the following Enumerate module is available:

require 'yaml'
require 'json'
require 'open3'

module Enumerate
  def self.list_files
    cmd = "find . -type f -not -path '*/\.*'"
    rv = Open3.popen3(cmd) do |stdin, stdout, stderr, wait|
      stdin.close

      files = []
      while line = stdout.gets
        line.chomp!
        line.gsub!(/^\.\//, "")
        fields = splitquote(line)
        files.push(fields[0])
      end
      [wait, files]
    end
    if rv[0].value.exitstatus != 0 then
      raise "find . -type f -not -path '*/\.*' fails"
    end
    return rv[1]
  end

  def self.match(list, pat)
    return list.select { |fname| File.fnmatch?(pat, fname) }
  end

  def self.match_to_files(patterns, filenames)
    compiled_patterns = compile_pattern_sequence(patterns, nil)
    file_list = filenames.select do |f|
      match_pattern_sequence(compiled_patterns, f)
    end
    return file_list
  end

  # Compile a sequence of include/exclude patterns in our format to an
  # internal form accepted by match_pattern_sequence.
  #
  # In our include/exclude format, a pattern is either
  #  * a one-element map with key 'include' or 'exclude' and
  #    value a glob string, or
  #  * a glob string, taken as an exclude pattern.
  # The last pattern that matches a string governs whether it's
  # included or excluded.
  #
  # The caller should have already validated the format; we do not.
  def self.compile_pattern_sequence(patterns, prefix=nil)
    # The internal form is an array of [boolean, Regexp]
    # where the boolean indicates an include pattern.
    # The array is reversed from the external format,
    # i.e., the first match governs.
    patterns.reverse.map do |pat|
      if pat.is_a?(String)
        pat = File.join(prefix, pat) if prefix
        [true,
         self.compile_glob(pat)]
      else
        patstring = pat.values[0]
        patstring = File.join(prefix, patstring) if prefix
        [pat.keys[0] == 'include',
         self.compile_glob(patstring)]
      end
    end
  end

  # Compile a glob in our format -- POSIX plus **, minus character
  # classes due to laziness -- to a Ruby regexp.
  def self.compile_glob(glob)
    # The rule that wildcards don't expand to include a dot at the
    # start of a path component contributes most of the complexity
    # in the implementation, particularly in interaction with the
    # ** wildcard, which can begin new path components.
    re = '\A'
    i = 0
    nodot = true
    while i < glob.size
      nextnodot = false
      case glob[i]
        when '?'
          re << (nodot ? '[^/.]' : '[^/]')
        when '*'
          if i+1 < glob.size and glob[i+1] == '*'
            # **
            # Any sequence **, ***, ****, etc. is equivalent.
            while i+2 < glob.size and glob[i+2] == '*'
              i += 1
            end
            starstarre = '(?:[^/]*/+[^/.])*[^/]*/*' # Any string without '/.'
            # Proof sketch for complex REs below: consider leftmost [^/.], if any
            if i+2 < glob.size and glob[i+2] == '?'
              # **? (or ***? etc) -- nonempty, no '/.'
              i += 2
              re << (nodot ? "(?:/*[^/.]#{starstarre}|/+)"
              : "(?:[.]*/*[^/.]#{starstarre}|[.]*/+|[.]+)")
            else
              # ** (or *** etc), not followed by ? -- any string without '/.'
              i += 1
              re << (nodot ? "(?:/*[^/.]#{starstarre}|/*)"
              : starstarre)
            end
          elsif i+1 < glob.size and glob[i+1] == '?'
            # *?
            i += 1
            re << (nodot ? '[^/.][^/]*' : '[^/]+')
          else
            # plain *, not followed by ?
            re << (nodot ? '(?:[^/.][^/]*|)' : '[^/]*')
          end
        when '\\'
          case i+1 < glob.size and glob[i+1]
            when *%w(? * \\ [)
              re << "\\#{glob[i+1]}"
              i += 1
            else # including false, for end of pattern
              raise "Bad escape sequence in glob: #{glob}"
          end
        when '['
          raise "Character classes not supported, in glob: #{glob}"
        when '/'
          nextnodot = true
          re << '/'
        when /[a-zA-Z0-9]/
          re << glob[i]
        else
          re << "\\#{glob[i]}"
      end
      i += 1
      nodot = nextnodot
    end
    re << '\z'
    Regexp.new(re)
  end

  # Determine if a string is included or excluded by the given patterns,
  # which must have been compiled by compile_pattern_sequence.
  def self.match_pattern_sequence(compiled_patterns, s)
    # See the comment inside compile_pattern_sequence on the format.
    compiled_patterns.each do |incl, re|
      return incl if re.match(s)
    end
    return false
  end

  def self.splitquote(s)
    fields = []
    str = ''
    esc = false     # last character was an escape
    quote = false   # inside quotes
    s.chars do |c|
      if esc then
        case c
          when "t"
            str += "\t"
          when "n"
            str += "\n"
          else
            str += c
        end
        esc = false
        next
      elsif c == '\\' then
        esc = true
        next
      end
      if c == '"' then
        quote = !quote
      elsif !quote && c =~ /\s/ then
        if !str.empty? then
          fields.push(str)
          str = ''
        end
      else
        str += c
      end
    end
    if !str.empty? then
      fields.push(str)
    end
    return fields
  end
end

Basic

The following example finds all files that match a pattern. This is close to the logic that is run by default by Solano CI for ruby tests files.

PATTERNS = {
  "default"     => %W(test/**.rb features/**.rb),
  "default2"    => %W(test/**.rb),
}

def generate_test_files_json

  profile_name = ENV['SOLANO_PROFILE_NAME']
  puts "Using profile #{profile_name}"
  test_patterns = PATTERNS[profile_name]

  files = Enumerate.list_files
  file_list = Enumerate.match_to_files(test_patterns, files)
  file_list = file_list.uniq

  to_run = {'tests' => file_list}

  File.open("test_list.json", "w") do |f|
    f.write(JSON.pretty_generate(to_run))
  end

  puts 'Generated test_list.json'
end

generate_test_files_json

View on Github

Recently Edited

The following example finds any files that have been edited since the last passed build on a branch.

PATTERNS = {
  "default"     => %W(test/**.rb features/**.rb),
  "default2"    => %W(test/**.rb),
  "filtering"    => %W(test/**.rb spec/**.rb),
}

def generate_filters(filename)
  base = File.basename(filename, ".rb")
  return "**/" + base + "*.rb"
end

def generate_test_files_json
  profile_name = ENV['SOLANO_PROFILE_NAME']
  puts "Using profile #{profile_name}"
  test_patterns = PATTERNS[profile_name]

  files = Enumerate.list_files
  file_list = Enumerate.match_to_files(test_patterns, files)
  file_list = file_list.uniq


  #This will get the name of any files edited since the last pass on the branch
  recently_edited = `git diff --name-only #{ENV['TDDIUM_LAST_BRANCH_PASSED_COMMIT']} #{ENV['TDDIUM_CURRENT_COMMIT']}`.split("\n")
  puts "recently edited files #{recently_edited.inspect}"

  recently_edited = recently_edited.map{ |x| generate_filters(x)}
  puts "using generated filters #{recently_edited.inspect}"

  filtered_file_list = Enumerate.match_to_files(recently_edited, file_list)

  to_run = {'tests' => filtered_file_list}
  #if we do not want to run anywhing we need to send {"commands" => ["/bin/true"]}
  #in order for the build to still pass
  to_run["commands"] = %W(/bin/true) if filtered_file_list.empty?

  File.open("test_list.json", "w") do |f|
    f.write(JSON.pretty_generate(to_run))
  end

  puts 'Generated test_list.json'
end

generate_test_files_json

View on Github

Failed on last run

The following example uses Solano’s API to find the last run session of a branch and rerun tests that failed in that session. If no tests failed on the last run, all tests are run. It expects a SOLANO_API_KEY environment variable to be set.

class ApiServer

  def initialize
    @debug = true
    @host = ENV['TDDIUM_API_SERVER'] || 'ci.solanolabs.com'
    #not set by defaul
    @user_api_key = ENV['SOLANO_API_KEY']
  end

  def curl_json_user_key(method, data, url)
    opts ="-k -#"
    cmd = "curl #{opts} -X #{method} -H 'Content-type: application/json' -H 'X-Tddium-Api-Key: #{@user_api_key }' -d '#{data.to_json}' https://#{@host}/#{url}"
    puts cmd if @debug
    res = `#{cmd}`
    puts res if @debug
    JSON.parse(res)
  end
end

PATTERNS = {
  "default"     => %W(test/**.rb features/**.rb),
  "default2"    => %W(test/**.rb),
  "rerun"       => %W(test/**.rb features/**.rb),
}

def generate_test_files_json
  api = ApiServer.new()
  profile_name = ENV['SOLANO_PROFILE_NAME']
  puts "Using profile #{profile_name}"
  test_patterns = PATTERNS[profile_name]

  files = Enumerate.list_files
  file_list = Enumerate.match_to_files(test_patterns, files)
  file_list = file_list.uniq

  if profile_name == 'rerun' then
    suite_id = api.curl_json_user_key("GET", {}, "1/sessions/#{ENV['TDDIUM_SESSION_ID']}")['session']['suite_id']
    last_session = api.curl_json_user_key("GET", {"suite_id" => suite_id, "active" => false, "limit" => 1}, "1/sessions")['sessions'].first
    test_run_info = api.curl_json_user_key("GET", {}, "1/sessions/#{last_session['id']}/test_executions/query")['session']

    #if there where tests failures rerun those tests
    if test_run_info['session_status'] == 'failed' then
      tests = test_run_info['tests']
      tests.delete_if{ |x| x['status']=='passed'}
      failed_tests = tests.map{ |x| x['test_name']}
      file_list.delete_if{ |x| !failed_tests.include?(x)}
    end
  end

  to_run = {'tests' => file_list}

  File.open("test_list.json", "w") do |f|
    f.write(JSON.pretty_generate(to_run))
  end

  puts 'Generated test_list.json'
end

generate_test_files_json

View on Github