# Continious Integration

Testomatio allows executing tests on CI from its interface. A single test, suite, test plan, or all tests can be executed automatically on CI.

Currently following CI systems supported:

  • Jenkins
  • Bamboo
  • GitHub Actions
  • GitLab
  • Circle CI

# Overview

Testomatio uses REST API to trigger jobs on external CI systems. IDs of tests or suites can be passed to the job so only a specific test or suite will be executed. The test runner greps all tests by their IDs and executes a subset of tests. Then a report is sent back to Testomatio via reporter.

image

Connecting CI server to Testomatio consist of the following steps:

  1. Create Testomatio Project
  2. Import automated tests into that project from a repository
  3. Assign Ids for imported tests source code
  4. Create a new job in CI according to instructions on this page
  5. Connect CI server to Testomatio
  6. Run your tests or suites to get reports!

# Assigning IDs

To execute a specific test or a suite a test runner should have a way to find a test by its unique name. For this reason, Testomatio IDs can be used. If tests in the source code will have Testomatio IDs it will be very simple to filter tests. We provide a semi-automatic way to assign Testomatio IDs to tests in source code.

For JavaScript frameworks use the same check-tests command you used for importing tests with --update-ids. The tests must be already imported in Testomatio:

TESTOMATIO={apiKey} npx check-tests <framework> <pattern> --update-ids 

For Cucumber tests use check-cucumber command with the similar --update-ids command. The command should be the same as for importing plus --update-ids option:

TESTOMATIO={apiKey} npx check-cucumber <pattern> --update-ids 

This command will update your source code. Please check the changes before committing it. If the Testomatio IDs were placed correctly you can commit your changes to repository.

From now on, Testomatio can use Test IDs to run exact tests and suites on Continuous Integration servers.

# Configuring CI

CI configuration has 3 steps:

  • establishing a connection with CI
  • setting up required input variables
  • setting up custom configuration variables

Follow the guide for a corresponding CI to set it up.

# Jenkins

To connect Jenkins to Testomatio you will need a user and an API Token created. API token can be added on "Configure" page of current user:

image

Then create a new Jenkins job. Select "Freestyle project".

image

It is recommended to avoid spaces in job name to prevent issues with connecting to this job via URL

Make this build parametrized:

image

Add the following parameters as a string with empty default values:

  • run
  • testomatio
  • grep

If you use on-premise Testomatio setup you will also need to add testomatio_url parameter.

Inside "Build Triggers" select "Trigger build remotely"

image

Proceed with configuring the job and set all required parameters like SCM and build steps.

image

Within a step pass in configured parameters as environment variables into the test runner. Let's take CodeceptJS command as an example:

TESTOMATIO_RUN=${run} TESTOMATIO=${testomatio} npx codeceptjs run --grep "${grep}"

Prepend TESTOMATIO_URL=${testomatio_url} if you use on-premise versoin

image

Save the build and switch to Testomat.io.

Open Settings and Connect a new CI:

image

Select "Jenkins" and fill in all required fields:

image

  • Jenkins Hostname - URL of Jenkins host
  • Username - a user on Jenkins which will trigger builds
  • API Token - a token we created previously in the user's settings.
  • Job Name - the name of a job we just created

Switch to Input variables tab and enable variables that was configured for parametrized builds:

image

Select testomatio_url if you use on-premise version.

Click "Save" and check the connection.

Now you can run a test or a group of tests via Jenkins CI. For a custom configuration read about Environment Variables

# Bamboo

Let's configure Bamboo CI server to work for CodeceptJS framework:

  1. Create a plan for testomatio to run in testomatio

Configure

  1. Note the plan key. In this case its "TES"

  2. Configure the job to install node dependencies install

  3. Add the script to run codeceptJS tests:

TESTOMATIO_RUN=${bamboo.run} npx codeceptjs run --grep "${bamboo.grep}"

Following environment variables must be set:

  • Add TESTOMATIO environment variable with API key of Testomatio project.
  • If you are running a self-hosted Testomatio instance, add TESTOMATIO_URL variable to specify a host to which reports will be sent.

Script

  1. Set an input variable. Open Plan configuration:

image

add grep and run variables with an empty string as a default value

image

Now, configure Bamboo integration at Testomatio:

  1. Go to settings > CI and enter the details of Bamboo server. Check this (opens new window) to generate API token

CI settings

A project and plan keys can be found from URL:

image

  1. Enable run option on Input Variables tab. This allows CI to send a report to a specific Run inside Testomatio.

image

You can pass more input variables if you set them in Environment Configuration

  1. Open Runs page (or any test or suite) then select Run in CI option in extra menu.

image

Select "Bamboo" profile in a list. Optionally, configure a Run Title and select a Test Plan.

image

  1. Launch a run and wait for the results.

Runs view

# GitLab CI

  1. Add new trigger https://docs.gitlab.com/ee/ci/triggers/README.html#adding-a-new-trigger

image

  1. Create .gitlab-ci.yml file or add the job to existing one. E.g. - https://gitlab.com/TetianaKhomenko/prod-setup/-/blob/main/.gitlab-ci.yml

image It should contain next commands:

    - export TESTOMATIO="${testomatio}"
    - export TESTOMATIO_URL="${testomatio_url}"
    - export TESTOMATIO_RUN="${run}"
  1. The job should include a step where the test runner is executed with --grep option and TESTOMATIO environment variables passed in. For instance:
    - npx codeceptjs run-workers 2 --grep "${grep}"
  1. Connect a GitLab CI in Testomatio:

image

  1. Save your connection

  2. Now, open the "Configuration" tab and check the default ref value. ref is a target branch or a tag on which tests will be executed. By default, it is set to master (most of the repositories still use master as the main branch name, but we will adjust defaults accordingly when things change), but you can choose a different one, like main.

image

  1. Run and testomatio inputs are passed from Testomatio. Enable them on the Input Variables tab. You can pass more input variables if you set them in Environment Configuration

image

  1. When the connection is saved, open a test and select "Run in CI". Select a target ref and click "Launch"

  2. This will start a new pipeline in GitLab CI please check that the job was successfully triggered and completed. After the job has finished a run report will be available on Runs page of Testomatio

# GitHub Actions

  1. Create an access token on GitHub (opens new window) with access to workflow scope:

image

  1. Create a workflow in a GitHub repository. Go to "Actions" tab in repository and click "Create Workflow" button. Then you will get a workflow template. A workflow filename will be used by Testomatio to call a specific workflow.

image

  1. This workflow will be used solely by Testomatio so it should start only on workflow_dispatch event. The event should be defined with the following input parameters:
name: Testomatio Tests

on:
  workflow_dispatch:
    inputs:
      grep:
        description: 'tests to grep '
        required: false
        default: ''
      run:
        required: false
      testomatio:
        required: false
  1. The job should include a step where the test runner is executed with --grep option and TESTOMATIO environment variables passed in. For instance:
    - run: npx codeceptjs run --grep "${{ github.event.inputs.grep }}"
      env:
        TESTOMATIO: "${{ github.event.inputs.testomatio }}"
        TESTOMATIO_RUN: "${{ github.event.inputs.run }}"
  1. Connect a GitHub Actions CI in Testomatio:

image

You will need to enter the following

  • GitHub Username
  • OAuth token (created at step 1)
  • organization/repository (or user/repository)
  • workflow name, a file name with a workflow, like testomatio.yml
  1. Save your connection
  2. Now, open "Configuration" tab and check the default ref value. ref is a target branch or a tag on which a tests will be executed. By default, it is set to master (most of the repositories still use master as the main branch name, but we will adjust defaults accordingly when things change), but you can choose a different one, like main.
  3. run and testomatio inputs are passed from Testomatio. Enable them on Input Variables tab

image

You can pass more input variables if you set them in Environment Configuration

  1. When the connection is saved, open a test and select "Run in CI". Select a target ref and click "Launch"

image

  1. This will start a new job in GitHub Actions, please check that the job was successfully triggered and completed. After the job has finished a run report will be available on Runs page of Testomatio

# Circle CI

  1. Create an access token on Circle CI (opens new window) :
image
  1. Create a workflow in config.yml file in .circle folder in the root folder in your repository.

  2. This workflow will be used solely by Testomatio so it should start only on workflow_dispatch event. The event should be defined with the following input parameters:

parameters:
  testomatio: 
    type: string
    default: ""

  run: 
    type: string
    default: ""

  testomatio_url: 
    type: string
    default: ""

  grep:
    type: string
    default: ""

  1. The job should include a step where the test runner is executed with --grep option and TESTOMATIO environment variables passed in. For instance:
     - run:
         name: Run tests
         command: npx codeceptjs run --grep "<< pipeline.parameters.grep >>"
         environment:
           TESTOMATIO: << pipeline.parameters.testomatio >>
           TESTOMATIO_RUN: << pipeline.parameters.run >>
           TESTOMATIO_URL: << pipeline.parameters.testomatio_url >>

Select testomatio_url if you use on-premise version.

  1. Connect a Circle CI in Testomatio:
image

You will need to enter the following:

  1. Save your connection
  2. Now, open "Configuration" tab and check Input Variables
image
  1. When the connection is saved, open a test and select "Run in CI". Select a target ref and click "Launch"
image
  1. This will start a new pipeline in Circle CI. Please check that the job was successfully triggered and completed. After the job has finished a run report will be available on the Runs page of Testomatio.

# Input Variables

While connection settings can be different across CI settings, the list of input variables is the same.

image

For example, Testomatio sends grep variable to CI to identify which tests should be executed. It may pass other input variables which can be used on CI to improve reporting.

Here is how run input variable can be accessed on different CIs:

  • Atlassian Bamboo: ${bamboo.run}
  • GitHub Actions: ${github.event.inputs.run}

Here is the list of preconfigured input variables:

  • run - passes Run ID to CI. If this option is toggled on, when a run is created in Testomatio it is instantly added to the list of runs marked as "Scheduled". On CI run variable must be passed as TESTOMATIO_RUN environment variable to a reporter. This allows mapping a scheduled run to the run which is currently processed. If TESTOMATIO_RUN is not set, a duplicate run will be created.
  • testomatio - passes project access key to CI. This input variable must be passed as TESTOMATIO environment variable to match the Testomatio project. Toggle on this option if you prefer not to hardcode Testomatio Project ID in CI configuration but to obtain this value on launch. This may be useful if you have a different Testomatio project configured for on CI run.
  • testomatio_url - when working on a self-hosted Testomatio instance, this variable can be used to pass Testomatio endpoint to CI system. Pass testomatio_url environment variable to TESTOMATIO_URL

# Environment Configuration

Sometimes extra configuration is required for CI job. For instance, extra configuration variables can be used to specify:

  • browser
  • target branch
  • staging/production environments

Testomatio allows to predefine configuration variables and adjust them for each run. Config variables can be set in "Configuration" tab on CI connection settings.

image

Config variables should be put per line with the default value passed in with =. The format is similar to .env file format:

browsers=chrome,firefox,safari
branch=main
user=admin@admin.com
password=123456

To set a variable without a default value just pass it as on a line without =

Those variables will be available for a reconfiguration on each CI Run executed from Testomatio. If a variable value contains comma , like in example above: chrome,firefox,safari, these values will be displayed with the select box. Otherwise, a simple input will be shown:

image

These variables will be passed to CI in the same manner as grep parameter. So, CI job should be prepared to handle these config variables. For instance, if GitHub Actions are used, values are passed as inputs and can be used like this:

    - run: npx codeceptjs run --grep "${{ github.event.inputs.grep }}" --profile "${{ github.event.inputs.branch }}"
      env:
        TESTOMATIO: "${{ github.event.inputs.testomatio }}" # passed from Testomatio by default
        BROWSER: "${{ github.event.inputs.browser }}"
        TEST_USER: "${{ github.event.inputs.user }}"
        TEST_PASSWORD: "${{ github.event.inputs.password }}"