cli

The Testomat.io Reporter CLI is a powerful tool for managing test runs, parsing XML reports, and uploading artifacts. CLI can be used to start and finish test runs, run tests, parse XML reports, and upload artifacts. It can be used in CI/CD pipelines or locally.

Reporter is designed to work with Testomat.io service but not exclusively

Installation

npm install -g @testomatio/reporter

General Usage

npx @testomatio/reporter <command> [options]

Available Commands

1. start

Starts a new test run and returns its ID. This requires an API key to be set in the TESTOMATIO environment variable.

Usage:

npx @testomatio/reporter start [options]

Environment Variables:

TESTOMATIO: Your Testomat.io API project key in format: tstmt_* (required).

Options:

--env-file <envfile>: Load environment variables from a specific env file. If none specified, it will look for .env file.

Previously known as: npx start-test-run --launch (before 1.6.0)

2. finish

Finishes a run by its ID.

Usage:

npx @testomatio/reporter finish [options]

Environment Variables:

TESTOMATIO: Your Testomat.io API key in format: tstmt_* (required).

Options:

--env-file <envfile>: Load environment variables from a specific env file. If none specified, it will look for .env file.

Previously known as: npx start-test-run --finish (before 1.6.0)

3. run

Creates a run, and runs tests with the specified command, then finishes the run. This command ensures that if test runner spawns parallel workers or processes and reports it will report to the same run.

Use npx @testomatio/reporter run if multiple run reports are created on the same launch

Usage:

npx @testomatio/reporter run [options]

Environment Variables:

TESTOMATIO: Your Testomat.io API key in format: tstmt_* (required).

Options:

-c, --command <cmd>: Test runner command (required).
--filter <filter>: Filter executed tests by tag, label, jira, plan.
--env-file <envfile>: Load environment variables from a specific env file.

Examples:

npx @testomatio/reporter run "npm test"
npx @testomatio/reporter run "npx jest" --filter "testomatio:tag=frontend"
npx @testomatio/reporter run "npx jest" --filter "testomatio:label=Smoke"
npx @testomatio/reporter run "npx jest" --filter "testomatio:jira=TC-123"
npx @testomatio/reporter run "npx jest" --filter "testomatio:plan=a123fb12"
npx @testomatio/reporter run "mocha tests/" --env-file .env.test

Previously known as: npx start-test-run -c "command" (before 1.6.0)

4. xml

Parses XML reports in JUnit (NUnit, xUnit, TRX) XML format and uploads them to Testomat.io or uses GitHub / GitLab / ButBucket pipe to create a mini-report

Usage:

npx @testomatio/reporter xml <pattern> [options]

Environment Variables:

TESTOMATIO: Your Testomat.io API key in format: tstmt_* if you want to upload reports to Testomat.io (optional).

Arguments:

<pattern>: XML file pattern (required).

Options:

-d, --dir <dir>: Project directory.
--java-tests [java-path]: Load Java tests from path (default: src/test/java).
--lang <lang>: Language used (python, ruby, java).
--timelimit <time>: Default time limit in seconds to kill a stuck process.
--env-file <envfile>: Load environment variables from a specific env file.

Examples:

npx @testomatio/reporter xml "test-results/**.xml" --lang python
npx @testomatio/reporter xml "junit-reports/**.xml" -d ./project --lang java
npx @testomatio/reporter xml "pytest-results.xml" --timelimit 300 --env-file .env.test

Previously known as: npx report-xml (before 1.6.0)

5. upload-artifacts

Testomat.io reporter automatically uploads artifacts during run. However, either some artifacts failed to upload or you intentioanlly disabled file upload during tests to speed up reporting. In this case you can use this command to upload artifacts after the run.

It is important to have the TESTOMATIO_RUN environment variable set to the run ID.

Usage:

npx @testomatio/reporter upload-artifacts [options]

Environment Variables:

TESTOMATIO: Your Testomat.io API key (required).
TESTOMATIO_RUN: The previous run ID you want to upload artifacts (optional). If none set, latest run will be used.

Options:

--force: Re-upload artifacts even if they were uploaded before.
--env-file <envfile>: Load environment variables from a specific env file.

You still need S3 artifacts configuration to be set to upload artifacts to storage. In order to disable artifacts upload during tests you can use TESTOMATIO_DISABLE_ARTIFACTS=1 while running tests.

Examples:

npx @testomatio/reporter upload-artifacts
npx @testomatio/reporter upload-artifacts --force
npx @testomatio/reporter upload-artifacts --env-file .env.prod

With Playwright:

TESTOMATIO=tstmt_* TESTOMATIO_DISABLE_ARTIFACTS=1 npx playwright test
TESTOMATIO=tstmt_* npx @testomatio/reporter upload-artifacts

With webdriverio:

TESTOMATIO=tstmt_* TESTOMATIO_DISABLE_ARTIFACTS=1 npx @testomatio/reporter run "npx wdio"
TESTOMATIO=tstmt_* npx @testomatio/reporter upload-artifacts

You can also upload small artifacts during the run, while large files can be uploaded after. For instance, all files uploading larger than 10MB will be skipped during the run.

TESTOMATIO=tstmt_* TESTOMATIO_ARTIFACT_MAX_SIZE_MB=10 npx playwright test
TESTOMATIO=tstmt_* npx @testomatio/reporter upload-artifacts

However, upload-artifacts command will upload all files after the run, without blocking the final result.

6. replay

The replay command allows you to re-send test data from debug files to Testomat.io. This is useful when your original test run failed to upload results properly.

Important: To make replay work, tests should be executed with DEBUG=1 variable set, to ensure they are running in debug mode and save data into a file.

Usage:

npx @testomatio/reporter replay [debug-file] [options]

Arguments:

debug-file (optional) - Path to debug file. Defaults to latest created debug file, i.e. /tmp/testomatio.debug.latest.json

Options:

--dry-run - Preview the data without sending to Testomat.io
--env-file <envfile> - Load environment variables from env file

Examples:

TESTOMATIO=<your-api-key> DEBUG=1 npx playwright test


TESTOMATIO=<your-api-key> npx @testomatio/reporter replay


TESTOMATIO=<your-api-key> npx @testomatio/reporter replay /path/to/debug.json


TESTOMATIO=<your-api-key> npx @testomatio/reporter replay --dry-run


npx @testomatio/reporter replay --env-file .env.staging

How it works:

The replay command uses the ReplayService class (located in src/replay.js) to:

Parse the debug file line by line
Extract environment variables, run parameters, test data, and finish parameters
Restore environment variables (without overriding existing ones)
Create a new test run using the TestomatClient
Send each test result individually
Update the run status when complete

For more details about debug files, see the Debug Pipe documentation.

Environment Variables

Many commands rely on environment variables. You can set these in a command line, in a .env file, or use the --env-file option to specify a custom env file. Important variables include:

TESTOMATIO: Your Testomat.io API key.
TESTOMATIO_RUN: The current run ID (usually set automatically by the start command).
TESTOMATIO_TITLE: Title for the test run (optional).
more..

Tips

Always ensure your Testomat.io API key is properly set before running commands.
Use the --env-file option when you have different configurations for various environments.
The run command is versatile and can be used with various test runners like Jest, Mocha, or custom scripts.
When using the xml command, ensure your XML reports are in a supported format.
The upload-artifacts command is useful for adding additional files or logs to your test runs.

For more detailed information or troubleshooting, please refer to the Testomat.io documentation or contact support.