Skip to content

Python Test Frameworks

Support Ukraine Badge

A powerful pytest plugin that integrates your tests with testomat.io platform for test management, reporting and analytics

  • ✅ Sync tests with testomat.io
  • 📊 Real-time test execution reporting
  • 🏷️ Test labeling and categorization
  • 📁 Test run grouping and environments
  • 📎 Artifact management with S3 integration
  • 🔍 Advanced filtering and debugging

Prerequisites:

  • Python 3.10+
  • Pytest 6.2.5+
  • Active testomat.io account

Install via pip:

Terminal window
pip install pytestomatio
  1. Login to testomat.io
  2. Create project or go to existing project
  3. Click on “Import Tests from Source Code”
  4. Copy your project token(starts with “tstmt_“)

Synchronize tests to testomat.io:

Terminal window
TESTOMATIO=your_token pytest --testomatio sync

Execute tests and send results to testomat.io:

Terminal window
TESTOMATIO=your_token pytest --testomatio report

To make the experience more consistent, it uses standard pytest markers.
testomat.io test id is a string value that starts with @T and has 8 symbols after.

from pytest import mark
@mark.testomatio('@T96c700e6')
def test_example():
assert 2 + 2 == 4

Synchronize tests to testomat.io and get back test id.

Terminal window
pytest --testomatio sync

Clarification:

  • tests will not be executed
  • tests can be synced even without @pytest.mark.testomatio('@T96c700e6') decorator.
  • test title in testomat.io == test name in pytest
  • test suit title in testomat.io == test file name in pytest

Remove all test ids from source code. Tests will not be executed

Terminal window
pytest --testomatio remove

Run pytest and send test results into testomat.io.
Test can be sent to testomat.io without ids in your test code. If testomat.io failed to match tests by title, it will create new tests for the run

Terminal window
pytest --testomatio report

Run pytest with debug parameter to get test data collected in metadata.json file

Terminal window
pytest --testomatio debug

Create empty run and obtain its RUN_ID from testomat.io.

Terminal window
pytest --testomatio launch

Finish running or scheduled run on testomat.io. TESTOMATIO_RUN_ID environment variable is required.

Terminal window
TESTOMATIO_RUN_ID=***run_id*** pytest --testomatio finish

To configure test environment, you can use additional option testRunEnv. The configured environment will be added to the test report. Use it with report command:

Terminal window
pytest --testomatio report --testRunEnv "windows11,chrome,1920x1080"

Environment values are comma separated, please use double quotation.

To automatically clean empty test suites on testomat.io you can use no-empty option. Use it with sync command:

Terminal window
pytest --testomatio sync --no-empty

If a test from a previous import was not found on next import it is marked as “detached”. This is done to ensure that deleted tests are not staying in Testomatio while deleted in codebase. To disable this behaviour and don’t mark anything as detached on import use no-detach option. Use it with sync command:

Terminal window
pytest --testomatio sync --no-detach

By default, when importing tests, testomat.io does not preserve original file structure. Use option keep-structure with sync command to keep original file structure:

Terminal window
pytest --testomatio sync --keep-structure

By default, tests are imported into the root of the project. You can use directory option to specify directory to use for test file structure. Use this option with sync command:

Terminal window
pytest --testomatio sync --directory imported_tests

Note: keep-structure option takes precedence over directory option. If both are used keep-structure will be used.

You can filter tests by testomat.io id, using test-id option. You pass single or multiple ids to this option. Use this option with report command:

Terminal window
pytest --testomatio report --test-id "Tc0880217|Tfd1c595c"

Note: Test id should be started from letter “T”

You can use environment variable to control certain features of testomat.io

Env variableWhat it doesExamples
TESTOMATIOProvides token for pytestomatio to access and push data to testomat.io. Required for sync and report commandsTESTOMATIO=tstmt_***** pytest —testomatio sync
TESTOMATIO_SYNC_LABELSAssign labels to a test case when you synchronise test from code with testomat.io. Labels must exist in project and their scope must be enabled for testsTESTOMATIO_SYNC_LABELS=“number:1,list:one,standalone” pytest —testomatio report
TESTOMATIO_CODE_STYLECode parsing style for test synchronization. If you are not sure, don’t set this variable. Default value is ‘default’TESTOMATIO_CODE_STYLE=pep8 pytest —testomatio sync
TESTOMATIO_CI_DOWNSTREAMIf set, pytestomatio will not set or update build url for a test run. This is useful in scenarios where build url is already set in the test run by Testomat.io for test runs that a created directly on Testomat.io.TESTOMATIO_CI_DOWNSTREAM=true pytest —testomatio report
TESTOMATIO_URLCustomize testomat.io urlTESTOMATIO_URL=https://custom.com/ pytest —testomatio report
BUILD_URLOverrides build url run testsBUILD_URL=http://custom.com/ pytest —testomatio report
Env variableWhat it doesExamples
TESTOMATIO_TITLEName of a test run to create on testomat.ioTESTOMATIO_TITLE=“Nightly Smoke Tests” pytest —testomatio report
TESTOMATIO_RUN_IDId of existing test run to use for sending test results toTESTOMATIO_RUN_ID=98dfas0 pytest —testomatio report
TESTOMATIO_RUNGROUP_TITLECreate a group (folder) for a test run. If group already exists, attach test run to itTESTOMATIO_RUNGROUP_TITLE=“Release 2.0” pytest —testomatio report
TESTOMATIO_ENVAssign environment to a test run, env variant of testRunEnv option. Has a lower precedence than testRunEnv option.TESTOMATIO_ENV=“linux,chrome,1920x1080” pytest —testomatio report
TESTOMATIO_LABELAssign labels to a test run. Labels must exist in project and their scope must be enabled for runsTESTOMATIO_LABEL=“smoke,regression” pytest —testomatio report
TESTOMATIO_UPDATE_CODESend code of your test to Testomat.io on each run. If not enabled(default) assumes the code is pushed using sync commandTESTOMATIO_UPDATE_CODE=True pytest —testomatio report
TESTOMATIO_EXCLUDE_SKIPPEDExclude skipped tests from the reportTESTOMATIO_EXCLUDE_SKIPPED=1 pytest —testomatio report
TESTOMATIO_PUBLISHPublish run after reporting and provide a public URLTESTOMATIO_PUBLISH=true pytest —testomatio report
TESTOMATIO_PROCEEDDo not finalize the runTESTOMATIO_PROCEED=1 pytest —testomatio report
TESTOMATIO_SHARED_RUNReport parallel execution to the same run matching it by title. If the run was created more than 20 minutes ago, a new run will be created instead.TESTOMATIO_TITLE=“Run1” TESTOMATIO_SHARED_RUN=1 pytest —testomatio report
TESTOMATIO_SHARED_RUN_TIMEOUTChanges timeout of shared run. After timeout, shared run won`t accept other runs with same name, and new runs will be created. Timeout is set in minutes, default is 20 minutes.TESTOMATIO_TITLE=“Run1” TESTOMATIO_SHARED_RUN=1 TESTOMATIO_SHARED_RUN_TIMEOUT=10 pytest —testomatio report
Env variableDescription
S3_REGIONYour S3 region
S3_ACCESS_KEY_IDYour S3 access key ID
S3_SECRET_ACCESS_KEYYour S3 secret access key
S3_BUCKETYour S3 bucket name
S3_ENDPOINTYour S3 endpoint
S3_BUCKET_PATHPath to your bucket
TESTOMATIO_PRIVATE_ARTIFACTSStore artifacts in a bucket privately

In case you are using private testomat.io service, create pytest.ini file in your project root directory. Specify testomat.io url in it

[pytest]
testomatio_url = https://app.testomat.io

testomat.io does not store any screenshots, logs or other artifacts.

In order to manage them it is advised to use S3 Buckets (GCP Storage). https://docs.testomat.io/usage/test-artifacts/

Analyser needs to be aware of the cloud storage credentials. There are two options:

  1. Enable Share credentials with testomat.io Reporter option in testomat.io Settings -> Artifacts.
  2. Use environment variables ACCESS_KEY_ID, SECRET_ACCESS_KEY, ENDPOINT, BUCKET, BUCKET_PATH, REGION

You would need to decide when you want to upload your test artifacts to cloud storage

  1. Upload page screenshot when test fails, using fixtures reference
# content of conftest.py
import pytest
import random
import os
from typing import Dict
from pytest import StashKey, CollectReport
from playwright.sync_api import Page
phase_report_key = StashKey[Dict[str, CollectReport]]()
@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
rep = yield
item.stash.setdefault(phase_report_key, {})[rep.when] = rep
return rep
@pytest.fixture(scope="function")
def handle_artifacts(page: Page, request):
yield
report = request.node.stash[phase_report_key]
if ("call" not in report) or report["setup"].failed or report["call"].failed:
random_string = ''.join(random.choices(string.ascii_letters + string.digits, k=8))
filename = f"{random_string}.png"
screenshot_path = os.path.join(artifacts_dir, filename)
page.screenshot(path=screenshot_path)
# file_path - required, path to file to be uploaded
# file_bytes - required, bytes of the file to be uploaded
# key - required, file name in the s3 bucket
# bucket_name - optional,name of the bucket to upload file to. Default value is taken from testomat.io
artifact_url = pytest.testomatio.upload_file(screenshot_path, filename)
# or
# artifact_url = pytest.testomatio.upload_file_object(file_bytes, key, bucket_name)
pytest.testomatio.add_artifacts(request.node, [artifact_url])
page.close()

⚠️ Please take into account s3_connector available only after pytest_collection_modifyitems() hook is executed.

  1. If you prefer to use pytest hooks - add pytest_runtest_makereport hook in your conftest.py file.
def pytest_runtest_makereport(item, call):
artifact_url = pytest.testomatio.upload_file(screenshot_path, filename)
pytest.testomatio.add_artifacts([artifact_url])

The plugin supports reporting the same test multiple times in a single run. This is especially useful for Cross-Platform testing, when you run the same test on different environments. To use this feature you need to specify test run environment through TESTOMATIO_ENV environment variable or by using —testRunEnv option. Example:

Terminal window
TESTOMATIO=***api_key*** TESTOMATIO_RUN_ID=***run_id*** pytest --testomatio report --testRunEnv "os:ubuntu, integration"
TESTOMATIO=***api_key*** TESTOMATIO_RUN_ID=***run_id*** pytest --testomatio report --testRunEnv "os:windowns, integration"

Executing these commands will include the tests in the same run, but as separate instances. Each test will contain metadata with information about the test run environment.

Note: Only key:value envs will be passed into tests metadata

Use python 3.12

  • Run unit tests
  • import into empty project
  • updated test - (resync)
  • test run
  • test run into a folder
  • test run labels, tags
  1. pip install ".[dev]"
  2. python ./smoke.py
  3. Test things manually
  4. Verify no regression bugs
  5. cz commit --prerelease beta
  6. cz bump
  7. git push remoteName branchName --tags