API Reference

Reference documentation for TestLab's block-based test authoring system.

Architecture Overview

A standard TCK test suite has three block tiers plus one configuration layer:

graph TB
    subgraph TCK["Standard TCK"]
        subgraph Tests["Tests"]
            direction TB
            V["🔴 Validation Blocks<br/>validate/assert · validate/field<br/>validate/object · validate/schema"]
            M["🔵 Main Blocks<br/>connector/ · dtr/ · mock/"]
            O["🟢 Operator Blocks<br/>util/ · http/ · flow/ · filter_expression"]
        end
        C["🟠 Environment Configuration<br/>(Infrastructure, Services, Variables, Schemas)"]
    end

    M -->|"returns:"| V
    O -->|"supports"| M
    C -->|"provides env"| Tests

    style V fill:#8B1A4A,color:#fff
    style M fill:#0EA5E9,color:#fff
    style O fill:#16A34A,color:#fff
    style C fill:#F97316,color:#fff

Three Tiers

Main Blocks (blue) are domain-specific. They understand Tractus-X protocols (DSP, EDC Management API, AAS). They interact with connectors, registries, and mock services.
Operator Blocks (green) are generic utilities. They handle HTTP, JSON extraction, flow control, and filtering. They support main blocks without knowing about Tractus-X.
Validation Blocks (red) check results. They receive return variables from steps and assert conditions. They never execute actions — only verify.
Environment Configuration (orange) is defined in the TCK manifest. Services, variables, and schemas provide the runtime context for all tests.

Data flows: Main blocks returns: values → Validation blocks check them. Operator blocks support main blocks (HTTP calls, retries, UUID generation). Environment config provides services and variables to all blocks via ${{ env.x }} interpolation.

Design Rationale

The TestLab YAML syntax draws inspiration from established CI/CD and infrastructure-as-code tools:

Inspiration	What We Took
GitHub Actions	`uses:` / `with:` pattern, namespaced identifiers, composable steps
Bruno / Newman	Variable scoping model: `env.` vs `vars.`, secret masking
Terraform / OpenTofu	Services declared once, referenced by name — same test logic across environments
pytest / JUnit	Setup/teardown model: setup → test → implicit teardown

Decision	Rationale
`uses:` not `type:`	`uses:` is a verb — "this step uses this capability"
Namespaces (`connector/provider/create_asset`)	Flat names don't scale across 27+ blocks. Reads like a REST path.
`with:` not positional args	Named parameters are self-documenting
`returns:` not `outputs:`	Functions "return" values. "Output" is overloaded.
Separate Main / Operators / Validators	Three distinct concerns: DO things, CONFIGURE behavior, CHECK results

For detailed rationale: ADR-0010: YAML Syntax v2

Sections

Section	Description
Steps	Every step per category and module — inputs, outputs, the validation kinds and operators. Generated from the step contracts
TCK Syntax	Manifest, test, phase and step syntax the steps are written in

CLI Reference

Command	Description
`testlab compile <source>`	Compile a TCK source directory into a `.tck` package
`testlab run <target>`	Execute a TCK against a live dataspace. A `.tck` package runs as given; a manifest is compiled into a throwaway package first, so nothing executes that has not compiled
`testlab validate <package>`	Validate a compiled TCK package without executing steps
`testlab inspect <package>`	Report what a package contains — tests, manifest, variables, infrastructure — without running it

`testlab inspect`

Inspects a compiled .tck package and prints its static metadata without executing any steps against a live environment.

Text Only

testlab inspect <package> [--player-keys <path>] [--compiler-pub <path>]
                         [--variables] [--infrastructure] [--manifest]
                         [--checkpoints [--skip-tests <a.yaml,b.yaml>]]
                         [--extract <dir>] [--json]

Option	Description
`<package>`	Path to a `.tck` (plain or encrypted) file
`--player-keys`	Directory holding the player identity — required if the package is encrypted
`--compiler-pub`	The compiler's signing public key — required if the package is signed
`--variables`	Also print the variable list (ID, source, scope, type) declared in the TCK
`--infrastructure`	Also print the infrastructure requirements (capability, side, required, standard) declared in the TCK
`--manifest`	Also print the manifest: identity, checksum, signer, authorized players
`--checkpoints`	Also print the checkpoints a run meets, in run order (Checkpoints); with `--json`, a `checkpoints` list of entries
`--skip-tests`	With `--checkpoints`: the tests the run skips, comma-separated, as `skip_tests` names them
`--extract <dir>`	Write the package's verified contents to a directory
`--json`	Emit one JSON object keyed by section instead of the tables

Default output (human-readable table):

Text Only

TCK: Certificate Management Conformity
  Total Steps       : 12
  Total Validations : 8

  Test: request-certificate  |  ID: request_certificate.yaml  |  Skippable: No
  ┌────────────────────────────────────────────────┬────────────────────────────────────────────────┬───────────┬─────────────┐
  │ Step Name                                      │ Uses                                           │ Phase     │ Validations │
  ├────────────────────────────────────────────────┼────────────────────────────────────────────────┼───────────┼─────────────┤
  │ Request certificate                            │ connector/consumer/request_certificate         │ Execution │ 2           │
  └────────────────────────────────────────────────┴────────────────────────────────────────────────┴───────────┴─────────────┘

  Test: catalog_policy_validation  |  ID: catalog_policy_validation.yaml  |  Skippable: Yes
  ┌────────────────────────────────────────────────┬────────────────────────────────────────────────┬───────────┬─────────────┐
  │ Step Name                                      │ Uses                                           │ Phase     │ Validations │
  ├────────────────────────────────────────────────┼────────────────────────────────────────────────┼───────────┼─────────────┤
  │ Validate catalog policy                        │ validate/assert                                │ Execution │ 3           │
  └────────────────────────────────────────────────┴────────────────────────────────────────────────┴───────────┴─────────────┘

JSON output (--json flag) — returns an envelope with the requested sections:

JSON

{
  "inspection": {
    "name": "Certificate Management Conformity",
    "total_steps": 12,
    "total_validations": 8,
    "tests": [
      {
        "name": "request-certificate",
        "test_id": "request_certificate.yaml",
        "skippable": false,
        "steps": [
          {
            "step_name": "Request certificate",
            "uses": "connector/consumer/request_certificate",
            "phase": "EXECUTION",
            "validation_count": 2
          }
        ]
      },
      {
        "name": "catalog_policy_validation",
        "test_id": "catalog_policy_validation.yaml",
        "skippable": true,
        "steps": [
          {
            "step_name": "Validate catalog policy",
            "uses": "validate/assert",
            "phase": "EXECUTION",
            "validation_count": 3
          }
        ]
      }
    ]
  },
  "variables": [
    { "id": "provider_bpn", "source": "input", "scope": "sut",    "type": "string" },
    { "id": "testlab_management_url", "source": "input", "scope": "engine", "type": "string" },
    { "id": "certificate_type",  "source": "value", "scope": null, "type": "string" }
  ],
  "infrastructure": {
    "engine": { "connector": { "required": true, "standard": null } },
    "sut":    { "connector": { "required": true, "standard": null } }
  }
}

variables and infrastructure keys are only populated when their respective flags (--variables, --infrastructure) are passed; otherwise their values are null.

The inspection key (step/validation counts) is always populated. Import the result models directly from the library:

Python

from tractusx_testlab.models import (
    TckInspectionResult, TestInspection, StepMeta,   # inspection
    VariableDefinition, VariableScope, VariableSource,  # variables
    InfrastructureConfig, CapabilityRequirement,        # infrastructure
    TestStatus, SkipNotAllowedError,                  # skip configuration
)

`testlab run` — Checkpoints

A run that needs someone to act — your system calling the Test Suite, the Test Suite calling your system, you choosing an option — lists its checkpoints before anything starts, in a window, and asks whether to go on. A run without checkpoints asks nothing.

Text Only

================================================================================
  THIS RUN NEEDS YOU AT 2 CHECKPOINTS
--------------------------------------------------------------------------------
  Push the certificate
    1  await_push   inbound · your system calls us
       1m 00s to answer · Push the certificate to the Test Suite
  Choose the offer
    2  pick_offer   internal · you choose
       5m 00s to answer · only on some paths · may repeat · Pick the offer
================================================================================
Continue? [y/N]: y
Approved.

The answer defaults to no: an empty answer, or anything but y/yes, stops the run before any test. When a step asks you to choose, the run shows the options in a window of their own:

Text Only

================================================================================
  CHECKPOINT 2 of 2 · YOU CHOOSE                                    4m 52s left
--------------------------------------------------------------------------------
  Pick the offer

     1  CCMAPI notification 3.0 · asset urn:uuid:7f3c… · Membership, FrameworkAgreement
     2  CCMAPI notification 3.0 · asset urn:uuid:a91e… · Membership
================================================================================
Choose [1-2] (4m 52s left, q to stop): 3
Type a number from 1 to 2, or q to stop.
Choose [1-2] (4m 41s left, q to stop): 2
Chosen: 2 CCMAPI notification 3.0

A number chooses its option; anything else asks again. q stops the run: the current test's teardown runs, no further test starts. When the step's timeout runs out, the checkpoint times out. Ctrl-C at either prompt stops the run at once; the test it interrupted ends CANCELLED (in the trace tck.test.failed with status: "CANCELLED"), and the run fails. While a window is up the run's own log lines wait, and follow it when it closes. The windows are drawn at the terminal's width (60 to 100 columns), in plain ASCII; the title is bold in a coloured terminal, never with NO_COLOR. The transcript keeps them, with what you typed.

To run without a terminal, answer up front:

Option	Effect
`--yes`, `-y` (or `TESTLAB_YES=1`)	Approve the checkpoints without asking. It never answers a choice.
`--choose <step_id>=<option id\\|n>`	Answer the choice of step `<step_id>` with the option whose id is the value, else with option `n` (from 1). The id wins: with options `5` and `1`, `pick=1` chooses option `1`. Repeatable.

Without a terminal and without the answer, the run does not wait: it stops, naming the flag to pass.

Exit code	Meaning
`0`	The run passed.
`1`	The run failed, or could not start.
`2`	The run could not get an answer: no terminal and no `--yes` or `--choose`, the input ended (Ctrl-D) at a prompt, a `--choose` value that names no option, or the operator could not be asked.
`3`	The checkpoints were not approved; no test ran.
`4`	The run was stopped at a choice (`q`).

The engine and any other host using the Python API are never asked: they show the checkpoints in their own UI and answer a choice on JobManager.selections. tractusx_testlab.player.operator.run_with_operator is how a host with a person at hand takes part instead.

Bash

testlab run index.yaml --yes --choose pick_offer=2

`testlab run` — Skipping Optional Tests

Tests marked skippable: true in the TCK manifest can be bypassed at runtime via the skip_tests runtime variable. Skipped tests produce a SKIPPED result and are not counted as failures.

Single test:

Bash

testlab run index.yaml \
  --var skip_tests=catalog_policy_validation.yaml \
  --config your-env.yaml

Multiple tests — use a config YAML (repeating --var overwrites the previous value):

YAML

# skip.yaml
skip_tests:
  - catalog_policy_validation.yaml
  - error_handling.yaml

Bash

testlab run index.yaml --config skip.yaml

Error handling — validation runs before any test executes. Requesting a skip on an unknown or non-skippable test raises SkipNotAllowedError immediately:

Text Only

Error: Cannot skip test(s) 'request_certificate.yaml': not marked skippable.
Set skippable: true on the test entry in the TCK manifest to allow skipping.

Author opt-in required

A test can only be skipped at runtime when the TCK author has set skippable: true on that test entry. Tests without this flag cannot be bypassed, protecting mandatory conformance checks.

For the architectural rationale see ADR-0024: Test-Level Skip Configuration.

Provider resource namespaces

TestlabPlayer(resource_prefix="testlab:") gives every provider asset, policy, and contract definition the ID <resource_prefix><run-id>:<local-id>. The default prefix is testlab:. Applications that share a connector should supply their own prefix when constructing the player:

Python

import asyncio
from pathlib import Path

from tractusx_testlab.player.execution.player import TestlabPlayer


async def main():
    player = TestlabPlayer(resource_prefix="cx-test-suite:")
    return await player.run(Path("assessment.tck"), job_id="run-123")


result = asyncio.run(main())

The job ID is the run ID. An authored asset ID asset becomes cx-test-suite:run-123:asset; omitted IDs are generated inside the same namespace. Creation steps return the actual ID, and the dataplane listener announces it. Use creation outputs for subsequent references and teardown. Already scoped IDs from this run stay unchanged. Contract definitions constrain their selectors to this run's namespace, including selectors based on asset type.

${{ execution.id }} and ${{ execution.resource_prefix }} are immutable run metadata. Use the latter when an asset ID must be embedded in a notification or mock response before creation, for example ${{ execution.resource_prefix }}certificate-asset. Raw HTTP requests retain their authored bodies; authors must use this metadata when provisioning EDC resources directly through HTTP. Raw provisioning calls to the engine management API are refused if their IDs, policy references, or contract selectors escape this namespace.

For raw HTTP creation, declare returns: { response_body: { type: object } } and read its @id with util/json_path_extract (path: "@id"). Publish the extracted value and use it unchanged for policy links and cleanup, just as typed creation outputs are used for filters and assertions. Do not reconstruct IDs from the prefix downstream; the prefix belongs in creation payloads and namespace selectors.

Cleanup steps delete the exact resource IDs supplied to them. The host application must run connector cleanup after completion, failure, and cancellation, including when execution exits before a YAML teardown. Immutable agreement history and assets pinned by that history may remain in EDC after offers are withdrawn.

API Reference

Architecture Overview

Three Tiers

Design Rationale

Sections

CLI Reference

testlab inspect

testlab run — Checkpoints

testlab run — Skipping Optional Tests

Provider resource namespaces

`testlab inspect`

`testlab run` — Checkpoints

`testlab run` — Skipping Optional Tests