Architecture documentation (arc42)

The IRS is a:

The IRS API is being consumed by the dismantler dashboard and other parties which are part of the Catena-X network. They need to provide valid credentials issued by the Catena-X IAM. Additionally, they must provide a base global asset identifier to retrieve information for as well as configuration details for the view on that information.

The IRS retrieves data from the Catena-X network (using the necessary infrastructure, see Technical Context), aggregates it and provides it back to the consumers. This connection is mandatory. If the Catena-X services are unavailable, the IRS cannot perform any work.

As of now, the IRS uses its own IAM credentials to gather the required data. This might be changed to use the consumer credentials in the future.

The IRS REST controller is used to provide a RESTful web service.

The RecursiveJobHandler component provide the logic to build jobs recursively to retrieve items over the complete C-X network and assembles the partial results into a single item graph result.

The TransferProcessManager creates executions and provides them to the executor service. Each execution contains HTTP requests to the asset administration shell registry and to the submodel interface.

The AASTransferProcessManager coordinates creation of Runnables which are then passed to the ExecutorService for asynchronous and parallel execution. The Runnable consists of different Delegates which are each responsible for requesting certain types of data.

The ESS REST controller is used to provide a RESTful web service to related Environmental and Social Standards functionalities.

The Swagger documentation can be found in the local deployment of the reference application. More information can be found in the GitHub repository: https://github.com/eclipse-tractusx/item-relationship-service/blob/main/README.md

Since we cannot rely on synchronous responses regarding the requests of submodel endpoints, we designed the IRS in a way that it will handle the job management of requesting all needed endpoints in order to build a BoM tree.

This section describes how the IRS fetches submodel payload.

This section describes the job orchestration in IRS.

This section describes the policy store in IRS.

This section describes the way IRS is validating policies found in assets.

If a job is registered via the IRS API, it will be persisted (with its parameters) in the JobStore, where it can be retrieved by further calls and processes. Then, the starting item ID is extracted, and a new transfer process is handed to the ExecutorService, which will process it asynchronously (see Scenario 2: Job Execution).

In the meantime, the JobOrchestrator returns a response to the API caller, which contains the UUID of the new job. This UUID can then be used by the caller to retrieve information about the job via the API. The input provided by the caller determines how the job will operate (starting point, recursion depth, aspect filter, …​).

After a job has been created (see Scenario 1: Create Job), the first transfer containing the root item ID is passed to the ExecutorService. The transfer is then started asynchronously and retrieves the necessary information from the Catena-X network, first by fetching the AAS information from the DigitalTwin registry and then calling the SubmodelProviders for the submodel payload.

At least the aspect SingleLevelBomAsBuilt is required for the tree to be built. If the customer that started the job provided more aspects to be fetched, they will be retrieved here too. The result of each transfer is stored in the BlobStore.

After the transfer process has finished, any subsequent child IDs will be extracted and new transfer processes will be scheduled for those via the ExecutorService. This cycle repeats until all leafs were reached, the specified max depth has been reached, or the job was canceled externally.

As soon as all transfers are finished, the results will be combined and stored in the BlobStore again. The job itself will be marked as completed.

When a user requests job details, the IRS looks up the jobId in the persistent job store. If the job exists, it will proceed to fetch the job details and prepare a response object. Only if the job is in state "COMPLETED" or if the caller has set the parameter "includePartialResults" to true, the IRS will fetch the payload data for the job (relationships, AAS shells and submodels, if applicable) and attach it to the response object.

This will then be passed to the caller.

When a user wants to cancel a job execution, the IRS will lookup that job in the persistent job store and transition it to the CANCELED state, if it exists. If a job is canceled, no further requests to the Catena-X network will be performed.

Afterwards, the IRS will return the updated job details of the canceled job to the user.

This component enables data providers to provide the BoM as planned aspect models via the Catena-X defined solution stack (i.e. EDC). The BoM as planned aspect models consists of three aspect models:

Note: ESS supplier responses are involved in each step of the process.

This section focuses only on the IRS itself, detached from its neighbors. It shows the resources deployed in Kubernetes for the IRS.

For detailed information about the API model, please refer to the API specification.

A job can be in one of the following states:

Credentials must never be stored in Git!

For the IRS, we utilize the dependency inversion mechanisms provided by Spring Boot as much as possible.

The principle says:

High-level modules should not import anything from low-level modules. Both should depend on abstractions (e.g., interfaces). Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions.

Adhering to this, we define clear interfaces between the different domains (e.g. job orchestration and AAS communication) in the IRS and let dependencies be injected by the framework. This improves testability of the classes as well.

The hexagonal architecture divides a system into several loosely-coupled interchangeable components, such as the application core, the database, the user interface, test scripts and interfaces with other systems. This approach is an alternative to the traditional layered architecture.

For the IRS, this means decoupling the application logic from components like the BLOB store, the REST API controllers or the AAS client connection. With an interface between the parts (so-called port), it is easy to switch to other implementations, e.g. if you want to change the persistence implementation. No changes to the application logic will be necessary.

The IRS stores two types of data in a persistent way:

All of this is data is stored in an object store. The currently used implementation is Minio (Amazon S3 compatible). This reduces the complexity in storing and retrieving data. There also is no predefined model for the data, every document can be stored as it is. The downside of this approach is lack of query functionality, as we can only search through the keys of the entries but not based on the value data. In the future, another approach or an additional way to to index the data might be required.

To let the data survive system restarts, Minio needs to use a persistent volume for the data storage. A default configuration for this is provided in the Helm charts.

There currently is no transaction management in the IRS.

There is no session handling in the IRS, access is solely based on bearer tokens, the API is stateless.

All interfaces to other systems are using RESTful calls over HTTP(S). Where central authentication is required, a common OAuth2 provider is used.

For outgoing calls, the Spring RestTemplate mechanism is used and separate RestTemplates are created for the different ways of authentication.

For incoming calls, we utilize the Spring REST Controller mechanism, annotating the interfaces accordingly and also documenting the endpoints using OpenAPI annotations.

There are two types of potential errors in the IRS:

The heart of the IRS is the parallel execution of planned jobs. As almost each job requires multiple calls to various endpoints, those are done in parallel as well to reduce the total execution time for each job.

Tasks execution is orchestrated by the JobOrchestrator class. It utilizes a central ExecutorService, which manages the number of threads and schedules new Task as they come in.

Data validation happens at two points:

The IRS gives its users the ability to manage, create and delete complex policies containing permissions and constraints in order to obtain the most precise control over access and use of data received from the edc provider. Policies stored in Policy Store will serve as input with allowed restriction and will be checked against every item from EDC Catalog.

The structure of a Policy that can be stored in storage can be easily viewed by using Policy Store endpoints in the published API documentation. Each policy may contain more than one permission, which in turn consists of constraints linked together by AND or OR relationships. This model provides full flexibility and control over stored access and use policies.

In order to work with the decentral network approach, IRS requires the Digital Twin to contain a "subprotocolBody" in each of the submodelDescriptor endpoints. This "subprotocolBody" has to contain the "id" of the EDC asset, as well as the "dspEndpoint" of the EDC, separated by a semicolon (e.g. "subprotocolBody": "id=123;dspEndpoint=http://edc.control.plane/api/v1/dsp").

The "dspEndpoint" is used to request the EDC catalog of the dataprovider and the "id" to filter for the exact asset inside this catalog.

If the "dspEndpoint" is not present, every available EDC endpoint in DiscoveryService will be queried until a asset with the "id" can be found.

The IRS caches data provided externally to avoid unnecessary requests and reduce execution time.

Caching is implemented for the following services:

The IRS is built using Apache Maven and utilizes all the standard concepts of it. Test execution is part of the build process and a minimum test coverage of 80% is enforced.

The project setup contains a multi-module Maven build. Commonly used classes (like the IRS data model) should be extracted into a separate submodule and reused across the project. However, this is not a "one-size-fits-all" solution. New submodules should be created with care and require a review by the team.

The Maven build alone only leads up to the JAR artifact of the IRS. To create Docker images, the Docker build feature is used. This copies all resources into a builder image, builds the software and creates a final Docker image at the end that can then be deployed.

Although the Docker image can be deployed in various ways, the standard solution are the provided Helm charts, which describe the required components as well.

There are two methods of code generation in the IRS:

There currently is no data migration mechanism for the IRS. In case the model of the persisted data (Jobs) changes, data is dropped and Jobs will need to be recreated.

The IRS utilizes the configuration mechanism provided by Spring Boot. Configuration properties can be defined in the file src/main/resources/application.yml

For local testing purposes, there is an additional configuration file called application-local.yml. Values can be overridden there to support the local dev environment.

Other profiles should be avoided. Instead, the configuration can be overwritten using the Spring Boot externalized configuration mechanism. The operator must have total control over the configuration of the IRS.

If the number of consumers raises, the IRS can be scaled up by using more resources for the Deployment Pod. Those resources can be used to utilize more parallel threads to handle Job execution.

The IRS can run in clustered mode, as each running job is only present in one pod at a time. Note: as soon as a resume feature is implemented, this needs to be addressed here.

Logs are being written directly to stdout and are picked up by the cluster management.

The application can be monitored using Prometheus and Grafana. Both systems are defined in the Helm charts with a default setup. A number of Grafana dashboards are deployed automatically, to display data about:

The umbrella chart provides a pre-configured catena-x network which includes all necessary components for the IRS to work.

For IRS this includes:

Services marked with * are currently mocked, since there is no FOSS-Component available for IATP/DIM and the configuration and data seeding for EDC Discovery is not possible at the moment.

These Service are used together with the IRS Helm Chart to provide a E2E Testing Infrastructure which can be spun up directly in a GitHub Workflow to execute E2E Tests against.

The action https://github.com/eclipse-tractusx/item-relationship-service/blob/main/.github/actions/setup-umbrella-env/action.yaml creates this environment and seeds Testdata for IRS use-cases directly to the EDC Provider components. After Umbrella setup and seeding, the IRS docker image is built based on the branch, the workflow is run from, and the IRS helm chart from the same branch is installed. This ensures, that features on branches can be tested as well.

The workflows tavern-UMBRELLA.yaml and cucumber-integration-test-UMBRELLA.yaml use the action to execute the tests. Test results are automatically added to the workflow summary and in case of the cucumber tests uploaded to the IRS Cucumber Report

Due to limitations in the IATP mock, the only way of using multiple BPNs in the test data is to spin up multiple EDC provider instances. However, the chart is used inside GitHub Workflows, so compute resources are limited. For this reason, the IRS test data was changed to include only data pointing to a single BPN.