Skip to main content

Traffic Management

Traffic Management - External API Gateway, Blockchain Gateway, Ingress, Service mix​

Architecture decisions​

1. Traffic Management & Security (Istio)​

We leverage Istio to manage network traffic and enforce security policies across the infrastructure through three primary roles: A. Istio Ingress Gateway

  • Role: Acts as the primary entry point for all external traffic entering the Kubernetes cluster.
  • Function: Serves as a reverse proxy routing requests from client applications to internal microservices. B. Istio External API Gateway
  • Role: A dedicated gateway for exposing and securing public-facing APIs.
  • Security Implementation:
    • mTLS: Enforces Mutual TLS to ensure identity verification for both communicating parties.
    • OAuth2 & JWT Validation: Integrated with Keycloak (IDP). The gateway natively validates JWT tokens to ensure authenticity before forwarding requests to backend services. C. Istio Service Mesh
  • Scope: Implemented for all synchronous inner-service communication within the deusscore and deusbroker namespaces.

2. Asynchronous Messaging (Kafka Strimzi)​

For event-driven workflows and decoupled communication within the Kubernetes cluster, we utilize Apache Kafka managed by the Strimzi operator.

  • Role: Provides a robust, scalable message bus for internal asynchronous communication.
  • Key Features:
    • Service Decoupling: Allows services to communicate without direct dependencies, enhancing overall system fault tolerance.
    • Strimzi Operator: Simplifies Kafka management using Kubernetes-native Custom Resources (CRDs) for brokers, topics, and users.
    • Reliability: Ensures high availability and persistence of data streams across internal service boundaries.

Istio traffic architecture overview

3. Blockchain Gateway​

3.1. Backend-Only EBSI Access (Blockchain Gateway) All EBSI communication is performed exclusively via a dedicated Blockchain Gateway in the Broker Backend. Rationale: Centralizes VC/DID handling, access token management and signing operations in a secure backend component. Eliminates frontend exposure to sensitive credentials and improves auditability and regulatory safety.

3.2. DID Model One DID for DEUSS Core (platform), one DID per Broker entity. No per-user DID. Rationale: Reduces operational complexity while maintaining clear separation of platform and broker identity. Aligns with EBSI confirmation that infrastructure VC-based access is acceptable.

3.3. Signing Strategy User-level transactions are signed in FE (passkey). Platform/administrative transactions are signed in BE. Rationale: Protects private keys in backend infrastructure, enforces transaction policies, and supports controlled blockchain interaction.

3.4. Centralized Access Token Management Blockchain Gateway handles acquisition, caching, and renewal of EBSI access tokens. Rationale: Prevents token leakage, simplifies lifecycle management, and increases operational resilience.

3.5. Unified Blockchain Access Layer All backend services access blockchain only through the Blockchain Gateway. Rationale: Ensures consistent security controls, authorization enforcement, logging, and traceability across the platform.

Secure External API Communication (DEUSS External API Gateway)​

DEUSS exposes a controlled set of external interfaces through an External API Gateway (deployed in Kubernetes) to integrate with third-party systems such as the Payment Service Provider (PSP), Distributor/Originator application, and Broker/Operator application. The gateway provides a single governed entry point for external traffic, enforcing authentication, authorization, traffic policies, and security controls consistently across all exposed APIs.

Interfaces and Protocols (REST API, WebSocket)​

  • REST API (HTTPS) is the primary integration mechanism used for:
    • Queries and data retrieval (e.g., issuer status, bond details, reference data).
    • Command operations (e.g., submit onboarding request, initiate issuance workflow, initiate settlement steps).
    • Document metadata operations (e.g., register document, request a download link, upload confirmations).
  • WebSocket (WSS) is used only where low-latency server-to-client streaming is required:
    • Real-time status updates and event notifications (e.g., onboarding state changes, trade lifecycle updates).
    • Live market / order or system notifications, if applicable to the regulated operating model.
  • Documents / binary payloads are not transported via WebSocket or Kafka. Documents are stored in S3-compatible object storage and exchanged using time-limited pre-signed URLs (download/upload), while REST APIs transport only document metadata and integrity attributes (hash, size, type).

Synchronous vs. Asynchronous Operations​

  • Synchronous operations use REST request/response for immediate validation and acknowledgement.
  • Asynchronous operations (e.g., AML/scoring processing, settlement workflows, long-running onboarding steps) return 202 Accepted with a correlationId and statusUrl, and completion is delivered via:
    • WebSocket push (when the external party is a UI-driven application), or
    • Webhook callback (for system-to-system B2B integration), secured with mTLS and signed payloads.

Security Controls (mTLS + OIDC)​

DEUSS applies a layered Zero-Trust approach:

  • mTLS (mutual TLS) is enforced for B2B system-to-system integrations (e.g., PSP, Broker Platform backends):
    • The external caller must present a valid client certificate issued by a trusted CA.
    • The gateway authenticates the calling system identity at the transport level.
  • OAuth2 / OIDC is used for end-user facing applications (issuer frontend, investor frontend):
    • Users authenticate via the central IdP (Keycloak) and obtain access tokens.
    • The API Gateway validates tokens and enforces authorization based on scopes/roles and tenant context (e.g., jurisdictionId, partnerId).
  • Least privilege & tenant isolation:
    • External clients receive only the minimal API surface and authorization scopes required.
    • Requests are authorized per partner and per jurisdiction (multitenancy), including strict access to issuer/investor data partitions.
  • Auditability and traceability:
    • Every external request carries a traceId/correlationId propagated into backend logs and Kafka events (OpenTelemetry / W3C Trace Context), enabling full forensic and operational traceability.

External API Gateway and Edge Protection​

  • The Istio External API Gateway is operated inside Kubernetes to align with the platform’s deployment model and policy enforcement.
  • Recommended gateway options (Kubernetes-native):
    • Istio External API Gateway (API management, policies, OAuth2 integration, rate limiting, analytics). + API Gateway policies as viable alternatives depending on required API management features.
  • Cloudflare is used as the internet edge layer in front of the gateway to provide:
    • DDoS protection, WAF rules, bot mitigation,
    • rate limiting at the edge,
    • TLS termination and traffic filtering (while preserving end-to-end encryption to the gateway as required).

Blockchain Gateway - responsibilities & functionality​

Blockchain Gateway is a backend integration component deployed within DEUSS Core that provides a single, secure entry point for all blockchain write operations towards EBSI (and optionally also towards non-EBSI “dev” chains). Read operations will use preferably Indexer service. The gateway intentionally prevents direct frontend access to EBSI and removes the need for frontend handling of Verifiable Credentials (VCs) and EBSI access tokens, which is both operationally simpler and regulatorily safer.

Key responsibilities​

  • EBSI authentication handling (VC/DID based):
    • Stores and protects VC(s), DID(s), and private keys required to obtain an EBSI access_token.
    • Automatically requests and refreshes access_token via https://api-*.ebsi.eu/authorisation/v4/token and caches it until expiry (typically 2 hours), with safe renewal logic.
  • Signing strategy enforcement:
    • Supports a controlled signing model:
      • User-signed transactions: frontend signs the transaction payload using the user’s passkey/wallet and sends it to the gateway for submission.
      • Platform/admin-signed transactions: the gateway signs transactions using an admin/platform key (stored in Vault/HSM-like storage) when required by platform operations.
  • Policy-based routing & chain abstraction:
    • Routes requests to either EBSI JSON-RPC endpoint or a configured “dev” JSON-RPC endpoint.
    • Adds EBSI authorization header (Bearer access_token) only when the target chain requires it (EBSI vs. others)
  • Security & integrity controls:
    • Validates caller identity (mTLS / internal service identity) and enforces authorization (RBAC / client allowlists).
    • Validates request structure, schema, and applies allowlisted method policy for JSON-RPC calls.
    • Provides request integrity checks, replay protection (where applicable), and consistent auditing of administrative actions.
  • Observability & auditability:
    • Generates/propagates W3C Trace Context (traceparent) and logs correlation IDs for cross-system traceability.
    • Emits security/audit events (e.g., token refresh, signing operation, policy decision, failure reasons).

Identity model​

  • 1 DID for DEUSS Core (platform operations).
  • 1 DID per Broker entity (Broker-level operational identity).
  • User-level wallet can remain off-chain; users do not directly authenticate to EBSI.
  • FE signs user-level transactions (passkey); BE signs platform/admin operations (admin wallet/key).

Sequence diagram​

Sequence diagram (draw.io viewer)

Istio / Service mesh​

Overview​

This document covers Istio service mesh deployment on K3S/RKE2 clusters running Cilium CNI, managed through Rancher. The installation uses ArgoCD with Helm charts for GitOps-based deployment. The setup focuses on namespace-scoped mTLS rather than cluster-wide enforcement, providing secure service-to-service communication where needed while maintaining flexibility for services that don't require mesh capabilities.

Two ingress patterns are documented: direct LoadBalancer exposure with TLS termination at the Istio gateway, and integration with Cloudflare Tunnel for scenarios where external LoadBalancer isn't available or desired.

Architecture Summary​

  • Istio installed in istio-system namespace with HA control plane (3 replicas)
  • Istio Gateway deployed separately in istio-ingress namespace
  • PeerAuthentication set to PERMISSIVE globally, STRICT per namespace
  • Sidecar injection controlled via namespace labels
  • Compatible with Cilium CNI (kube-proxy replacement mode supported)

Prerequisites​

Cluster Requirements​

  • K3S/RKE2 cluster (v1.30+) with Cilium CNI installed
  • Rancher management configured (optional but assumed)
  • ArgoCD installed and operational
  • kubectl access with cluster-admin privileges

Broker Platform ⇄ DEUSS Core ⇄ Chain Integration Patterns​

Communication Principles​

  • Process orchestration is handled by DEUSS Core
  • Broker Platform application calls DEUSS Core via REST API (POST) to initiate an operation (e.g. Create Investor)
  • DEUSS Core generates unique Id for the operation and returns it to Broker Platform in REST responses
    • This ID is used as the integration ID to track the operation's progress
  • DEUSS Core handles all calls to Chain asynchronously using the JobRunr task scheduler
    • Submitting a transaction to the Chain returns a txHash
  • DEUSS Core polls the Chain via JSON-RPC using txHash to fetch the transaction receipt and orchestrate the communication workflow
  • Broker Platform reads status from DEUSS Core via polling (using the operation Id generated by DEUSS Core)
    • Broker Platform does NOT listen to Chain events for orchestration
  • DEUSS Core handles:
    • transaction mining delays (via TxNotMinedYetException and JobRunr retries)
    • reconciliation and error handling
    • manual intervention for failed jobs
  • Operators work with DEUSS Core processes, not directly with Chain

Communication Flow - Create Investor​

Broker Platform​

The Broker Platform acts as the client-facing orchestrator for user journeys (e.g., investor onboarding, creating bond issuances). Its responsibilities within this integration pattern include:

  • Initiating Workflows: Submits commands to DEUSS Core via REST API to begin a process.
  • Tracking State: Receives the operation integration Id (or entity ID) synchronously in the REST response and stores it against the local entity record (e.g., tracking a registerAccountOperationId).
  • Polling for Completion: Implements scheduled polling jobs (such as ExchangeAsyncRequestPollingJob) that periodically call the DEUSS Core REST API using the tracking Id.
  • User Feedback: Translates the PENDING, DONE, or FAILED statuses from DEUSS Core into UI states (loading spinners, success notifications, error banners) to keep the end user informed of progress.
  • Separation of Concerns: The Broker Platform is strictly shielded from blockchain complexities. It has no knowledge of txHash, block mining delays, smart contract event listeners, or transaction retries.

DEUSS Core​

  • Owns full process orchestration
  • Generates and manages unique operation integration Id
  • Stores process state in the operations DB table
  • Calls Chain asynchronously using JobRunr scheduled jobs
  • Drives workflow steps, a generic operations table tracks everything. Fields:
    • id (UUID)
    • type (String, e.g. REGISTER_ENTITY, CREATE_WALLET, REGISTER_ACCOUNT)
    • status (Enum: PENDING, DONE, FAILED)
    • transaction_hash
    • failure_reason
    • created, last_edit
Process States:​

The lifecycle of an operation is simple:

  • PENDING: Operation is enqueued or transaction is submitted but not yet mined.
  • DONE: Transaction was successfully mined on-chain and confirmed.
  • FAILED: Transaction failed, reverted, or encountered an unrecoverable error (e.g. Unauthorized).

Handles:

  • waiting for block mining
  • reconciliation via transaction receipt fetching
  • manual intervention
Handling Delayed Mining (Timeout & Retries)​

Because transactions take time to be mined, DEUSS Core implements a retry loop using the TxNotMinedYetException:

Retry/Reconciliation Job​
  • JobRunr Scheduled Job:
    • If an operation has a transaction_hash, the job queries the Chain to verify the actual transaction state using fetchReceiptByHash.
    • If the response is Pending (not mined yet), the job throws a TxNotMinedYetException.
    • JobRunr catches this exception and schedules a retry based on configured backoff limits (e.g., maximum 4 retries).
  • Updates process based on Chain:
    • success → update operation to DONE and continue to next workflow job
    • unrecoverable error → update operation to FAILED
    • exhausted retries → requires MANUAL_INTERVENTION_REQUIRED

Manual Intervention by operator​

  • Failed or unresolved processes require manual inspection.
  • Operators work only with DEUSS Core, no direct interaction with Chain required.