Regulatory Reporting

ComplianceEvidence v1 (Backend-Owned Compliance Data Mart)

Related to: #743 (API Boundary), #744 (Permission Model), #745 (File Lifecycle). This issue introduces a read-side reporting domain (separate database, separate service plane) that consumes events from the runtime services. It does not add new write paths to runtime data; it adds an evidence/reporting plane sized for regulators.

1. Context

The platform must generate regulator-grade submission packs:

• STOR (Suspicious Transaction or Order Report) under MAR Art. 16 → HCMC channel.
• STR (Suspicious Transaction Report, AML) → FIU (e-STR / FIU channel).
• Extensible to MiFID / MiFIR (RTS 22, FIRDS), DORA ICT-incident, and GDPR breach notifications.

Today the data needed for these packs is scattered across runtime services (registry-service, core-payment-service, blockchain-event-service, etc.). Some of it doesn't exist yet (secondary-market orders/trades, MiFIR identifiers, non-repudiation auth context, governance-action endpoints). There is no separate reporting database; there is no Change-Data-Capture path; there is no Compliance Ingestor.

The goal of this issue is to define one canonical ComplianceEvidence model, backed by a relational compliance.* schema in a dedicated database, fed by an event-driven pipeline from the runtime services, and to scope which fields exist today vs. need to be added.

2. Terminology (use these terms consistently)

Term	Meaning
Runtime plane	The existing live services (registry-service, core-payment-service, blockchain-event-service, whitelabel/*). Source of truth for live business state.
Compliance plane	The new namespace introduced by this issue: compliance-ingestor, compliance-db, optional compliance-api, in-cluster object storage. Read-optimized for reports.
ComplianceEvidence	A single logical case object that aggregates references to all evidence items (subjects, orders, trades, settlements, chain transactions, governance actions, snapshots, attachments) needed for one or more report submissions. Implemented relationally as compliance.compliance_case plus link tables.
Case	A compliance.compliance_case row — one investigation, possibly producing one or many submissions.
Submission	A compliance.report_submission row — one filed report to one authority, with status (DRAFT → READY → SUBMITTED → ACKNOWLEDGED / REJECTED).
Subject	A Company or User referenced by a Case (with a role such as PRIMARY_SUBJECT, ISSUER, BUYER, SELLER). Same Subject concept as #744; same identities.
Evidence item	Any concrete row in market_order, market_trade, settlement, chain_transaction, governance_action, bond_snapshot, company_portfolio_snapshot, or attachment that a Case links to via a case_* link table.
Auth context	Persisted per-event identity record: (distributor_id, x_api_key, mtls_cert_fingerprint, signer_address, signature, signature_payload_hash) used for non-repudiation. Stored in compliance.auth_context and referenced (FK) from any event that needs to prove who triggered it.
CDC	Change-Data-Capture. The mechanism by which runtime DB writes become Kafka events the Compliance Ingestor can consume.
Outbox pattern	Domain row + outbox row written in the same DB transaction; an external CDC reader (e.g. Debezium / peerdb) tails the outbox and publishes to Kafka. Guarantees at-least-once event delivery aligned with the runtime commit.
Compliance Ingestor	A new service in the compliance namespace that consumes Kafka topics, UPSERTs into compliance.* tables, and records ingested_event(topic, partition, offset, event_id) for idempotency.

3. Current State (grounded in the offchain code)

3.1 Service-name reconciliation between proposal and code

Proposal name	Real service	Status
Registry	offchain/core/registry-service/	✅ Exists
Payment	offchain/core/core-payment-service/	✅ Exists
Indexer / Eventer	offchain/indexer/ + offchain/whitelabel/blockchain-event-service/	✅ Exists
Core	offchain/common/framework/ + service-local cz.deuss.{service}.* packages	✅ Exists
Backoffice	—	❌ No backoffice-service directory. Operations the proposal labels "Backoffice" are scattered across registry-service controllers (AML state update) and mostly do not exist as endpoints yet. The label is conceptual, not a service.

3.2 Entities — what exists upstream vs. what the proposal needs

Company (core/registry-service/.../CompanyEntity.kt L29–71)

• ✅ Has: id, name, businessId, vatNumber, legalEntityId (the proposal calls this lei_code), walletAddress, registrationEntity, amlVerificationState, riskScore, frozenUntil, distributorID.
• ⚠️ Naming drift: proposal lei_code ↔ entity legalEntityId. Pick one in the canonical schema and document the mapping.
• ⚠️ Whitelabel company-service/.../CompanyEntity.kt is a parallel write model that lacks amlVerificationState, riskScore, frozenUntil, distributorID — only registry-service is the regulatory source of truth.

User

• ✅ core/registry-service/.../UserEntity.kt (L19–28) has id, distributorID, walletAddress — all three proposal-required fields.
• ⚠️ whitelabel/user-service/.../DeussUser.kt (L12–21) is auth-side only (email, firstName, lastName); not the regulatory source.

Bond (core/registry-service/.../BondEntity.kt L31–73, migration V0_6__bonds.sql)

• ✅ All upstream fields present: id, issuer FK, isin, name, state (DRAFT | WAITING_FOR_ISSUANCE | ISSUED), bondCount, bondNominalValue, currency, interestRate, issueDate, maturityDate, paymentFrequency, contractAddress.
• ❌ MiFIR / FIRDS fields are NEW: cfi_code, fisn, primary_venue_mic, primary_segment_mic. Not in any upstream entity. Either add upstream or enrich in compliance from a reference table.

Distributor

• ❌ No Distributor entity or table in any service. distributorID: UUID is carried by CompanyEntity and UserEntity, but the dimension itself does not exist.
• Implication: compliance.distributor must be populated from configuration or from a future distributor registry — the proposal cannot assume CDC will deliver distributor rows.

Market Order, Market Trade

• ❌ Do not exist. core-payment-service models BondPurchasePaymentEntity (one-way primary purchase) and coupon/repayment paydays. There is no two-sided order book, no MarketOrderEntity****, no MarketTradeEntity****, no offer/accept flow, no order status state machine.
• Implication: every compliance.market_order and compliance.market_trade field — including the NEW MiFIR fields venue_mic, segment_mic, tvtic — is contingent on a secondary-market feature being built upstream first. This is a hard dependency, not a data-modeling task.

Settlement (core/core-payment-service/.../BuyerSettlementPaymentEntity.kt L22–41 + PaydayEntity.kt L19–42)

• ✅ Has: id, paymentReference, amount, status, FK to payday (which carries activeAt, paymentType, couponRate).
• ❌ Missing: operation_id, value_date (only payday.activeAt exists), confirmed_at, psp_id. These must either be added upstream or derived/enriched in the compliance plane.
• ⚠️ The proposal's settlement_status enum (RECEIVED | REJECTED | SETTLED_ONCHAIN | FAILED_ONCHAIN | REVERSED) does not match BuyerSettlementPaymentStatus 1:1 — define the mapping explicitly.

Chain transaction (whitelabel/blockchain-event-service/.../BlockchainEventEntity.kt L16–47)

• ✅ Has: composite id (blockHash, logIndex), eventType: String, state (PENDING | NOT_MATCHED | ERROR | PROCESSED), timeReceived.
• ❌ Missing almost everything regulatory: no explicit tx_hash, no amount, no price_per_unit, no currency, no counterparty_name, no from_address / to_address, no chain_timestamp (only the off-chain timeReceived), no link to company_id or bond_id.
• Implication: compliance.chain_transaction is mostly an enrichment table, not a CDC mirror. The ingestor will join blockchain events with payment + registry rows to populate it. The proposal's "Source: Indexer/Eventer" is too optimistic.

Governance actions

Proposal action	Real endpoint	Service	Status
AML_UPDATE	PUT /v1/companies/{id}/verification-state → setAmlVerificationState(id, state)	registry-service CompanyController L107	✅ Exists
FREEZE_COMPANY / UNFREEZE_COMPANY	— (would be inline updates of CompanyEntity.frozenUntil)	registry-service	⚠️ Field exists, no dedicated endpoint found
FORCE_TRANSFER	—	—	❌ Absent
WALLET_RECOVERY	—	—	❌ Absent
BURN_BONDS	—	—	❌ Absent (bond burn state is implicit via BondState, no action endpoint)

Only 1 of the 6 proposed governance actions is a real endpoint. The compliance plane cannot record what doesn't fire upstream.

Auth context / non-repudiation

• ❌ None of X-API-Key****, X-Signer-Address****, X-Signature****, mTLS client cert fingerprint are extracted, logged, or persisted by any current controller / filter. Search across offchain/ finds nothing.
• Today's auth model: @Secured(SecurityRule.IS_AUTHENTICATED) with JWT (see #743). Non-repudiation as the proposal envisions it is aspirational — it requires both a wire-level capture point (filter / interceptor) and the gateway-side identity layer (#743) to be defined first.

3.3 CDC / streaming plumbing — what exists today

• Kafka: ✅ exists. offchain/deuss-platform-offchain-framework/kafka/ provides framework-level clients. media-service and notification-service consume topics. Helm chart configures a Strimzi-style operator (helm-chart/values.yaml L31–62, L85–100). Current use is internal service messaging, not CDC.
• Outbox table / pattern: ❌ no outbox table in any migration; no @Outbox or equivalent annotation; no outbox publisher.
• Debezium: ❌ not configured anywhere. No connector config in helm-chart/, no Debezium deployment.
• peerdb.io: ❌ no reference in the repo.

So the entire CDC plane is greenfield from the offchain side, with Kafka itself already operationally available.

3.4 Internal contradiction in the original proposal (needs resolution)

The proposal contains two incompatible statements about the ingestion path:

• Diagrams (mermaid-diagram.png, mermaid-diagram-2.png) show: Domain TX → outbox table (same TX) → Debezium logical replication slot → Kafka → Compliance Ingestor.
• Prose (in §"Deployment Schema"): "Not using outbox table as agreed on Arch WS, not using triggers to filter data — offload to Compliance ingestor."

These cannot both be true. Possible interpretations:

1. No outbox table; CDC reads runtime tables directly via logical replication. Debezium watches production tables. Simpler, but Debezium then publishes raw payment_db.market_order row changes — leaks production schema to the compliance plane, and any refactor of upstream tables breaks the contract.
2. Outbox table, written in the same TX, then Debezium tails the outbox. What the diagrams show. Stable contract surface; runtime services own the event shape. Slightly more code in the runtime services.
3. No outbox, no Debezium — Compliance Ingestor polls runtime read APIs. Simplest infra, weakest delivery guarantees (poll lag, missed deletes, fragile retention assumptions).

The Arch Workshop decision must be the authoritative one — the diagrams need to be redrawn to match the prose, or the prose corrected to match the diagrams, before implementation begins. This is the single most load-bearing inconsistency in the spec.

4. Goal

Stand up a backend-owned, separately-deployed compliance data mart that aggregates regulator-grade evidence from runtime services through a durable event pipeline, models cases and submissions relationally, and can render STOR / STR / MiFIR / DORA / GDPR submission packs without depending on runtime service availability or runtime-table retention.

5. Scope

5.1 Compliance schema (compliance.*)

The DDL in the ticket's DEUSS_report_mart.txt is the v1 baseline. Key sections:

• Reference dimensions (CDC-fed snapshots from runtime, point-in-time): compliance.company, compliance.user_identity, compliance.bond, compliance.bond_document, plus the compliance.distributor table (today seeded out-of-band — see §3.2).
• Transactional evidence: market_order, market_trade, settlement, chain_transaction.
• Governance actions: base governance_action + 6 subtype tables (one per action type), classical table-per-subtype design.
• Supporting snapshots: bond_snapshot, bond_snapshot_owner, company_portfolio_snapshot, company_portfolio_position.
• Cases & evidence linking: compliance_case + 9 case_* link tables (company, user, order, trade, settlement, chain_transaction, governance_action, bond_snapshot, portfolio_snapshot, attachment).
• Submissions: authority, report_type, report_submission, report_submission_attachment.
• Non-repudiation: auth_context referenced (FK) from market_order, market_trade, settlement, governance_action, bond_snapshot, compliance_case.
• Optional subtype tables for non-MAR/AML regimes: case_ict_incident (DORA / NIS2), case_privacy_breach (GDPR).
• Views: v_case_overview (one row per case with joins + evidence counts), v_case_timeline (UNION ALL of evidence items as a sortable event stream), v_report_submission_overview.

5.2 NEW fields the compliance plane requires (not present upstream today)

Tracked explicitly so the runtime work needed is visible:

• Bond — cfi_code, fisn, primary_venue_mic, primary_segment_mic. Either add to registry-service.BondEntity or load via a reference table maintained in compliance only.
• Market Trade — venue_mic, segment_mic, tvtic. Blocked on the secondary-market feature existing upstream.
• Order / Trade timestamps — orderCreatedAtUtc, orderUpdatedAtUtc, tradeCreatedAtUtc. Blocked on the same.
• Settlement — operation_id, value_date, confirmed_at, psp_id, plus the new settlement_status enum. Some of these can be sourced today; operation_id and psp_id need to be added to core-payment-service.
• Chain transaction — from_address, to_address (AML-critical), plus the full tx_hash / amount / currency / counterparty_name set. blockchain-event-service needs to be enriched or a join layer added in the ingestor.
• Auth context — capture and persist x_api_key, mtls_cert_fingerprint, signer_address, signature, signature_payload_hash. None of these wire-level signals are extracted today; requires interceptor/filter work coordinated with #743.
• Generic evidence attachments — attachment(kind, file_name, checksum_sha256, storage_uri, …) — extends today's bond-document S3 pattern (#745's lifecycle applies).

5.3 Ingestion pipeline (decision required — see §3.4)

Once the Arch decision is resolved, the canonical path is:

1. Each runtime service writes domain rows and (per the picked option) an event signal in the same DB transaction.
2. A CDC component (Debezium or peerdb — also a pending decision) reads the signal via a logical replication slot and publishes to a Kafka topic per stream (e.g. payment.market_order.v1, registry.company.v1).
3. The Compliance Ingestor consumes the topics, UPSERTs into compliance.* tables, and records compliance.ingested_event(topic, partition, offset, event_id) for idempotency.
4. Consumer offsets are committed only after the compliance DB transaction commits (the diagram is correct on this — at-least-once with idempotent upserts).
5. Backfill / replay is supported by a separate compliance-ingestor job mode that reads from a topic's earliest offset.

5.4 Deployment topology (Kubernetes)

Three K8s namespaces (matches mermaid-diagram.png):

• deuss-runtime — domain services + their per-service Postgres DBs. Unchanged in scope except for the in-same-TX event write.
• streaming — Strimzi-managed Kafka cluster + Kafka Connect; plus the CDC component (Debezium or peerdb, TBD).
• compliance — compliance-ingestor Deployment, compliance-db Postgres (operator-managed preferred), optional compliance-api (cases + submission orchestration), in-cluster MinIO for the final report storage, CronJobs for backfill / periodic aggregates / report cycles.

5.5 Historicization (_hist) — currently an open question

The original ticket mentions: "Resolve history — working proposal: _hist tables with after-insert/update/delete triggers."

Decide in scope of this issue:

• The compliance plane is itself the historical record (every UPSERT could write the old row to _hist). Triggers add overhead but produce a uniform audit trail.
• Alternative: append-only ingestion — compliance.* tables are never UPDATEd; each event writes a new row with a valid_from / valid_to pair. Heavier schema but no trigger drama.
• Decision needed before the DDL is treated as final.

5.6 Submissions & report generation

• report_type is a table (not an enum) so new reporting regimes can be added without migration. STOR, STR, MIFIR_RTS22, FIRDS, DORA_ICT, GDPR_BREACH are values.
• report_submission carries payload_uri (S3/MinIO key), payload_format (XML / XLSX / PDF), payload_checksum_sha256, external_reference (authority's receipt id), and links via FK to a compliance_case, an order/trade/bond/company, plus a period.
• Optional compliance-api orchestrates draft → ready → submitted → acknowledged transitions and the manual-or-API channel choice.

6. Functional & technical requirements

• The compliance plane is a separate database (and a separate Postgres instance) — runtime services must not be able to read or write compliance.* directly. This is the point of the separation.
• All event ingestion is idempotent ((topic, partition, offset) unique constraint on ingested_event).
• Schema versioning: every topic has an explicit vN suffix; the ingestor must accept multiple versions during rollover.
• auth_context rows are written before the event that references them, so the FK is always satisfiable. Events that arrive without an auth context attach to a synthetic "unknown" auth context for backfill periods; this is logged.
• Reference dimensions (compliance.company, compliance.bond, etc.) carry an updated_at and an as_of snapshot semantic — a Case must be able to reproduce the state of a Subject "as of the moment of detection".
• The compliance plane carries its own retention policy independent of runtime retention.
• The DDL in DEUSS_report_mart.txt lands as the initial V1_0__compliance_schema.sql migration of a new compliance-db; subsequent changes go through normal Flyway migrations.
• OpenAPI for the optional compliance-api follows #743's boundary classification — it is a user_jwt API for backoffice operators, not exposed to partners.

7. Acceptance criteria

• The DDL in DEUSS_report_mart.txt is applied as a Flyway baseline migration in a new compliance-db Postgres instance, deployed under the compliance Kubernetes namespace.
• The ingestion-path contradiction from §3.4 is resolved in writing (Arch decision recorded), and the diagrams + prose in this spec agree.
• The Compliance Ingestor consumes at least one topic end-to-end (e.g. registry.company.v1) and idempotently UPSERTs into compliance.company, with ingested_event rows tracking (topic, partition, offset).
• compliance.distributor is populated (config-driven or via a future distributor registry) — every compliance.company.distributor_id resolves.
• The runtime gaps from §3.2 are tracked as explicit upstream tickets:
  • Add MiFIR / FIRDS fields to registry-service.BondEntity (or accept compliance-only enrichment).
  • Add operation_id, value_date, confirmed_at, psp_id to core-payment-service settlement model.
  • Enrich blockchain-event-service (or add ingestor-side join) so compliance.chain_transaction is populable with tx_hash, amount, currency, addresses.
  • Define endpoints for FREEZE_COMPANY / UNFREEZE_COMPANY / FORCE_TRANSFER / WALLET_RECOVERY / BURN_BONDS — or formally drop them from v1 scope.
  • Extract and persist auth_context fields at the runtime API boundary (coordinated with #743).
• The secondary-market dependency is explicitly captured: market_order and market_trade ingestion is out of scope until the runtime feature exists; the spec marks this and does not block on it.
• Historicization decision (§5.5: triggered _hist tables vs. append-only ingestion) is recorded in this spec.
• A first end-to-end STR / STOR submission pack can be assembled from v_case_overview + v_case_timeline + at least one compliance.attachment (CSV extract + draft report).
• All sensitive operator endpoints on the optional compliance-api are classified per #743 (user_jwt, not partner_authenticated or service_internal) and consult the Permission Model from #744.

8. Risks & notes

• Single largest risk: the §3.4 contradiction (outbox vs. no-outbox vs. polling) is unresolved. Choosing wrong cascades into months of rework. Resolve at Arch before any code lands.
• backoffice-service does not exist. Five of six proposed governance actions have no upstream endpoint today (§3.2). Either scope them in or drop them — the compliance plane cannot record events that the runtime never emits.
• Secondary market is not modeled upstream. market_order and market_trade plus their MiFIR fields are blocked on the runtime team. Land the compliance tables now (schema-only); land the ingestion when the runtime feature ships.
• blockchain-event-service is far thinner than the proposal assumes. No tx_hash, no amount, no addresses, no currency. compliance.chain_transaction is mostly enrichment, not CDC mirror. Plan the join logic explicitly — it is not a free lunch.
• Non-repudiation is aspirational today. None of X-API-Key, X-Signer-Address, X-Signature, mTLS cert fingerprint are extracted by any current filter. Coordinate with #743 — the API boundary work is the natural place to introduce these capture points.
• legalEntityId vs. lei_code naming drift. Pick one in the canonical compliance.company.lei_code column and document the mapping. Don't let the names diverge across the runtime and the compliance plane.
• CDC component choice (Debezium vs. peerdb) is unresolved in the original ticket. peerdb claims lower operational overhead; Debezium has more battle-testing in financial data planes. Decide before infra ticket lands.
• Retention separation: runtime DBs can prune; the compliance plane must not. Migration plans for any retention shrink in registry-service / core-payment-service must include a "compliance has its copy" check.
• GDPR cross-cuts: storing wallet addresses, AML risk scores, and signer signatures in a long-retention compliance DB is a personal-data exposure category. Confirm with legal that the lawful basis (regulatory obligation) covers the retention scope before going live.