DevOps & CI/CD

Platform Maintenance, Codebase Management, and Governance

Platform operation follows DevSecOps and change-management principles:

• Continuous Integration / Continuous Delivery is implemented using GitLab CI and ArgoCD
• CI pipelines in GitLab ensure build, automated testing, static code analysis (SonarQube), and vulnerability scanning before deployment
• CD is managed via ArgoCD (GitOps approach) with version-controlled deployment definitions
• Deployment packaging is standardised using a single parametrized Helm chart, allowing selective deployment of platform components based on configuration
• The platform is currently deployed into a single Kubernetes cluster supporting multiple environments:
  • Dynamic environment for the master branch — automatically redeployed on each merge to master; represents the latest integrated state of the platform
  • Dynamic environments for merge requests — created on demand per feature branch; deployment is triggered by adding the DEPLOY label to the merge request; used for validation, testing, and stakeholder review
• Version-controlled Helm templates ensure reproducible and consistent releases across all environments
• Dual approval is required for production releases (DevOps + Compliance Officer)
• All code and configuration changes are traceable and logged for audit purposes

Routine infrastructure maintenance may be supported by a designated DevOps partner, but deployment control and approvals remain solely with DEUSS. This ensures compliance with outsourcing control and four-eyes governance principles.

The off-chain part of the solution is intended to be open source. The source code will be publicly available; however, the project will follow a controlled contribution model:

• the main repositories remain protected
• direct modifications are not allowed
• changes are introduced through controlled forks and review/approval processes

Security-sensitive operational configuration, secrets, deployment credentials, and runtime infrastructure settings are not intended to be publicly exposed.

---

Executive Summary & Repository Matrix

The project infrastructure is structured as a distributed, multi-repository monorepo and microservice architecture. CI/CD automation spans four major operational domains:

1. Infrastructure & Continuous Delivery (argocd-k8s-dev): Central GitOps control repository utilizing ArgoCD App-of-Apps and ApplicationSets patterns to manage Kubernetes platform infrastructure and application workloads.
2. Frontend Applications (showcase/frontend): React / Nx monorepo containing the Whitelabel (deuss-whitelabel) and Backoffice/Core (deuss-core) applications, with automated Playwright E2E matrix testing and container build delivery.
3. Offchain Microservices Core (offchain): Java 25 Gradle microservices monorepo and TypeScript Indexer with path-filtered matrix pipelines, GraphQL schema auto-sync, and dual-mode deployment (GitOps & SSH Docker Compose).
4. Internal Diligence Agent Stack (ida-stack): Distributed 5-repository AI compliance engine managed by a central orchestrator (ida-deployment) with pipeline-id lockstep image tagging and Docker Swarm stack deployments.

Repository & Documentation Map

Domain / Subsystem	Primary Repository	Key Technologies	Deployment Target	Detailed Documentation
GitOps Infrastructure	argocd-k8s-dev	Helm, ArgoCD, Kubernetes, AVP	Kubernetes Clusters	GitOps & ArgoCD
Web Frontend	frontend	Nx, React, Nginx, Playwright, Docker	K8s via ArgoCD GitOps	Frontend Applications
Offchain Core	offchain	Java 25, Gradle, TypeScript, CNPG	K8s (ArgoCD) & SSH (Docker Compose)	Offchain Microservices
IDA Stack	ida-deployment, ida-backend, ida-detector, ida-source-pi, ida-admin	FastAPI, Node.js, Next.js, MongoDB	Docker Swarm (SSH Remote Host)	IDA Stack

---

End-to-End CI/CD Architecture

The following diagram illustrates how code updates, container builds, and GitOps / SSH triggers flow across all system components from developer push to deployment:

---

Subsystem CI/CD Breakdown

GitOps Infrastructure (argocd-k8s-dev)

• Role: Centralized infrastructure and delivery engine. Declaratively provisions platform services (Istio, Ceph CSI, CloudNativePG, Debezium, Loki, Strimzi Kafka) and application workloads.
• Core Design:
  • App-of-Apps Pattern: Top-level root Helm chart references sub-charts and manifests under templates/apps/.
  • ApplicationSets: Dynamic PR generators (fe-mr.yaml, offchain-mr.yaml) poll GitLab MR API every 15 seconds. When an MR is tagged with DEPLOY, ArgoCD automatically provisions an ephemeral K8s preview namespace (deuss-fe-<MR_ID>, deuss-offchain-<MR_ID>).
  • Validation Pipeline: Standard .gitlab-ci.yml runs helm lint and helm template dry-run checks on every commit.
• Detailed Documentation: GitOps & ArgoCD

Frontend Applications (frontend)

• Role: Modern monorepo housing the public Whitelabel and admin Core/Backoffice web applications.
• Core Design:
  • Build & Quality Gates: Parallel compilation via Nx, ESLint code auditing, and npm audit dependency security checks.
  • E2E Testing Matrix: Automated Playwright test matrix across 5 smoke specs and 13 full specs with merged Allure HTML reports. Includes a dedicated scheduled nightly E2E suite against the development environment.
  • Container Security & Runtime Injection: Compiles static assets into Alpine Nginx containers. Custom entrypoint scripts perform runtime string replacement (sed) to inject environment variables (Keycloak, RPC endpoints, smart contract addresses) into JavaScript bundles on container startup.
  • GitOps Promotion: Release jobs automatically clone argocd-k8s-dev, update image tags in values.yaml via yq, and commit back to trigger cluster sync.
• Detailed Documentation: Frontend Applications

Offchain Microservices Core (offchain)

• Role: Java 25 microservices monorepo and TypeScript Indexer backing core blockchain domain logic and persistence.
• Core Design:
  • Incremental & Path-Filtered Pipelines: Utilizes GitLab CI path rules (ci/rules.yml) to only compile and test microservices modified in a commit.
  • GraphQL Schema Auto-Sync: If indexer/schema.graphql is updated in a Merge Request, the sync:schema_graphqls job automatically generates Java model bindings and commits schema.graphqls directly back to the developer's MR branch.
  • Docker Build & Smart Retagging: Build jobs execute BuildKit multi-stage builds. To save runner resources, untouched services execute docker-retag:*, copying existing master-latest registry tags to the short commit SHA.
  • Dual Deployment Model:
    1. Kubernetes GitOps: Managed via argocd-k8s-dev for Dev and Master integration clusters.
    2. SSH Docker Compose: Legacy and Staging/Test/Pilot host deployments push Compose files and inject environment secrets via SSH.
• Detailed Documentation: Offchain Microservices

Internal Diligence Agent Stack (ida-stack)

• Role: Distributed multi-repository AI compliance stack consisting of 5 repositories (ida-deployment, ida-backend, ida-detector, ida-source-pi, ida-admin).
• Core Design:
  • Central Orchestration: ida-deployment acts as the master pipeline controller.
  • Lockstep Pipeline Tagging: The setup stage creates DEPLOY_TAG=$CI_PIPELINE_ID. It invokes child component pipelines in sibling repositories via GitLab trigger syntax with strategy: depend.
  • Unified Artifact Versioning: All 4 microservices are built and tagged with the exact same $DEPLOY_TAG image label to guarantee component version compatibility.
  • Docker Swarm Stack Deployment: The deploy-demo stage uses SSH/SCP to transfer docker-compose-ida.yml to the target host, substitutes ${DEPLOY_TAG} via envsubst, and executes docker stack deploy --prune.
• Detailed Documentation: IDA Stack

---

Environment Architecture & Matrix

Environment Tier	Target Subsystem	Infrastructure / Host	Deployment Mechanism	Trigger & Approval
Local Development	All Systems	Developer Machine	docker compose up -d / Nx dev server	None (Local execution)
Dynamic MR Previews	Frontend & Offchain	K8s Namespace (deuss-*-MR)	ArgoCD ApplicationSets (PR Generator)	Attach DEPLOY label on GitLab MR
Static Integration	Frontend & Offchain	K8s Namespace (sc-100, ebsi-test)	ArgoCD App-Sets (values.yaml)	Push to branch registered in values.yaml
Master / Dev Integration	All Systems	K8s Cluster (deuss-*-master)	ArgoCD GitOps (argocd:update-master)	Automatic on merge into master
Test / Staging	Offchain	SSH target host	SSH + Docker Compose (deploy:test)	Manual trigger on release/* branch
Demo / Pilot	IDA Stack & Offchain	SSH pilot host	SSH + Docker Swarm (deploy-demo) / Compose	Manual trigger on master or release/*

---

Secrets Management

To run or administer CI/CD pipelines across all repositories, the following secrets must be configured in GitLab CI/CD Settings (Settings > CI/CD > Variables):

Variable Name	Scope / Target	Purpose	Type / Security
ARGO_PROJECT_TOKEN	Frontend & Offchain	Token with write access to clone and push image tags to argocd-k8s-dev.git	Masked Secret
SCHEMA_SYNC_TOKEN	Offchain	Token with write access to push auto-generated schema.graphqls back to MR source branches	Masked Secret
E2E_ENV_FILE_B64	Frontend	Base64-encoded payload of .env.e2e-dev for scheduled nightly E2E tests	File
SSH_KEY / DEV_SSH_KEY / PILOT_SSH_KEY	IDA & Offchain	Private SSH keys authorized for remote deployment hosts	Masked Secret / File
DEPLOY_HOST / DEPLOY_USER	IDA Stack	Target SSH host address and user for Docker Swarm stack deployments	Variable
CI_REGISTRY / CI_REGISTRY_USER / CI_REGISTRY_PASSWORD	All Repositories	Standard GitLab Container Registry authentication credentials	Predefined
DEV_*_PGPASS, PILOT_*_PGPASS	Offchain	Database passwords and connection secrets across environments	Masked Secret
DEV_*_KEY, PILOT_*_KEY	Offchain	Blockchain private keys for smart contract governance and transactions	Masked Secret

Kubernetes Secrets

• argocd-repo-creds-gitlab-access-git: Located in argocd namespace. Contains GitLab access token allowing ArgoCD PR generators to poll MR state for Frontend and Offchain projects.
• ArgoCD Vault Plugin (avp-helm-optional-values): Injects runtime secrets and overrides during Helm template rendering.

---

Operational Runbook

Scenario A: Running Local Development Workloads

IDA Stack:

cd ida-deployment/development
docker compose up -d --build
# Admin UI available at http://localhost:3000

Offchain Microservices:

cd offchain/deploy/local
docker compose -f docker-compose.local-full.yml up -d

Frontend Apps:

cd frontend
npm install
npx nx run deuss-frontend:serve

Scenario B: Provisioning an On-Demand MR Preview Environment

1. Open a Merge Request in showcase/frontend.git or offchain-microservices-core/offchain.git.
2. Add the label DEPLOY to the Merge Request.
3. ArgoCD polls GitLab API within 15 seconds, detects the tag, and automatically provisions an isolated Kubernetes namespace (deuss-fe-<MR_ID> or deuss-offchain-<MR_ID>).
4. To override environment parameters for your MR, commit a custom values file to values/<MR_ID>.yaml inside your application repository.
5. Closing or merging the MR automatically triggers ArgoCD to tear down the preview namespace (prune: true).

Scenario C: Executing a Production / Staging Release

Frontend / Offchain GitOps Release:

1. Merge feature branch into master.
2. The release pipeline builds Docker images, tags them with :latest and :$CI_COMMIT_SHORT_SHA, and executes argocd:update-master.
3. The job updates values.yaml in argocd-k8s-dev. ArgoCD detects the change and automatically syncs updated containers to the cluster.

IDA Stack Pilot Release:

1. Merge changes into master branches of component repos and ida-deployment.
2. Navigate to the pipeline run on ida-deployment master branch.
3. Verify all 4 downstream component build jobs pass.
4. Click the deploy-demo manual play button. The orchestrator will deploy the synchronized $DEPLOY_TAG stack to the Swarm host.

Scenario D: Updating CI Runner Base Images & Dependencies

• Offchain CI Base Image: Edit offchain/ci/Dockerfile and trigger the build:ci-image scheduled pipeline under CI/CD > Schedules.
• Dependency Updates: Renovate bot runs automatically via scheduled pipelines in offchain/ci/scheduled-tasks.yml, opening MRs and triggering sequential child pipelines (trigger:services-build).

---

Troubleshooting & Emergency Rollbacks

Common Failures & Quick Fixes

Issue Symptom	Affected Domain	Likely Root Cause	Remediation Step
helm-lint-and-template failure	argocd-k8s-dev	YAML formatting or missing Helm value	Run helm lint . locally in argocd-k8s-dev and fix broken indentation.
MR Environment not created	Frontend / Offchain	Missing DEPLOY label or closed MR	Ensure MR status is opened and label DEPLOY is attached.
[e2e:dev:nightly] Missing E2E env input	Frontend	E2E_ENV_FILE_B64 missing or invalid	Re-encode .env.e2e-dev to base64 (base64 -w 0) and save as File variable in GitLab settings.
sync:schema_graphqls git push error	Offchain	Expired SCHEMA_SYNC_TOKEN	Refresh SCHEMA_SYNC_TOKEN with write repository permissions in GitLab variables.
deploy-demo SSH Permission denied	IDA Stack	Invalid SSH key or host setup	Verify $SSH_KEY in ida-deployment and check ~/.ssh/authorized_keys on target host.
ArgoCD ComparisonError	All GitOps	Expired argocd-repo-creds-gitlab-access-git	Re-create the secret in the argocd K8s namespace with a valid GitLab Personal Access Token.

Emergency Rollback Procedures

GitOps Rollback (Kubernetes Workloads)

Git Revert (Recommended):

cd argocd-k8s-dev
git revert HEAD
git push origin master

ArgoCD will detect the reverted commit in values.yaml and instantly re-deploy the previous container image tags.

ArgoCD UI Immediate Rollback:

1. Open ArgoCD UI.
2. Select target Application (e.g. deuss-fe-master).
3. Click Sync Options → Disable Automated Sync.
4. Perform manual sync targeting the previous stable Git commit SHA.

SSH / Docker Swarm / Compose Rollback

GitLab CI One-Click Re-run:

1. Open historical pipelines in ida-deployment or offchain.
2. Locate the last known stable pipeline run.
3. Re-run / Retry the manual deployment job (deploy-demo, deploy:test, or deploy:pilot).

Manual Server Host Rollback:

ssh deploy@<your-pilot-host>
cd ida-deployment
export DEPLOY_TAG=<previous-stable-pipeline-id>
cat ./demo/docker-compose-ida.yml | envsubst | docker stack deploy --prune --with-registry-auth --compose-file - ida-demo