Skip to main content

Offchain Microservices CI/CD (offchain)

This document provides comprehensive technical documentation for the CI/CD pipeline and deployment infrastructure of the offchain microservices monorepo.

Prerequisites & Configuration​

GitLab CI/CD Variables​

The pipeline relies on several masked and protected environment variables configured in GitLab CI/CD settings (Settings > CI/CD > Variables):

Variable NameDescription / PurposeTarget Stage / Jobs
ARGO_PROJECT_TOKENPersonal Access Token with write access to clone and push image tag updates to the argocd-k8s-dev repository.argocd:update-static-env, argocd:update-master
SCHEMA_SYNC_TOKENGitLab access token with write access to push auto-generated schema.graphqls updates back to MR source branches.sync:schema_graphqls
RENOVATE_TOKENAPI token for Renovate bot to fetch open MRs and source branches.trigger:services-build
RENOVATE_PIPELINE_TOKENTrigger token to launch child pipelines for Renovate branches.trigger:services-build
DEV_SSH_KEYPrivate SSH key for authenticating with the development server.deploy:service:dev, deploy:dev:platform
TEST_SSH_KEYPrivate SSH key for authenticating with the test deployment host.deploy:test
PILOT_SSH_KEYPrivate SSH key for authenticating with the demo/pilot deployment host.deploy:demo, deploy:pilot
DEV_*_PGPASS, TEST_*_PGPASS, DEMO_*_PGPASS, PILOT_*_PGPASSDatabase credentials for individual microservices across Dev, Test, Demo, and Pilot environments.Deployment jobs (deploy:*)
DEV_*_KEY, TEST_*_KEY, PILOT_*_KEYBlockchain private keys (CHAIN_GOVERNANCE_PRIVATE_KEY, CHAIN_ADMIN_PRIVATE_KEY, etc.) for chain interactions.Deployment jobs (deploy:*)
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEYAWS S3 storage credentials per environment.Deployment jobs (deploy:*)
JWT_GENERATOR_SIGNATURE_SECRETSecret key for JWT signature generation and verification.Deployment jobs (deploy:*)
CI_REGISTRY_USER, CI_REGISTRY_PASSWORDDocker registry credentials (automatically populated by GitLab CI runner)..docker, build:ci-image

Runner Requirements​

  • Runner Tags: Dedicated tags for build/test jobs and for deployment/GitOps jobs.
  • Docker-in-Docker (DinD): Jobs extending .docker or .gradle-template require access to a Docker daemon via docker:dind service. Data root caching is configured per runner to optimize build speeds.

Developer Permissions​

  • Standard Developers: Require Developer access to push feature branches, open Merge Requests, trigger manual collection generation (generate:bruno-collection), and view pipeline logs.
  • Maintainers / Release Leads: Require Maintainer access to push to master, merge MRs, create release branches (release/*), and execute manual deployment jobs (deploy:test, deploy:demo, deploy:pilot).

Repository Structure & CI Configuration​

.
├── .gitlab-ci.yml                      # Root entrypoint importing sub-configurations
├── ci/
│   ├── defaults.yml                    # Service matrices, global variables, stages & default image
│   ├── rules.yml                       # Workflow rules, inhibitor rules & path-based trigger rules
│   ├── templates.yml                   # Reusable job templates (Gradle, Indexer, Docker build/retag)
│   ├── scheduled-tasks.yml             # Cron tasks (Renovate bot, child pipeline triggers, CI image build)
│   └── Dockerfile                      # Custom base CI runner image (Java 25, Node.js/pnpm, Docker CLI, Helm)
├── deploy/
│   ├── .gitlab-ci.yml                  # Deployment job definitions (ArgoCD GitOps & Docker Compose SSH)
│   ├── chain-config/                   # Environment chain variables (.env.chain-dev, .env.chain-test, etc.)
│   ├── dev/                            # Dev environment compose overrides
│   ├── docker-compose.demo.yml         # Demo environment compose orchestration
│   ├── docker-compose.pilot.yml        # Pilot environment compose orchestration
│   ├── docker-compose.test.yml         # Test environment compose orchestration
│   └── local/                          # Local developer compose setups (local-full, local-db, local-chain)
└── helm-chart/                         # Kubernetes Helm chart for ArgoCD deployments
    ├── Chart.yaml
    ├── values.yaml                     # Default values and container image tags
    └── templates/                      # Kubernetes manifests (CNPG Postgres, Kafka, Ingress, Deployments)

Key Configuration Files​

  • .gitlab-ci.yml: Root configuration importing ci/defaults.yml, ci/rules.yml, ci/templates.yml, ci/scheduled-tasks.yml, and deploy/.gitlab-ci.yml. Defines the pipeline matrix jobs for Helm chart linting, microservice Gradle builds & tests, Indexer code quality, GraphQL schema synchronization, Bruno API collection generation, and container image builds/retagging.

  • ci/defaults.yml: Standardizes service groups (common, core, shared, whitelabel), defines pipeline stages (lint, build-common, test-common, build, test, docker, release), sets default base image (ci:latest), and configures Gradle flags.

  • ci/rules.yml: Manages workflow rules to eliminate duplicate MR pipelines, defines inhibitor rules (preventing scheduled/triggered pipelines from triggering standard builds), and implements change-path filtering (.core_rules, .whitelabel_rules, .indexer_rules, etc.) for incremental CI runs.

  • ci/templates.yml: Contains reusable job templates:

    • .gradle-build-template & .gradle-test-template: Handles Gradle compilation, testing, and artifact extraction.
    • .cq: Sets up Node.js/pnpm environment for indexer linting/testing.
    • .docker: Logs into GitLab registry, builds multi-tagged images using BuildKit, and pushes tags to registry.
    • .docker-retag: Retags pre-built master-latest images for untouched services to save build resources.

  • ci/scheduled-tasks.yml: Scheduled tasks including dependency updates via Renovate, sequential execution of child pipelines for Renovate branches (trigger:services-build), and automated CI base image rebuilds (build:ci-image).

  • deploy/.gitlab-ci.yml: Controls environment deployments via two mechanisms:

    1. GitOps via ArgoCD: argocd:update-static-env and argocd:update-master clone argocd-k8s-dev, update values.yaml with the short SHA, and commit/push changes.
    2. Docker Compose via SSH: .deploy-template:dev, deploy:test, deploy:demo, and deploy:pilot inject environment secrets, apply envsubst, transfer compose files via SSH, and execute docker compose up --detach --force-recreate.

Environments & Deployment​

The system supports two parallel deployment paradigms: GitOps via ArgoCD (Kubernetes) and Docker Compose via SSH.

Environment Overview​

EnvironmentHost / PlatformDeployment MethodTrigger MechanismApproval / Guardrails
Dev (Static Envs)K8s / ArgoCDGitOps (argocd:update-static-env)Push / MR on matched branchAutomated on matching branch name
Dev (Master)K8s / ArgoCDGitOps (argocd:update-master)Push to master branchAutomated upon merge to master
Dev (Legacy SSH)SSH target hostSSH Docker Compose (deploy:service:dev)Manual trigger in pipelineManual action required
TestSSH target hostSSH Docker Compose (deploy:test)release/* branch or manualManual action required
DemoSSH pilot hostSSH Docker Compose (deploy:demo)release/* branch or manualManual action required
PilotSSH pilot hostSSH Docker Compose (deploy:pilot)release/* branch or manualManual action required

Approval Mechanisms & Guardrails​

  1. GitOps Branch Safety: argocd:update-master creates a GitOps commit with ci.skip flags to prevent execution loops.
  2. Manual Approval for Staging/Prod Deployments: All SSH deployment tasks (deploy:test, deploy:demo, deploy:pilot, deploy:service:dev) enforce when: manual. A release lead must explicitly trigger the deployment action in GitLab.
  3. Inhibitor Rules: Prevent scheduled pipelines or child triggers from executing deployment tasks unexpectedly.

CI/CD Workflow​

Pipeline Variants​

1. Merge Request / Feature Branch Pipeline:

  • Path-Based Filtering: Only affected microservices run build and test jobs. If files in common/, gradle/, or ci/ change, all services are tested.
  • GraphQL Schema Auto-Sync: If indexer/schema.graphql changes in an MR, sync:schema_graphqls builds the indexer, runs the schema generator, commits schema.graphqls, and pushes the commit directly back to the MR source branch.
  • Docker Build & Retag: Modified services execute docker-build:*. Untouched services execute docker-retag:*, creating short-SHA tags from master-latest images to avoid redundant rebuilds.

2. Master Branch Pipeline:

  • Executes full compilation and test suites across all core, whitelabel, shared, and indexer modules.
  • Builds Docker images with :master-latest and :${CI_COMMIT_SHORT_SHA} tags.
  • Executes argocd:update-master, committing the updated image short SHA to argocd-k8s-dev to trigger automatic deployment in Kubernetes.

3. Release Branch / Tag Pipeline:

  • Executed on release/* branches or git tags (e.g. user-service@1.2.0).
  • Extracts version tag (${CI_COMMIT_TAG#*@}) and tags Docker images accordingly.
  • Unlocks manual deployment jobs (deploy:test, deploy:demo, deploy:pilot).

4. Scheduled Pipelines (Cron Tasks):

  • renovate: Scans dependencies using renovate.json5 and opens MRs.
  • trigger:services-build: Iterates through open renovate/* branches sequentially, triggering child pipelines via GitLab API and polling status to prevent overloading CI runners.
  • build:ci-image: Rebuilds the base ci:latest runner image from ci/Dockerfile and pushes it to the container registry.

How to Use the System​

Standard Developer Tasks​

Working on a Feature Branch​

  1. Create a feature branch off master (e.g. feature/add-payment-status).
  2. Push your changes and create a Merge Request targeting master.
  3. The CI pipeline will automatically trigger path-filtered test and build jobs.
  4. If modifying indexer/schema.graphql, wait for sync:schema_graphqls to push the updated schema.graphqls back to your branch, then pull locally.

Generating Bruno API Collections​

  1. Go to your MR's pipeline in GitLab CI.
  2. Locate the generate:bruno-collection job in the test stage.
  3. Click Play (manual trigger).
  4. Upon completion, download the job artifact (output/) to retrieve updated Bruno collections.

Administrative & Release Tasks​

Releasing a Version to Staging / Production​

  1. Create a release branch named release/vX.Y.Z or push a tagged commit (e.g. service-name@1.0.0).
  2. Verify that all build, test, and Docker build jobs pass successfully.
  3. Navigate to the release stage in the GitLab CI pipeline UI.
  4. Click the manual play button next to the desired target environment (deploy:test, deploy:demo, or deploy:pilot).

Updating the CI Base Image​

  1. Modify ci/Dockerfile (e.g., to upgrade JDK or pnpm versions).
  2. Push changes to master or trigger the build:ci-image scheduled pipeline manually under CI/CD > Schedules.

Troubleshooting & Rollbacks​

Common Pipeline Failures & Resolutions​

Issue / Error MessageRoot CauseResolution
Cannot connect to the Docker daemon at tcp://docker:2375Docker-in-Docker service failed to initialize or runner storage is full.Check runner disk space. Restart the runner service if persistent.
Testcontainers Ryuk failed to startTestcontainers MappedPort / Ryuk resource container conflict in shared CI runner.Ensure TESTCONTAINERS_RYUK_DISABLED: "true" is retained in ci/defaults.yml.
git push origin HEAD:... failed in sync:schema_graphqlsSCHEMA_SYNC_TOKEN missing, expired, or lacks write permissions.Renew SCHEMA_SYNC_TOKEN in GitLab CI/CD Variables with api and write_repository scopes.
yq: command not found or git clone $ARGO_PROJECT_CLONE_PATH failureARGO_PROJECT_TOKEN invalid or missing access to argocd-k8s-dev repository.Verify ARGO_PROJECT_TOKEN validity and permissions in GitLab CI/CD Variables.
SSH connection timeout during deploy:* jobsTarget host unreachable or SSH key invalid/expired.Check host status and verify $DEV_SSH_KEY, $TEST_SSH_KEY, or $PILOT_SSH_KEY.

Emergency Rollback Procedures​

Option A: Kubernetes / ArgoCD Rollback (Recommended for Dev/Master Envs)​

  1. Clone the argocd-k8s-dev repository.
  2. Revert the latest commit in values.yaml to point back to the previous known good short SHA and push: 
    git revert HEAD
    git push

ArgoCD UI Rollback:

  1. Access the ArgoCD dashboard.
  2. Select the target application (e.g., offchain-master).
  3. Click History and Rollback, select the previous healthy deployment revision, and click Rollback.

Option B: Docker Compose SSH Rollback (Test / Demo / Pilot Envs)​

  1. Re-run Pipeline: Locate the previous successful pipeline for the target release branch in GitLab CI. Navigate to the release stage and click Re-deploy / Play on the corresponding deployment job.

  2. Manual Host Rollback (Emergency):

    ssh deploy@<your-pilot-host>
    # Navigate to environment folder and edit compose file to revert image tags
    docker compose -p offchain-test -f docker-compose.test.yml up -d --force-recreate