Skip to main content

IDA Stack CI/CD (ida-stack)

The IDA (Internal Diligence Agent) Stack is a multi-repository distributed system designed to monitor, ingest, decode, analyze, and visualize transactions for financial compliance and fraud detection. The system consists of 5 core repositories:

  1. ida-deployment: Central CI/CD orchestrator, Docker Swarm / Docker Compose stack configurations, environment setups, and deployment manifests.
  2. ida-backend: TypeScript / Node.js core server for transaction feed ingestion, journal event storage (MongoDB), and detector routing.
  3. ida-detector: Python / FastAPI fraud detection engine (aml-detector) running evaluation algorithms and returning alerts to the backend.
  4. ida-source-pi: Scenario simulation driver and replay server (source-pi-replay) serving pre-recorded DEUSS runs and synthetic patterns.
  5. ida-admin: Next.js standalone web dashboard providing real-time UI monitoring of events, alerts, feeds, and detector states.

Primary Goals​

  • Centralized Orchestration: Using ida-deployment to control build dependency ordering, global versioning, and environment deployments across all 5 sub-projects.
  • Unified Pipeline Tagging: Assigning a single pipeline tag (DEPLOY_TAG=$CI_PIPELINE_ID) to ensure all 4 microservices are built, tagged, and deployed in complete lockstep.
  • Automated Containerization & Registry Management: Building OCI-compliant container images for each microservice and publishing them to the GitLab Container Registry.
  • Declarative & Safe Deployment: Deploying containerized stacks via Docker Swarm (docker stack deploy) onto remote target hosts with automated health checks and manual approval gates for demo and production releases.

Prerequisites & Configuration​

Required GitLab CI/CD Variables​

Variable NameScope / TargetDescription
GROUP_PATHOrchestratorGroup path on GitLab containing all 5 repositories (e.g., bebi/ai-agents)
DEPLOY_USERDeployment JobSSH user authorized to connect to target deployment host
DEPLOY_HOSTDeployment JobDomain name or IP of the target deployment host
SSH_KEYDeployment JobPrivate SSH key (masked/protected) used to authenticate deployment tasks via SSH/SCP
CI_REGISTRYAll RepositoriesGitLab Container Registry endpoint
CI_REGISTRY_USERAll RepositoriesUsername for GitLab Container Registry authentication
CI_REGISTRY_PASSWORDAll RepositoriesPassword / Access Token for GitLab Container Registry
IMAGE_TAGDownstream JobsDynamically passed down from ida-deployment to sub-pipelines (inherited from $DEPLOY_TAG)
APP_ENVida-admin BuildEnvironment name passed into Next.js build-time injection (e.g., development)

Runner Requirements​

  • Orchestrator Runner Tag: Dedicated runner tag for the orchestrator pipeline.
  • Orchestrator Base Image: Alpine-based CI image with utility tools (openssh-client-default, envsubst, dos2unix).
  • Component Build Image: Docker-in-Docker (docker:24.0.5 base image with docker:24.0.5-dind service) enabled for building container images.
  • Network Access: Runners must have outbound network access to the target host (DEPLOY_HOST) on SSH port 22 and access to the GitLab Container Registry.

Required Developer & Administrative Permissions​

  • Developer Access: Permission to create branches, open Merge Requests, and trigger manual pipelines in component repositories.
  • Maintainer / Owner Access: Required to trigger the deploy-demo manual stage in ida-deployment, manage CI/CD secret variables (SSH_KEY), and push directly to protected branches (master).
  • Target Host Access: SSH public key corresponding to $SSH_KEY must be placed in ~/.ssh/authorized_keys for DEPLOY_USER on DEPLOY_HOST.

Repository Structure​

<workspace>/
├── ida-deployment/                ← Central Orchestrator Repository
│   ├── .gitlab-ci.yml             ← Master pipeline definition (setup, build triggers, deploy)
│   ├── demo/
│   │   ├── docker-compose-ida.yml ← Production/Demo Docker Swarm compose template (${DEPLOY_TAG})
│   │   ├── config/
│   │   │   └── ida-backend.jsonc  ← Demo-specific backend feed & storage configuration
│   │   └── README.md              ← Demo environment documentation
│   ├── demo-local/
│   │   └── docker-compose.yml     ← Local checkout-based compose stack (no registry pull)
│   └── development/
│       ├── docker-compose.yml     ← Local live-chain dev stack (Anvil EVM + local builds)
│       └── docker-compose.jsonreplay.yml ← Local JSON fixture replay overlay
├── ida-backend/                   ← Microservice 1: Backend API & Ingest
│   └── .gitlab-ci.yml             ← Container build & registry push stage
├── ida-detector/                  ← Microservice 2: Python AML Detector
│   └── .gitlab-ci.yml             ← Container build & registry push stage
├── ida-source-pi/                 ← Microservice 3: Replay Server & Scenario Generator
│   └── .gitlab-ci.yml             ← Packaging scenarios/demo-scenarios.zip & container build
└── ida-admin/                     ← Microservice 4: Next.js Frontend Dashboard
    └── .gitlab-ci.yml             ← Next.js standalone container build

Detailed Configuration Breakdown​

ida-deployment/.gitlab-ci.yml (Master Pipeline):

  • setup stage (generate-tag): Creates a build.env dotenv artifact setting DEPLOY_TAG=$CI_PIPELINE_ID.
  • build stage (build-backend, build-detector, build-source-pi, build-admin): Uses GitLab trigger syntax to invoke downstream pipelines in sibling repositories with strategy: depend. Passes IMAGE_TAG=$DEPLOY_TAG to all downstream jobs.
  • deploy stage (deploy-demo): Manually triggered on master branch. Copies ./demo directory to DEPLOY_HOST, substitutes ${DEPLOY_TAG} inside docker-compose-ida.yml using envsubst, and triggers docker stack deploy on the host.

Component .gitlab-ci.yml files (ida-backend, ida-detector, ida-source-pi, ida-admin):

  • Each repository contains a single build stage running Docker-in-Docker.
  • Authenticates to $CI_REGISTRY via docker login.
  • Tags built images with both $IMAGE_TAG (the unique pipeline ID) and latest.
  • Pushes images to the GitLab Container Registry.

demo/docker-compose-ida.yml (Target Deployment Manifest):

  • Configures the 5 container services: mongo, ida-backend, ida-detector, source-pi-replay, ida-admin.
  • References images dynamically using ${DEPLOY_TAG}.
  • Sets healthcheck probes (mongosh, fetch, urllib.request) for every service.
  • Configures container networking and port mappings (8090, 8091, 9100, 3010).

Environments & Deployment​

EnvironmentHost / LocationImage SourcePrimary PurposeApproval Mechanism
Development (development/)Local Developer MachineLocal docker compose buildFull feature development with live in-memory Anvil EVM chain simulatorNone
Demo Local (demo-local/)Local Developer MachineLocal docker compose buildChain-free local testing of replay scenario feeds and UINone
Demo / Pilot (demo/)Remote pilot hostGitLab Container Registry (:${DEPLOY_TAG})Chain-free public pilot demonstration running recorded scenario replaysManual Gate: Required trigger on deploy-demo in ida-deployment master branch
TestnetRemote Target HostGitLab Container RegistryMulti-node testnet integration testingTBD
ProductionProduction ClusterGitLab Container RegistryProduction rolloutTBD

Approval Mechanisms & Deployment Gates​

  1. Pipeline Synchronization Gate (strategy: depend): The orchestrator pipeline forces all 4 downstream component build pipelines to complete successfully before allowing the deployment stage to unlock.
  2. Branch Protection & Manual Execution: Deployment to the demo environment is restricted to commits on the master branch of ida-deployment and requires an explicit Manual Action (when: manual) executed by an authorized maintainer.

CI/CD Workflow​

Pipeline Variants & Execution Triggers​

  1. Orchestrated Master Release Pipeline:

    • Trigger: Commit or Merge Request merge into master branch of ida-deployment.
    • Flow: generate-tag → Parallel downstream builds → Manual deploy-demo gate → Swarm Deployment.

  2. Component Repository Pipelines:

    • Trigger: Direct commit or merge inside ida-backend, ida-detector, ida-source-pi, or ida-admin.
    • Flow: Builds container image tagged with commit/pipeline tag and :latest for feature branch testing or standalone verification.

  3. Bugfix & Feature Branch Pipelines:

    • Developers create feature branches (e.g. fix/detector-scoring). Downstream builds can be tested independently without altering the published demo stack tag until merged into master.

How to Use the System​

Standard Developer Tasks​

Running the Full Live Stack Locally​

cd ida-deployment/development
docker compose up -d --build
# Access Admin GUI at http://localhost:3000

Working on a Single Microservice (e.g. ida-backend)​

  1. Create a feature branch in ida-backend (e.g., feature/new-feed-parser).
  2. Test locally using demo-local or development compose stacks.
  3. Commit and push your feature branch. The ida-backend .gitlab-ci.yml will automatically build and publish a test container image.
  4. Create a Merge Request to master in ida-backend.

Updating Scenario Recordings or Fixtures​

  1. Add or modify scenario JSON files in ida-source-pi/scenarios/.
  2. Package updated recordings into scenarios/demo-scenarios.zip.
  3. Commit changes to ida-source-pi. When merged, the next orchestrator run will bake the new zip into the ida-source-pi image.

Administrative & Release Tasks​

Triggering a Deployment to the Demo Environment​

  1. Ensure all component code changes are merged into their respective master branches.
  2. Navigate to ida-deployment in GitLab and push to or merge into master branch.
  3. Monitor the setup and build stages — confirm build-backend, build-detector, build-source-pi, and build-admin jobs pass.
  4. In the GitLab CI UI, click on the deploy-demo action in the deploy stage and click Play (Run).
  5. Check the pipeline logs to verify SCP transfer, envsubst expansion, and successful execution of docker stack deploy.

Inspecting the Live Demo Stack on the Target Server​

ssh deploy@<your-pilot-host>
docker stack ps ida-demo
docker service logs -f ida-demo_ida-backend

Troubleshooting & Rollbacks​

Common Pipeline Failures & Resolutions​

Failure SymptomCauseResolution
build-* downstream trigger job failsCompilation or test failure in component repositoryClick into the child pipeline link in GitLab UI to inspect component build logs. Fix error in component repo and retry.
deploy-demo fails with Permission denied (publickey)SSH key mismatch or missing $SSH_KEY variableEnsure SSH_KEY is present in ida-deployment CI/CD variables and that the public key is registered in ~/.ssh/authorized_keys for DEPLOY_USER on DEPLOY_HOST.
docker stack deploy image pull error on hostDocker host cannot authenticate to GitLab RegistryVerify $DEPLOY_USER on host has run docker login <registry-domain> or ensure --with-registry-auth flag is supplied.
Container stuck in starting / unhealthy stateContainer health check failing after startupCheck service logs on host: docker service logs ida-demo_<service-name>. Verify MongoDB ping response or HTTP health check endpoints.

Rollback Procedures​

Method 1: GitLab CI One-Click Pipeline Rollback​

  1. Open the ida-deployment repository in GitLab and go to CI/CD → Pipelines.
  2. Locate the last known working pipeline on master (note its Pipeline ID, e.g. 78451).
  3. Click on the pipeline details to open the stage view.
  4. Locate the deploy-demo job in the deploy stage of that historical pipeline.
  5. Click Retry / Play — the orchestrator will re-run the deployment using DEPLOY_TAG=78451, instantly deploying the previously verified container images.

Method 2: Manual Emergency Command-Line Rollback​

ssh deploy@<your-pilot-host>

# Identify the previously working pipeline ID
docker image ls | grep ida-backend

# Deploy with the known-good tag
cd ./ida-deployment
export DEPLOY_TAG=78451
cat ./demo/docker-compose-ida.yml | envsubst | docker stack deploy --prune --with-registry-auth --compose-file - ida-demo

# Verify rollback state
docker stack ps ida-demo