GitOps & ArgoCD Infrastructure (argocd-k8s-dev)

The argocd-k8s-dev repository serves as the central GitOps continuous delivery and infrastructure management module for Kubernetes environments. Utilizing the ArgoCD "App of Apps" pattern and ApplicationSets, this module automates the lifecycle management of platform infrastructure services (Istio service mesh, Ceph CSI storage, CloudNativePG, Debezium, Loki logging, Prometheus monitoring, Strimzi Kafka, and Reflector) as well as application workloads (deuss microservices frontend and offchain backend).

Primary Goals

• GitOps-driven Deployment: Maintain declarative Kubernetes cluster states synchronized directly from Git repositories.
• Automated Validation: Ensure syntax and template integrity of Helm charts via pre-commit GitLab CI pipelines.
• Dynamic On-Demand Preview Environments: Provision isolated, ephemeral feature and Merge Request (MR) preview environments on-demand when GitLab MRs are tagged with the DEPLOY label.
• Multi-Tenant Environment Management: Seamlessly manage static long-running environments (sc-100, ebsi-test, master) and infrastructure components through centralized configuration in values.yaml.

---

Prerequisites & Configuration

Required GitLab CI/CD Variables

• CHART_NAMES: Configured in .gitlab-ci.yml (defaults to "."). Defines the directory path(s) for Helm chart linting and templating verification.

Kubernetes Secrets & Credentials

• argocd-repo-creds-gitlab-access-git: A Kubernetes secret located in the argocd namespace containing the password key (GitLab personal/project access token). This credential is required by ArgoCD ApplicationSet PR generators to authenticate against the internal GitLab API for the frontend and offchain projects.
• ArgoCD Vault Plugin (avp-helm-optional-values): Configured within the ArgoCD instance to allow secret injection and custom parameter overrides during Helm template rendering.

Runner Requirements

• GitLab CI Runner: Standard Docker executor runner capable of pulling alpine:latest with outgoing HTTPS access to install helm via apk.
• Network Connectivity: Unrestricted access between ArgoCD controller, internal GitLab instance, and the target Kubernetes API server (https://kubernetes.default.svc).

Developer & Administrative Permissions

• GitLab Permissions: Developer role minimum to submit Merge Requests and add labels (DEPLOY); Maintainer role to push updates to values.yaml on the master branch.
• ArgoCD Access: Read/Sync permissions in ArgoCD UI/CLI for inspecting application sync status and troubleshooting deployment failures.

---

Repository Structure & Key Configurations

File / Location	Description
.gitlab-ci.yml	GitLab CI pipeline configuration. Runs automated syntax validation (helm lint) and dry-run rendering (helm template) on every push.
Chart.yaml	Top-level Helm chart metadata (name: apps, version: 0.1.0) defining the root App-of-Apps chart.
values.yaml	Centralized values file containing image tags, repository URLs, chart versions, and parameter maps for all applications, static environments, and dynamic ApplicationSets.
templates/projects/	ArgoCD AppProject manifests defining access controls, allowed source repositories, and destination namespaces (e.g., deuss.yml, cluster-config.yml, istio.yml, monitoring.yml).
templates/apps/	Root ArgoCD Application manifests templated from values.yaml for tracking static application instances (deuss/fe-master.yaml, deuss/offchain-master.yaml, cluster-config/config.yml, csi/cephfs.yml, etc.).
templates/app-sets/deuss/	ArgoCD ApplicationSet specifications for dynamic and static multi-environment deployments (fe-mr.yaml, offchain-mr.yaml, fe-static-envs.yaml, offchain-static-envs.yaml).
directory-apps/	Directory containing raw Kubernetes manifests (e.g., cluster-config/ingress.yml, debezium-operator/limit-range.yml) managed via ArgoCD Directory source type.

note

This repository is dedicated exclusively to Helm/ArgoCD GitOps manifest delivery. Container images are built and pushed to the container registry by upstream application repositories (frontend.git and offchain.git).

---

Environments & Deployment

Environment Matrix

1. Dynamic Preview Environments (Dev / Feature MRs):

   • Namespaces: deuss-fe-<MR_ID>, deuss-offchain-<MR_ID> (e.g., deuss-fe-42)
   • Target Cluster: https://kubernetes.default.svc
   • Lifecycle: Ephemeral. Created automatically when a Merge Request in frontend or offchain is assigned the DEPLOY label. Destroyed automatically when the MR is closed/merged.

2. Static Environments (Stage / Integration):

   • Namespaces: deuss-fe-<env_name>, deuss-offchain-<env_name> (e.g., sc-100, ebsi-test)
   • Target Cluster: https://kubernetes.default.svc
   • Lifecycle: Persistent. Parameterized and declared under projects.deuss.app-sets.*.elements in values.yaml.

3. Master / Production Workloads:

   • Namespaces: deuss-fe-master, deuss-offchain-master, infrastructure namespaces (kube-system, istio-system, monitoring, loki, etc.)
   • Target Cluster: https://kubernetes.default.svc
   • Lifecycle: Persistent long-running core services tracking specified image tags or release versions.

Sync Policies & Approval Mechanisms

• Automated GitOps Synchronization: All ArgoCD Applications and ApplicationSets enforce automated sync with prune: true and selfHeal: true.
• Approval Mechanisms:
  • MR Preview Deployments: Require developer approval to attach the DEPLOY label on the GitLab Merge Request.
  • Static & Master Deployments: Require Git Merge Request approval to update values.yaml on the master branch.

---

Workflow

Pipeline Variants & Triggers

1. Validation Pipeline (GitLab CI):

   • Trigger: Any push or merge request against argocd-k8s-dev.
   • Execution: Runs helm-lint-and-template job inside alpine:latest. Installs helm, lints the root chart, and verifies template rendering.

2. Dynamic Preview Deployment (ArgoCD PR Generator):

   • Trigger: Developer creates an MR in frontend.git or offchain.git and adds label DEPLOY.
   • Execution: The ArgoCD ApplicationSet generator polls GitLab every 15 seconds. Once detected, it creates a dedicated Application rendering Helm with --set global.environmentName=mr-<number> and --set global.serviceDefaults.image.tag=<head_short_sha>.

3. Static Environment Release:

   • Trigger: Commit to values.yaml updating branch or imageTag under app-sets.<name>.elements.
   • Execution: ArgoCD detects Git state change and updates deployment pods in target static namespaces (ebsi-test, sc-100).

4. Master / Infrastructure Release:

   • Trigger: Commit updating imageTag or chartVersion in values.yaml.
   • Execution: ArgoCD automatically syncs target Application resources across the cluster.

---

How to Use the System

Triggering an MR Preview Environment

1. Open a Merge Request in frontend.git or offchain.git.
2. Add the label DEPLOY to the Merge Request.
3. ArgoCD will automatically detect the MR within 15 seconds and provision namespace deuss-fe-<MR_NUMBER> or deuss-offchain-<MR_NUMBER>.
4. To modify specific environment variables for this MR, commit a custom values file at values/<MR_NUMBER>.yaml in your application repository.

Updating Image Tags for Static Environments (ebsi-test, sc-100)

1. Edit values.yaml in argocd-k8s-dev.
2. Locate the environment under projects.deuss.app-sets.offchain-static-envs.elements or fe-static-envs.elements:

   ebsi-test:
     branch: "484-ebsi-gateway"
     imageTag: "NEW_IMAGE_SHA"

3. Commit and push your changes to master. ArgoCD will synchronize the updated image.

Adding a New Infrastructure Helm Chart

1. Add chart details under the appropriate project key in values.yaml.
2. Create an AppProject manifest under templates/projects/<project_name>.yml if creating a new tenant project.
3. Add the Application manifest under templates/apps/<project_name>/<app_name>.yaml.
4. Run local validation:

   helm lint .
   helm template .

5. Push changes to master.

---

Troubleshooting & Rollbacks

Common Pipeline & Deployment Failures

Issue	Root Cause	Resolution
GitLab CI helm-lint-and-template failure	Syntax errors or missing required Helm values in values.yaml.	Run helm lint . locally. Fix broken YAML indentation or missing references.
MR Environment not created	Missing DEPLOY label or closed MR status.	Verify MR state is opened and label DEPLOY is attached. Inspect ArgoCD ApplicationSet controller logs.
ArgoCD Sync ComparisonError / Git Auth Failure	Expired or missing argocd-repo-creds-gitlab-access-git secret.	Re-create the secret in the argocd namespace with a valid GitLab Personal Access Token.
AVP Plugin Error (avp-helm-optional-values)	Missing secret values or broken extra values file path.	Verify EXTRA_VALUES_FILE exists in the application repo or is handled gracefully by AVP.

Rollback Procedures

Standard GitOps Rollback

1. Identify the last known stable commit SHA in argocd-k8s-dev.
2. Revert the commit in Git:

   git revert HEAD
   git push origin master

3. ArgoCD will detect the updated Git revision and automatically roll back cluster resources to the previous state.

Immediate Emergency Rollback (ArgoCD UI / CLI)

1. Access ArgoCD UI or CLI.
2. Select the failing Application (e.g., deuss-fe-master).
3. Click Sync Options → Disable Automated Sync.
4. Perform a manual Sync targeting the previous Git revision SHA.
5. Revert the git repository to match the deployed state as soon as possible to avoid configuration drift when re-enabling auto-sync.