Skip to main content

High Availability

This document describes the target (to-be) High Availability architecture for the DEUSS platform. It represents the intended design for production deployment and serves as the architectural reference for infrastructure planning and implementation.

The target architecture is designed for continuous availability with no single point of failure at either the infrastructure or the application layer. The HA design is based on a dual-site deployment model with active-active or active-passive configuration per component tier, as detailed in the sections below.

1. Dual-Site Deployment Model​

The platform is designed to be deployed across two geographically separated data center locations within the EU. Both sites should run an independent full-stack deployment of all DEUSS off-chain components.

Site topology​

Load balancer (F5)​

A global load balancer (e.g. F5 BIG-IP or equivalent) sits in front of both sites and handles:

  • Health monitoring of each site's ingress endpoints
  • Automatic failover — in the event of a full site outage, traffic is routed exclusively to the surviving site within seconds
  • Traffic distribution — in normal operation, traffic can be split across both sites (active-active) or directed to the primary site with the secondary on standby (active-passive), depending on the operational model selected during deployment

DNS-based failover (e.g. via low-TTL DNS records or Anycast routing) may be used as a complementary or alternative mechanism depending on the infrastructure provider.

AKS cluster per site​

Each site should run a fully independent Azure Kubernetes Service (AKS) cluster. Key design properties:

  • All DEUSS components should be deployed to the cluster using the standard Helm + ArgoCD GitOps pipeline (same as single-site deployments)
  • Namespaces and their isolation boundaries should be identical across both sites
  • Clusters should be independently managed through Rancher — there should be no cross-site Kubernetes control-plane dependency
  • Persistent volumes and stateful workloads should be managed independently per site (see per-component HA sections below)

2. Application Layer HA​

All DEUSS microservices are designed as stateless workloads and should be deployed with a minimum of two pod replicas per service across multiple AKS nodes. The following Kubernetes-native mechanisms are proposed to provide application-level resilience:

MechanismDescription
Pod replica count ≥ 2Survives a single-node failure within a site
Pod Disruption Budgets (PDB)Ensures minimum available pods during node maintenance or eviction
Liveness & readiness probesUnhealthy pods are removed from load balancing and restarted automatically
Horizontal Pod Autoscaler (HPA)Scales out under load; scales back after traffic normalizes
Anti-affinity rulesPods scheduled across different nodes to avoid co-location

3. PostgreSQL HA​

PostgreSQL should be operated using the CloudNativePG Kubernetes operator, which provides Kubernetes-native cluster management including primary election, streaming replication, and automated failover.

Within a single site​

  • A PostgreSQL cluster with one primary and at least one synchronous replica should be provisioned per site
  • Failover within the site should be automatic — if the primary becomes unavailable, the replica is promoted by the CloudNativePG operator without manual intervention
  • Read replicas may optionally be added for reporting or analytics workloads

Cross-site replication — active-passive​

The proposed cross-site operating mode for PostgreSQL is active-passive: Site A hosts the active primary; Site B maintains a warm standby.

  • A read replica (or a standby cluster) in Site B should asynchronously replicate data from Site A's primary
  • In a site-failure scenario, the standby in Site B can be promoted to primary following a controlled switchover procedure
  • Recovery Point Objective (RPO) and Recovery Time Objective (RTO) depend on replication lag and the switchover automation level configured during deployment

Backup​

  • Continuous WAL archiving to S3-compatible storage (Ceph) with point-in-time recovery (PITR) support should be enabled
  • Daily base backups with configurable retention period are recommended
  • Backup integrity should be verified periodically through automated restore tests

4. Kafka HA​

Kafka should be deployed using the Strimzi operator running inside the AKS cluster.

Within a single site​

  • Kafka brokers should be deployed as a multi-broker cluster (minimum 3 brokers)
  • Topics should be configured with a replication factor of 3 and min.insync.replicas = 2
  • ZooKeeper (or KRaft mode for newer Strimzi versions) quorum should be deployed with a minimum of 3 nodes
  • A single broker failure should be transparent to producers and consumers — replication ensures no message loss

Cross-site considerations​

  • Kafka clusters at each site should operate independently — stretching a single Kafka cluster across sites is not recommended due to latency and operational complexity
  • If cross-site Kafka replication is required (e.g. for DR), MirrorMaker 2 (included with Strimzi) can be deployed to asynchronously replicate selected topics from Site A to Site B
  • Consumer offsets and topic configurations should be replicated alongside data to support failover without offset reset

Recovery​

  • In the event of a full site failure, consumers on the surviving site should resume from the last committed offset
  • Producers should reconnect to the surviving site automatically via the load balancer (F5) or DNS failover

5. S3 Object Storage HA (Ceph)​

Object storage should be provided by Ceph deployed as an S3-compatible service within or adjacent to the AKS cluster.

Within a single site​

  • Ceph should be deployed as a multi-node cluster with a minimum of 3 OSD (Object Storage Daemon) nodes
  • Data should be stored with a replication factor of 3 (or erasure coding for cost efficiency on large buckets)
  • Loss of a single OSD node should not result in data loss or service interruption; Ceph rebalances automatically

Cross-site replication — active-active​

The proposed cross-site operating mode for S3/Ceph object storage is active-active: both sites independently serve read and write requests without WAN dependency during normal operation.

  • Ceph RGW Multi-Site (or S3-compatible replication) should synchronize bucket contents between Site A and Site B
  • Replication is asynchronous; RPO depends on replication lag, which is typically in the range of seconds to low minutes
  • Each site's Ceph cluster should be capable of serving read and write requests independently — applications should not need to cross the WAN for object access during normal operation
  • In a site failure, the surviving site's Ceph cluster serves as the sole data source; no switchover procedure is required for read access

Backup​

  • Periodic snapshots of critical buckets should be stored to an independent offsite backup target
  • Distributor/Validator buckets with regulatory retention requirements should use object lock (WORM) where supported

6. Identity and Access Management (Keycloak) HA​

Keycloak should be deployed in clustered mode using the Keycloak Operator or Helm chart with:

  • Multiple Keycloak pods distributed across nodes within each site
  • A shared PostgreSQL backend (reusing the CloudNativePG HA cluster described above)
  • Active sessions should be synchronized across pods via Infinispan / JGroups within the site
  • Cross-site: each site should run its own Keycloak cluster; session synchronization between sites is optional and configurable via Infinispan cross-site replication

7. Observability and Failover Monitoring​

The Grafana / Prometheus / Loki stack should be deployed independently in each site. A cross-site observability layer (e.g. centralized Grafana with remote-read from both sites) is recommended to provide a unified view of cluster health during a site-failure event.

Key alerts should be configured for:

  • Site-level health (ingress unreachable, node not ready)
  • Replication lag for PostgreSQL, Kafka, and Ceph exceeding defined thresholds
  • Pod availability drops below PDB-defined minimums
  • Load balancer failover events

8. RTO / RPO Summary​

ComponentFailure scenarioRPORTO
AKS workloads (stateless)Pod/node failure within a site0Seconds (Kubernetes self-healing)
AKS workloads (stateless)Full site failure0Seconds to minutes (LB failover)
PostgreSQLPrimary failure within site0 (synchronous replica)Seconds (automatic CloudNativePG failover)
PostgreSQLFull site failureLow (async replication lag)Minutes (manual or scripted switchover)
KafkaBroker failure within site0 (replication factor ≥ 3)Seconds (automatic leader election)
KafkaFull site failureLow (MirrorMaker 2 lag)Minutes (consumer reconnect + offset catch-up)
S3 / CephOSD failure within site0 (replication factor ≥ 3)Seconds (automatic Ceph rebalance)
S3 / CephFull site failureLow (RGW multi-site sync lag)Seconds (surviving site serves directly)
KeycloakPod failure within site0 (clustered session replication)Seconds
KeycloakFull site failureLow to medium (cross-site sync optional)Minutes