App of Apps: GitOps Architecture for Hub-of-Hubs

This page explains why a Mode 2 hub-of-hubs deployment benefits from decomposing the monolithic Helm chart into an ArgoCD App of Apps pattern with Kustomize composition and Helm templating where each tool fits best.

This is architecture guidance, not a lab exercise. It documents the rationale behind the target-state GitOps structure for multi-hub deployments at fleet scale.

The problem: one chart, many roles

Your Mode 2 deployment has four OpenShift clusters. Each plays a different role:

  • Global Hub — ACM governance, Global Hub Operator, fleet observability (Thanos Query federation). No AMQ Broker, no Keycloak, no workshop.

  • Regional Hubs (east, central, west) — AMQ hub brokers, Keycloak OIDC, regional MCO observability, ACM management of local SNOs, Showroom workshop.

Today, a single Helm chart (artemis-edge-acm-demo, 39 templates, 7 sync waves) is deployed as one ArgoCD Application (field-content). The chart uses conditional blocks to switch behavior:

{{- if .Values.globalHub.enabled }}       # global hub only
{{- if .Values.keycloak.enabled }}        # regional hubs only
{{- if .Values.spokeProvisioning.enabled }} # regional hubs only
{{- if .Values.amqBrokerOperator.enabled }} # regional hubs only

This creates four problems at scale:

Problem Impact

CRD ordering failures

Operators install at wave 1. CRDs take minutes to register. A wait-for-crds Job hack at wave 3 polls until CRDs appear, but ArgoCD exhausts its retry budget before the job completes. The entire sync fails.

No independent lifecycle

Updating a Grafana dashboard resyncs all 39 resources — brokers, Keycloak, policies. A small observability change risks disrupting messaging.

Mode divergence in conditionals

The global hub and regional hubs share almost no resources. Branching with if/else makes one chart behave like five different charts. This is composition, not templating.

Scale ceiling

Scale Considerations documents Mode 2 for >2,000 clusters and PolicyGenerator for fleet parameterization. PolicyGenerator is a Kustomize plugin — it cannot consume Helm templates directly.

The solution: App of Apps with domain separation

ArgoCD’s App of Apps pattern replaces the single field-content Application with a root Application that renders child Applications — one per operational domain.

Three-layer GitOps architecture

App of Apps three-layer architecture
Figure 1. Mode 2 three-layer GitOps: root composition, domain apps, and inter-cluster fleet — click to enlarge
Layer details (Mode 2)
Layer 1 — Root App of Apps (Kustomize)
  Selects WHICH child apps deploy on each cluster role.
  apps/overlays/global-hub/       → operators (subset) + observability (fleet-tier)
  apps/overlays/regional-hub/     → operators + messaging + auth + observability + workshop

Layer 2 — Domain Apps (Kustomize or Helm per domain)
  Each child app manages ONE concern:
  components/operators/            Kustomize — static Subscriptions, Namespaces
  components/messaging/            Helm      — broker CRs (range loops over broker list)
  components/auth/                 Helm      — Keycloak CR, realm import, secrets
  components/observability/        Kustomize — dashboard CM, allowlist, two-tier overlays
  components/workshop/             Helm      — Showroom, per-user RBAC

Layer 3 — Inter-Cluster Fleet (Kustomize + ApplicationSets) [already exists]
  fleet-gitops/applicationsets/    Global Hub → pushes config to regional hubs
  fleet-gitops/hubs/               Regional Hub → pulls broker config to SNOs
  fleet-gitops/amq/                Site broker manifests for SNO edge clusters

How CRD ordering works without hacks

The current chart uses a wait-for-crds Job (sync wave 3) that polls until AMQ Broker and Keycloak CRDs are registered. This is fragile — the Job has a backoff limit and active deadline, and ArgoCD has a separate retry budget. When either runs out, the entire sync fails.

With App of Apps, ordering is built in:

# Child Application at wave 0 — installs operators
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: operators
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  source:
    path: components/operators/base

# Child Application at wave 1 — creates broker CRs
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: messaging
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  source:
    path: components/messaging

ArgoCD waits for the operators Application to reach Healthy status (meaning the operators are installed and their CRDs are registered) before it syncs messaging. No Job hack. No retry budget race. The ordering is declarative.

Kustomize vs Helm: which tool where

The design principle: Kustomize for composition and static resources; Helm for dynamic resources that need loops or conditionals.

Component Tool Rationale

apps/ (root composition)

Kustomize

Selects which child Applications exist per cluster role. Overlays add/remove entire Application CRs.

components/operators/

Kustomize

Namespaces, OperatorGroups, and Subscriptions are static YAML. Overlays include or exclude operators per role (AMQ Broker on regional hubs, not on global hub).

components/messaging/

Helm

Broker CRs iterate over a list (range .Values.hubBrokers). Broker count, names, connection topology, and address patterns are all parameterized. Kustomize cannot generate N resources from a list.

components/auth/

Helm

Keycloak realm import contains templated passwords and client secrets. TLS conditionals. Kustomize secretGenerator cannot handle the realm import JSON.

components/observability/

Kustomize

Dashboard ConfigMap and metrics allowlist are static. Two-tier overlays (regional vs fleet) select which monitoring scope applies. Scale Considerations discusses metric cardinality — tiered observability keeps each regional Thanos handling local scrape volume.

components/workshop/

Helm

Per-user RBAC generation (range .Values.workshop.users). Showroom configuration with templated URLs.

fleet-gitops/

Kustomize

Already uses this pattern. ApplicationSets and ACM Policies are static manifests with region overlays.

ztp/policygenerator/

Kustomize

PolicyGenerator is a Kustomize plugin. With domain components extracted, it can reference components/messaging/ directly for fleet-scale policy generation.

Why not pure Kustomize?

Generating three broker CRs from a list requires copy-paste or external generators. Helm’s range is cleaner:

{{- range .Values.hubBrokers }}
apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: {{ .name }}-broker
spec:
  deploymentPlan:
    size: {{ .replicas }}
{{- end }}

Per-user RBAC, conditional OIDC JAAS configuration, and Keycloak realm import all benefit from Helm templating. Kustomize strategic merge patches become complex when the objects themselves are dynamic.

Why not pure Helm?

The monolithic chart is already pure Helm — and that is the root cause of the problems listed above. Specifically:

  • Composition via conditionals is an anti-pattern. When if .Values.globalHub.enabled removes 80% of the templates, you have two charts masquerading as one. Kustomize overlays express this cleanly: the global hub overlay simply does not include app-messaging.yaml.

  • Sync waves across 39 resources in one ArgoCD Application create the CRD ordering problem. Splitting into child Applications gives ArgoCD native health-based ordering.

  • PolicyGenerator cannot consume Helm. The ztp/policygenerator/ scaffold (see PolicyGenerator at Scale) needs plain YAML or Kustomize bases as input manifests.

How it maps to RHACM governance

The App of Apps boundaries align with RHACM governance boundaries. Each tier manages its own ArgoCD root app and its own ACM scope:

Mode 2 ArgoCD + RHACM boundary alignment
GLOBAL HUB                              REGIONAL HUB (east/central/west)
ArgoCD root:                            ArgoCD root:
  apps/overlays/global-hub/               apps/overlays/regional-hub/
  ├── app-operators (subset)              ├── app-operators
  └── app-observability (fleet-tier)      ├── app-messaging
                                          ├── app-auth
ACM scope:                                ├── app-observability (regional-tier)
  fleet-gitops/hubs/ → push to           └── app-workshop
    regional hubs
  fleet-gitops/platform/ → policies      ACM scope:
    for managed hubs                       fleet-gitops/amq/ → pull to SNOs
                                           ACM Policies → enforce operator +
                                             broker config on managed SNOs

The inter-cluster layer (fleet-gitops/) does not change. The hubs ApplicationSet on the global hub still pushes Kustomize overlays to each regional hub. The amq-fleet ApplicationSet on each regional hub still pulls broker manifests to SNOs via ACM Placement.

Observability tiering

Scale Considerations notes that at 1,800 clusters scraping 6 AMQ metrics every 30 seconds, metric cardinality becomes significant. The App of Apps pattern enables two observability tiers via Kustomize overlays:

  • Regional tier (components/observability/overlays/regional-tier/): ServiceMonitors scrape the local hub broker. MCO aggregates metrics from local SNOs. Each regional Grafana shows only its own clusters.

  • Fleet tier (components/observability/overlays/fleet-tier/): Thanos Query on the global hub federates to regional Thanos gRPC endpoints. A fleet dashboard aggregates artemis_* metrics across all regions without duplicating the scrape load.

This keeps each regional Thanos instance handling local scrape volume. Only aggregated queries flow to the global hub — the same pattern described in the Module 8 background section.

Connection to existing patterns

This architecture does not invent new concepts. It extends patterns already proven in this repository:

Existing pattern How App of Apps extends it

fleet-gitops/ Kustomize + ApplicationSets

Already an App of Apps for the inter-cluster layer. The refactor applies the same approach to the intra-cluster layer.

ztp/policygenerator/ scaffold

PolicyGenerator at Scale shows how Kustomize bases become ACM policy sources. Extracting components/messaging/ means PolicyGenerator can reference broker manifests directly.

Git tag promotion (fleet-east, fleet-central, fleet-west)

Region-specific Kustomize overlays in apps/overlays/regional-hub-east/ use the same promotion model.

RHACM fromClusterClaim templating

Per-site broker parameters resolve from managed cluster metadata at policy evaluation time, not from a central values.yaml. This scales to 3,500 SNOs (ACM 2.17 tested limit).

Target directory layout

apps/                                 Root App of Apps (Kustomize)
  base/
    kustomization.yaml
    app-operators.yaml
    app-messaging.yaml
    app-auth.yaml
    app-observability.yaml
    app-workshop.yaml
  overlays/
    mode1-hub/                        Single hub (all 5 apps)
    global-hub/                       Operators (subset) + observability (fleet)
    regional-hub/                     All 5 apps (regional overlays)
    regional-hub-east/                Region-specific messaging values
    regional-hub-central/
    regional-hub-west/

components/                           Domain Kustomize bases / Helm charts
  operators/   (Kustomize)            Namespaces, OGs, Subscriptions
  messaging/   (Helm)                 Hub + edge broker CRs, OIDC, certs
  auth/        (Helm)                 Keycloak, PostgreSQL, realm, secrets
  observability/ (Kustomize)          Dashboard, allowlist, tiered overlays
  workshop/    (Helm)                 Showroom, user RBAC, userinfo

fleet-gitops/                         Inter-cluster GitOps (unchanged)
  applicationsets/                    Global Hub → regional hubs
  hubs/                               Regional Hub → SNOs
  amq/                                Site broker manifests
  platform/                           ACM operator + UWM policies

ztp/policygenerator/                  Fleet-scale policy generation (enhanced)

Further reading