Module 2: Provision SNOs and Observe ACM Edge Brokers

Objectives

  • Confirm SNO edge clusters are JOINED (provision with Hive only if they are missing)

  • See RHACM Governance install AMQ on those SNOs (edge-broker-policies)

  • Confirm ztp-policies stays unbound (no ApplicationSet children)

Background

Hive (via RHACM) can provision SNO edge clusters on GCP. After a cluster joins with labels sites: edge-sno, fleet: amq, and edge-region: NY|NJ|CT, ACM ConfigurationPolicies in namespace edge-broker-policies install the AMQ Broker operator and one ActiveMQArtemis per region. Those SNO brokers federate to hub-01 over AMQPS.

You are adding a SNO under regional ACM east; it never talks to Global Hub. After join, Global Hub inventory lists the site under this managed hub. Federation is to the AMQ hub broker on this regional ACM hub (hub-01, address classes 5603–5607). Preview of the required produce path (messages.NY.5604 / messages.NY.5603) is Module 4 and Module 5.

NY and NJ site brokers federate over AMQPS to AMQ hub-01 on regional ACM east. messages.NY.5604 reaches the AMQ hub; messages.NY.5603 stays on the SNO. Global Hub has no AMQ broker.
Figure 1. Mode 2 messaging — site brokers federate to AMQ hub-01 on east
On-premises bare-metal provisioning

This lab provisions SNOs on GCP using n2-standard-8 instances (8 vCPU / 32 GB RAM). For on-premises bare-metal deployments, the minimum SNO hardware is 8 vCPU / 16 GB RAM / 120 GB storage (see OCP 4.22 SNO requirements). With the AMQ Broker operator and a broker instance, 8 vCPU / 32 GB RAM is the practical floor.

On physical hardware, SNO provisioning uses the Assisted Installer (USB, Redfish virtual media, or iPXE) instead of Hive. A Baseboard Management Controller (BMC) with Redfish support is required for virtual media provisioning. See Scale and Production Considerations for full details.

The ApplicationSet artemis-edge-workloads in ztp-policies is intentionally unbound. Binding it would sync the full hub Helm chart to every SNO.

AgnosticD did not create SNOs. oc get managedcluster should show only local-cluster until you run deploy-spokes.sh on this regional hub.

Steps

1. Create GCP Credential in RHACM Console

Open the RHACM Console and create a cloud provider credential. This stores your GCP service account, pull secret, SSH keys, and base domain in a single ACM-managed secret.

RHACM Console URL: https://console-openshift-console.apps.hub.example.gcp.redhatworkshops.io/multicloud/home/overview

  1. Navigate to Infrastructure → Credentials → Add credential

  2. Select Google Cloud Platform as the provider type

  3. Set:

    • Credential name: gcp-credentials

    • Namespace: gcp-credentials (create new)

  4. Upload your GCP service account JSON key (or paste its contents)

  5. Enter your Base DNS domain: example.gcp.redhatworkshops.io

  6. Paste your Red Hat pull secret (from https://console.redhat.com/openshift/install/pull-secret)

  7. Paste your SSH public and private keys

  8. Click Add

Verify the credential was created:

oc get secret gcp-credentials -n gcp-credentials -o jsonpath='{.metadata.labels.cluster\.open-cluster-management\.io/type}'
Expected output
gcp

2. Run the Provisioning Script

The script extracts credentials from the ACM secret you just created and automates the entire Hive provisioning pipeline.

The provisioning script requires cluster-admin access. Verify before running:

oc whoami
oc auth can-i create clusterdeployments.hive.openshift.io --all-namespaces

You should see your user or service account name, and yes for the permission check. If not, ask your administrator to grant you cluster-admin.

 
./scripts/deploy-spokes.sh --clusters 2

The script performs these steps automatically:

  1. Pre-flight checks — verifies oc login, Hive pods, ClusterImageSet, ArgoCD app

  2. Credential extraction — reads GCP key, pull secret, and SSH keys from the ACM credential (or bastion files)

  3. Secret creation — creates per-spoke namespaces with gcp-creds, pull-secret, and ssh-private-key secrets

  4. ArgoCD patch — enables spokeProvisioning in the Helm values, which creates:

    • ClusterDeployment CRs (triggers Hive provisioning)

    • ManagedCluster CRs (auto-imports clusters into RHACM)

    • KlusterletAddonConfig CRs (enables ACM add-ons)

    • RHACM Policies in edge-broker-policies (AMQ operator + per-region broker + federation)

  5. Hive monitoring — polls ClusterDeployments every 60 seconds until installed

Cluster Region GCP Machine Type

sno-edge-01

NY

n2-standard-8

sno-edge-02

NJ

n2-standard-8

First pass is two SNOs on east only. Central and west stay empty until a later lab.

Spoke cluster provisioning takes approximately 30-45 minutes per cluster. All clusters provision in parallel.

3. Monitor Provisioning Progress

While the script monitors automatically, you can also watch from a second terminal:

oc get clusterdeployments -A -o custom-columns=NAME:.metadata.name,INSTALLED:.spec.installed,PLATFORM:.spec.platform.gcp.region

Or use the watch flag for continuous updates:

oc get clusterdeployments -A -w
You can proceed to Module 3: Verify AMQ Brokers and Federation while the clusters provision. The spoke-specific AMQ brokers and federation will activate once the clusters join.

You can also check status at any time:

./scripts/deploy-spokes.sh --status

4. Verify ManagedCluster Labels

Once clusters are provisioned, they auto-register as ManagedClusters:

oc get managedcluster -o custom-columns=NAME:.metadata.name,REGION:.metadata.labels.edge-region,PROVIDER:.metadata.labels.cloud-provider,AVAILABLE:.status.conditions[0].status
Expected output (after provisioning completes)
NAME            REGION   PROVIDER   AVAILABLE
local-cluster   <none>   <none>     True
sno-edge-01     NY       gcp        True
sno-edge-02     NJ       gcp        True

Each spoke cluster shows its assigned edge region and gcp cloud provider label.

5. Check Placement Decisions (edge-broker-policies)

AMQ on SNOs is selected by Placement in edge-broker-policies, not ztp-policies.

oc get placement -n edge-broker-policies
Expected output
NAME                                       SUCCEEDED   REASON                    SELECTEDCLUSTERS
policy-amq-broker-operator-sno-placement   True        AllDecisionsScheduled     2
policy-amq-broker-spoke-01-placement       True        AllDecisionsScheduled     1
policy-amq-broker-spoke-02-placement       True        AllDecisionsScheduled     1

First pass is NY and NJ on east only. A CT spoke-03 placement appears only if configured in edgeBrokers[].

Confirm ztp-policies is not bound:

oc get placement -n ztp-policies

Those Placements show NoManagedClusterSetBindings. That is expected. Do not create a ManagedClusterSetBinding in ztp-policies.

6. Verify RHACM Policies (AMQ on SNOs)

These Policy objects were not written by hand. The hub Helm chart (templates/edge-broker-policies.yaml, synced by ArgoCD field-content) rendered one Policy + Placement per edgeBrokers[] entry. Inspect the live objects in the CLI or in ACM Governance. For hundreds of sites you would not grow values.yaml — use PolicyGenerator with cluster labels and fromClusterClaim instead. Do not apply ztp/policygenerator/ in this lab; it would collide with the Helm policies already on the hub.

Open ACM in the OpenShift console (the old multicloud-console host is not used on this catalog item). Go to Governance → Policies, or open Governance ↗, or run:

oc get policy -n edge-broker-policies
Expected output
NAME                               REMEDIATION ACTION   COMPLIANCE STATE
policy-amq-broker-operator-sno     enforce              Compliant
policy-amq-broker-spoke-01         enforce              Compliant
policy-amq-broker-spoke-02         enforce              Compliant
policy-block-spoke-01-federation   enforce
policy-sno-uwl-amq-metrics         enforce              Compliant

Only spoke-01 (NY) and spoke-02 (NJ) are configured on east. policy-block-spoke-01-federation stays disabled until Module 6. policy-sno-uwl-amq-metrics is the SNO user-workload metrics allowlist (Module 8).

RHACM Governance Policies table in the OpenShift console. policy-amq-broker-operator-sno
Figure 2. RHACM Governance — Helm-generated edge broker policies
Placement policy-amq-broker-spoke-01-placement with AllDecisionsScheduled and sno-edge-01 selected.
Figure 3. RHACM Placement — selected clusters for the NY broker policy

  • policy-amq-broker-operator-sno — Namespace, OperatorGroup, and AMQ Broker Subscription on every sites: edge-sno cluster

  • policy-amq-broker-spoke-01 / spoke-02 — one ActiveMQArtemis per region, with AMQP federation to hub-01

  • policy-block-spoke-01-federation — stays disabled until Module 6

See which clusters each broker policy hit:

oc get policy policy-amq-broker-spoke-01 -n edge-broker-policies -o jsonpath='{range .status.status[*]}{.clustername} {.compliant}{"\n"}{end}'
Expected output
sno-edge-01 Compliant

You cannot oc login to the SNOs as user1. Policy status on the hub is how you confirm AMQ landed on the edge.

7. Troubleshooting

If a cluster is stuck provisioning, check the ClusterDeployment conditions:

oc get clusterdeployment sno-edge-01 -n sno-edge-01 -o yaml | grep -A 10 conditions

Check the provisioning pod logs:

oc logs -n sno-edge-01 -l hive.openshift.io/cluster-deployment-name=sno-edge-01 --tail=50

Common issues:

  • GCP quota exceeded — ensure n2-standard-8 vCPU quota allows concurrent SNO installs (8 vCPUs per cluster)

  • DNS zone not found — verify the base domain matches your GCP sandbox DNS zone

  • Credential error — re-check the GCP service account JSON in the RHACM credential

To tear down spoke clusters and start over:

./scripts/deploy-spokes.sh --destroy