Module 8: Observability and Grafana Metrics

Objectives

  • Verify that Multicluster Observability (MCO) is deployed and healthy

  • Explore default RHACM Grafana dashboards across hub and spoke clusters

  • Create custom AMQ Broker alerts using Thanos Ruler

  • Import a custom AMQ Broker Grafana dashboard

  • Compare cross-cluster metrics between hub and SNO edge clusters

  • Use regional MCO Grafana for this hub’s SNOs, and Global Hub Thanos Query for fleet AMQ (not GH compliance Grafana)

Background

Red Hat Advanced Cluster Management includes a Multicluster Observability (MCO) addon that aggregates Prometheus metrics from all managed clusters into a central Thanos/Grafana stack on the hub. This deployment uses Google Cloud Storage (GCS) as the Thanos backend for long-term metric retention.

MCO is per regional ACM hub. This Grafana sees east only (local-cluster plus SNOs you joined in Module 2). Global Hub Grafana in Module 7 is compliance (PostgreSQL), not AMQ. Fleet AMQ is an extra Thanos Query on Global Hub whose stores are the three regional Thanos Query gRPC endpoints (passthrough + mTLS; fill fleetThanosQuery.regionalStores after the regionals exist).

Mode 2 control plane. This module uses MCO Grafana on ACM east and
Figure 1. Control plane Grafana surfaces — regional MCO on east vs GH compliance vs fleet Thanos Query

MCO was deployed automatically during cluster provisioning via an AgnosticD workload role (ocp4_workload_rhacm_observability_gcs). In this module you will verify the deployment, explore built-in dashboards, and build custom alerting and dashboards for AMQ Broker.

Prerequisites

Verify RHACM is running:

oc get mch -A -o custom-columns=NAME:.metadata.name,STATUS:.status.phase
Expected output
NAME              STATUS
multiclusterhub   Running

Steps

1. Verify MCO Deployment

Check that the MultiClusterObservability CR is healthy:

oc get multiclusterobservability observability -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
echo ""
Expected output
True

Verify all MCO pods are running:

oc get pods -n open-cluster-management-observability --no-headers | wc -l
echo "pods total"
oc get pods -n open-cluster-management-observability --no-headers | grep -v Running | grep -v Completed || echo "All pods healthy"
A healthy MCO deployment has approximately 30-35 pods including Thanos components, Grafana replicas, alertmanager, and metrics collectors.

Verify the GCS-backed Thanos object storage secret exists:

oc get secret thanos-object-storage -n open-cluster-management-observability -o jsonpath='{.data.thanos\.yaml}' | base64 -d | head -3
Expected output
type: GCS
config:
  bucket: ...

2. Explore Default Grafana Dashboards

Get the Grafana URL:

echo "https://$(oc get route grafana -n open-cluster-management-observability -o jsonpath='{.spec.host}')"

Open the URL in your browser. Grafana uses OpenShift authentication — log in with your cluster credentials.

Grafana uses OpenShift OAuth. Log in with admin / your cluster admin password, or with your user1 credentials.
Grafana Dashboards page after OpenShift login. ACM and Platform-MCOA folders are listed; Custom holds the AMQ Broker dashboard.
Figure 2. Grafana after OpenShift SSO — Dashboards list

Explore the built-in dashboards under Dashboards:

  • ACM - Clusters Overview: memory utilization, CPU usage, and etcd database size per cluster

  • ACM - Resource Optimization: resource requests vs actual usage across managed clusters

ACM Clusters Overview dashboard showing local-cluster
Figure 3. ACM - Clusters Overview (hub plus both SNO edge clusters)

With your SNO spoke clusters joined, you can compare metrics across:

  • local-cluster (this regional ACM hub)

  • sno-edge-01 (NY edge)

  • sno-edge-02 (NJ edge)

Dashboard Exploration Tasks

Using the ACM - Clusters Overview dashboard:

  1. Find the maximum API server latency for local-cluster

  2. Compare memory utilization % across all clusters

  3. Find the etcd database size for each cluster

  4. Identify which namespace consumes the most CPU on local-cluster

3. Verify Metrics Collection

Check that the metrics collector is running on the hub:

oc get pods -n open-cluster-management-observability -l component=metrics-collector

Check that endpoint-monitoring addons are enabled on spoke clusters:

oc get managedclusteraddon -A -l component=addon-agent | grep observability

Verify ServiceMonitors are collecting AMQ Broker metrics:

oc get servicemonitor -n artemis 2>/dev/null || echo "ServiceMonitor resources will appear after AMQ Broker is deployed"

4. Create Custom AMQ Broker Alerts

Switch to the Grafana Explore page (compass icon in the left sidebar). Click the Builder / Code toggle and switch to Code mode so you can paste PromQL expressions.

4.1. Verify AMQ Broker Metrics Are Available

Run this query to check that AMQ Broker metrics are being collected:

artemis_message_count

You should see results from the regional AMQ hub (cluster="local-cluster") and SNO brokers (cluster="sno-edge-01" / sno-edge-02). Helm ships ConfigMap observability-metrics-custom-allowlist in open-cluster-management-observability with both metrics_list.yaml and uwl_metrics_list.yaml — hub AMQ is scraped by user-workload Prometheus, so the UWL list is required or Explore stays empty.

SNO AMQ ServiceMonitors are also user-workload: the default spoke metrics-collector federates platform Prometheus (prometheus-k8s), which does not scrape the artemis namespace. Policy policy-sno-uwl-amq-metrics sets enableUserWorkload: true on each SNO (merged into openshift-monitoring/cluster-monitoring-config so ACM alert-forwarding is preserved) and creates a spoke observability-metrics-custom-allowlist in artemis. That starts uwl-metrics-collector so SNO artemis_* series reach Thanos and appear in the AMQ dashboard cluster dropdown.

Grafana Explore query artemis_message_count. Hub series use cluster=local-cluster; after policy-sno-uwl-amq-metrics
Figure 4. Grafana Explore — artemis_message_count

4.2. Deploy Custom Alert Rules

Create a ConfigMap with custom alert rules for AMQ Broker monitoring:

cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-ruler-custom-rules
  namespace: open-cluster-management-observability
data:
  custom_rules.yaml: |
    groups:
      - name: amq-broker-health
        rules:
        - alert: BrokerQueueDepthHigh
          annotations:
            summary: "AMQ Broker queue depth exceeds 1000 messages"
            description: "Queue {{ $labels.queue }} on broker {{ $labels.broker }} in cluster {{ $labels.cluster }} has {{ $value }} messages."
          expr: |
            artemis_message_count > 1000
          for: 2m
          labels:
            severity: warning
        - alert: BrokerMemoryUsageHigh
          annotations:
            summary: "AMQ Broker address memory usage exceeds 80%"
            description: "Broker {{ $labels.broker }} in cluster {{ $labels.cluster }} is using {{ $value | humanize }}B of address memory."
          expr: |
            artemis_address_memory_usage > 800000000
          for: 5m
          labels:
            severity: critical
EOF

4.3. Verify Alerts

Wait 60 seconds for the thanos-ruler pods to pick up the new rules, then run these queries in Grafana Explore:

Check current queue depths:

artemis_message_count

Check current memory usage:

artemis_address_memory_usage

Check if any alerts are firing:

ALERTS{alertname=~"Broker.*"}

5. Create Custom AMQ Broker Dashboard

Apply a custom Grafana dashboard for AMQ Broker monitoring. The dashboard is deployed as a ConfigMap:

cat <<'DASHBOARD_EOF' | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: artemis-amq-broker-dashboard
  namespace: open-cluster-management-observability
  labels:
    grafana-custom-dashboard: "true"
data:
  artemis-broker.json: |
    {
      "annotations": {"list": []},
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 0,
      "id": null,
      "links": [],
      "panels": [
        {
          "title": "Queue Message Counts",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
          "targets": [{"expr": "artemis_message_count{cluster=~\"$cluster\"}", "legendFormat": "{{cluster}} / {{broker}} / {{queue}}"}],
          "datasource": {"type": "prometheus", "uid": "Observatorium"}
        },
        {
          "title": "Messages Added & Acked",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
          "targets": [
            {"expr": "rate(artemis_messages_added{cluster=~\"$cluster\"}[5m])", "legendFormat": "added: {{cluster}} / {{broker}}"},
            {"expr": "rate(artemis_messages_acknowledged{cluster=~\"$cluster\"}[5m])", "legendFormat": "acked: {{cluster}} / {{broker}}"}
          ],
          "datasource": {"type": "prometheus", "uid": "Observatorium"}
        },
        {
          "title": "Connection Count",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
          "targets": [{"expr": "artemis_connection_count{cluster=~\"$cluster\"}", "legendFormat": "{{cluster}} / {{broker}}"}],
          "datasource": {"type": "prometheus", "uid": "Observatorium"}
        },
        {
          "title": "Address Memory Usage",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8},
          "targets": [{"expr": "artemis_address_memory_usage{cluster=~\"$cluster\"}", "legendFormat": "{{cluster}} / {{broker}}"}],
          "datasource": {"type": "prometheus", "uid": "Observatorium"},
          "fieldConfig": {"defaults": {"unit": "bytes"}}
        },
        {
          "title": "Consumer Count",
          "type": "timeseries",
          "gridPos": {"h": 8, "w": 24, "x": 0, "y": 16},
          "targets": [{"expr": "artemis_consumer_count{cluster=~\"$cluster\"}", "legendFormat": "{{cluster}} / {{broker}} / {{address}}"}],
          "datasource": {"type": "prometheus", "uid": "Observatorium"}
        }
      ],
      "schemaVersion": 39,
      "tags": ["artemis", "amq-broker", "custom"],
      "templating": {
        "list": [
          {
            "name": "cluster",
            "type": "query",
            "datasource": {"type": "prometheus", "uid": "Observatorium"},
            "query": "label_values(artemis_message_count, cluster)",
            "includeAll": true,
            "multi": true,
            "current": {"selected": true, "text": "All", "value": "$__all"}
          }
        ]
      },
      "time": {"from": "now-1h", "to": "now"},
      "title": "AMQ Broker - Artemis Edge",
      "uid": "artemis_amq_broker_acm"
    }
DASHBOARD_EOF

After applying, navigate to Grafana → Dashboards → Custom folder. The AMQ Broker - Artemis Edge dashboard should appear. The Helm chart may already have deployed this dashboard; the apply below is idempotent if the ConfigMap name matches. Each series legend includes cluster so hub (local-cluster) and SNO brokers are distinguishable when multiple clusters are selected.

Use the Cluster dropdown at the top to filter one node at a time. The screenshots below are the same dashboard with that filter set to the hub, then each SNO — that is how you tell which node the series come from.

AMQ Broker - Artemis Edge dashboard with cluster filter local-cluster.
Figure 5. Custom Grafana dashboard — AMQ Broker - Artemis Edge (cluster filter: local-cluster)
AMQ Broker - Artemis Edge dashboard with cluster filter sno-edge-01.
Figure 6. Same dashboard — cluster filter sno-edge-01
AMQ Broker - Artemis Edge dashboard with cluster filter sno-edge-02.
Figure 7. Same dashboard — cluster filter sno-edge-02

The dropdown values on this regional Grafana are local-cluster, sno-edge-01, and sno-edge-02. For all three regions at once, use Global Hub Thanos Query (not Module 7 compliance Grafana).

The dashboard has 5 panels:

Panel Metric

Queue Message Counts

artemis_message_count per broker/queue

Messages Added & Acked

rate(artemis_messages_added), rate(artemis_messages_acknowledged)

Connection Count

artemis_connection_count — watch for federation connections from spokes

Address Memory Usage

artemis_address_memory_usage — monitor against globalMaxSize=1GiB

Consumer Count

artemis_consumer_count per address

6. Cross-Cluster Observability

With MCO deployed, the endpoint-observability-operator automatically deploys a metrics-collector on each managed cluster.

Verify metrics are being collected from the hub:

oc get pods -n open-cluster-management-observability -l component=metrics-collector

In Grafana Explore, compare broker metrics across clusters:

artemis_connection_count

Filter by specific clusters:

  • {cluster="local-cluster"} — regional AMQ hub (hub-01) metrics

  • {cluster="sno-edge-01"} — NY edge metrics (once AMQ Broker is deployed via policy)

  • {cluster="sno-edge-02"} — NJ edge metrics

7. Correlate with Messaging Activity

  1. Go back to the Showroom terminal and produce 100 messages (see Module 5)

  2. Watch the Queue Message Counts panel spike for the target queue

  3. Switch to Connection Count to see federation connections

  4. Check Address Memory Usage to ensure you are within the 1GiB global max

8. Cleanup (Optional)

Only remove MCO if you no longer need cross-cluster observability.

To remove MCO and free resources:

oc delete multiclusterobservability observability

To remove the GCS bucket:

gsutil rm -r gs://acm-obs-example

Next

Official AMQ and ACM articles for the URLs and policies in this lab are on the References page.