Module 6: Store-and-Forward Resilience

Objectives

Simulate a network partition on the NY SNO (not between two pods on the hub)
Produce to the SNO AMQPS Route while federation egress is blocked
Observe store-and-forward when the ACM partition policy is disabled again

user1 cannot apply a NetworkPolicy on the SNO. The partition is an ACM policy in edge-broker-policies that a cluster-admin enables. Students still produce and consume from the hub.

The partition is between the SNO and this region’s AMQ hub. Global Hub is not on the messaging path.

Real WAN disconnection vs. lab simulation

This module simulates a partition using a NetworkPolicy that blocks pod-level egress. In a real deployment, a WAN failure is broader — the entire SNO loses connectivity to the hub cluster.

What happens during real WAN loss:

The klusterlet agent on the SNO enters a degraded condition but does not remove existing resources. Workloads already deployed — including the AMQ Broker — continue running autonomously.
ACM policies already applied remain enforced locally. The policy controller on the managed cluster continues to reconcile them. New policy changes from the hub will not propagate until connectivity restores.
The AMQ Broker continues accepting local producer connections and queuing messages in the Artemis journal. Federation auto-reconnects when WAN restores (reconnectAttempts=-1 in the broker properties).
Policy compliance status on the hub goes stale — the hub shows the cluster as offline. This is expected behavior, not a failure.

In short: losing hub connectivity does not break local operations. The SNO and its broker operate independently until the link comes back.

Steps

1. Start a Consumer on the Hub

Start a long-running consumer on messages.NY.5604 (the federated class). Store-and-forward in this module is on 5604, not NY.test.

Start the consumer before simulating the partition so it is subscribed to the topic.

oc -n artemis run consumer-m6 -ti \
  --image=registry.redhat.io/amq7/amq-broker-rhel9:7.14 \
  --rm=true --restart=Never -- /opt/amq/bin/artemis consumer \
  --url=amqp://hub-01-broker-amqp-acceptor-0-svc.artemis.svc.cluster.local:5672 \
  --protocol=AMQP --user=bob --password=bosco \
  --destination=topic://messages.NY.5604 --verbose

Leave this running in a separate terminal.

2. Simulate Network Partition on sno-edge-01

Policy policy-block-spoke-01-federation is created disabled. Enabling it enforces a NetworkPolicy on the NY SNO that blocks egress from spoke-01-broker (federation to hub-01 drops). Inbound AMQPS from your producer still works.

As cluster-admin (facilitator):

oc patch policy policy-block-spoke-01-federation -n edge-broker-policies \
  --type merge -p '{"spec":{"disabled":false}}'

oc get policy policy-block-spoke-01-federation -n edge-broker-policies

Expected output

NAME                               REMEDIATION ACTION   COMPLIANCE STATE   AGE
policy-block-spoke-01-federation   enforce              Compliant          3d20h

Wait until the policy is Compliant on sno-edge-01. Confirm the same state in ACM Governance (Governance → Policies → policy-block-spoke-01-federation): Status Enabled, remediation enforce, no violations on sno-edge-01. The hub consumer should stop receiving new NY messages from the SNO.

3. Produce While Disconnected

Produce to the NY SNO Route. Messages stay on the edge broker until federation resumes.

oc -n artemis run producer-m6 -ti \
  --image=registry.redhat.io/amq7/amq-broker-rhel9:7.14 \
  --rm=true --restart=Never -- /opt/amq/bin/artemis producer \
  --url='amqps://spoke-01-broker-amqps-acceptor-0-svc-rte-artemis.apps.sno-edge-01.%sandbox_guid%.gcp.redhatworkshops.io:443?sslEnabled=true&transport.trustAll=true&transport.verifyHost=false&useTopologyForLoadBalancing=false' \
  --protocol=AMQP --user=admin --password=admin \
  --message-count=20 --message='queued during SNO partition' \
  --destination=topic://messages.NY.5604 --verbose

Verify the messages are queued on the SNO and have not reached the hub:

curl -sS -u admin:admin \
  'http://spoke-01-broker-wconsj-0-svc-rte-artemis.apps.sno-edge-01.%sandbox_guid%.gcp.redhatworkshops.io/console/jolokia/read/org.apache.activemq.artemis:broker=%22amq-broker%22,component=addresses,address=%22messages.NY.5604%22/MessageCount' \
  | python3 -m json.tool

Expected output — 20 messages stored on the edge

{
    "value": 20
}

MessageCount: 20 on the SNO proves store-and-forward is working — the messages are in the Artemis journal, waiting for federation to resume.

4. Restore Connectivity

As cluster-admin, disable the policy (it stops enforcing but does not delete the NetworkPolicy). Then delete the leftover NetworkPolicy on the SNO with the Hive admin kubeconfig:

oc patch policy policy-block-spoke-01-federation -n edge-broker-policies \
  --type merge -p '{"spec":{"disabled":true}}'
oc extract secret/$(oc get clusterdeployment sno-edge-01 -n sno-edge-01 -o jsonpath='{.spec.clusterMetadata.adminKubeconfigSecretRef.name}') -n sno-edge-01 --keys=kubeconfig --to=/tmp
oc --kubeconfig /tmp/kubeconfig delete networkpolicy block-spoke-01-federation -n artemis

Confirm the policy is disabled again and the leftover NetworkPolicy is gone:

oc get policy policy-block-spoke-01-federation -n edge-broker-policies -o jsonpath='{.spec.disabled}{"\n"}'

Expected output

true

oc --kubeconfig /tmp/kubeconfig get networkpolicy -n artemis

Expected output

No resources found in artemis namespace.

5. Observe Store-and-Forward Flush

After restoring the link, federation resumes and the SNO flushes queued messages to hub-01. The primary proof is the Jolokia MessageCount dropping to zero on the SNO:

curl -sS -u admin:admin \
  'http://spoke-01-broker-wconsj-0-svc-rte-artemis.apps.sno-edge-01.%sandbox_guid%.gcp.redhatworkshops.io/console/jolokia/read/org.apache.activemq.artemis:broker=%22amq-broker%22,component=addresses,address=%22messages.NY.5604%22/MessageCount' \
  | python3 -m json.tool

Expected output — messages flushed

{
    "value": 0
}

MessageCount: 0 means all 20 messages have been forwarded off the SNO. If the count is still > 0, wait 30 seconds and re-run — federation reconnect takes time.

Verify the federation link itself is back up (same check as Module 3):

curl -sS -u admin:admin \
  'http://spoke-01-broker-wconsj-0-svc-rte-artemis.apps.sno-edge-01.%sandbox_guid%.gcp.redhatworkshops.io/console/jolokia/exec/org.apache.activemq.artemis:broker=%22amq-broker%22/listBrokerConnections' \
  | python3 -m json.tool

Expected output

{
    "request": {
        "mbean": "org.apache.activemq.artemis:broker=\"amq-broker\"",
        "type": "exec",
        "operation": "listBrokerConnections"
    },
    "value": "[{\"name\":\"hub-01-connection\",\"protocol\":\"AMQP\",\"started\":true}]",
    "timestamp": 1789396379,
    "status": 200
}

started: true confirms federation is live. timestamp will differ.

The hub consumer terminal may or may not show the 20 Received lines. Multicast topic delivery to the consumer depends on subscription timing during the reconnect. The Jolokia MessageCount drop from 20 to 0 is the definitive proof of store-and-forward.

6. Clean Up

oc delete pod consumer-m6 producer-m6 -n artemis --force 2>/dev/null