Module 8: AI-Assisted Capacity Operations with OpenShift Lightspeed

I have a pod called besteffort-app in the capacity-workshop namespace.
It has no memory requests or limits set. How do I check whether it has
been OOMKilled, and what Prometheus metric should I use to determine an
accurate memory request for it?

I have three apps in my capacity-workshop namespace:
- guaranteed-app: CPU and memory requests equal limits (200m CPU / 256Mi)
- burstable-app: has a 100m CPU request but no limit
- besteffort-app: no resource requests or limits set at all

Explain which QoS class each app is in, and what happens to each one
when the node runs low on memory. Which gets evicted first?

My load-generator app in the capacity-workshop namespace has CPU requests
of 100m and limits of 500m. I set up a basic CPU HPA on it in a previous
lab. Show me how to change it to scale based on a custom Prometheus metric
— for example, the number of HTTP requests per second — instead of CPU.
Show me the YAML.

Write a PromQL query that shows the CPU throttling rate as a percentage
for every pod in my capacity-workshop namespace. My cpu-throttle-demo pod
has a 200m CPU limit but runs a burn loop that tries to use far more.
I want to be able to see that throttling clearly in the query output.

100 * sum by (pod, namespace) (
  rate(container_cpu_cfs_throttled_seconds_total{namespace="capacity-workshop"}[5m])
) /
sum by (pod, namespace) (
  rate(container_cpu_cfs_periods_total{namespace="capacity-workshop"}[5m])
)

My OpenShift cluster has 3 nodes that are all running as combined
control-plane and worker nodes. I want to support more workloads in
the capacity-workshop namespace. How do I safely raise the maxPods
limit above the default 250 on OpenShift 4.21, and what are the specific
risks of doing this on nodes that run both control-plane and workload pods?

Write a PromQL query to check the current etcd database size on my
OpenShift cluster and estimate how quickly it is growing. My cluster
has 3 control-plane nodes. I want to know when I should defrag or
plan for additional control plane capacity.

# Current etcd DB size in bytes
etcd_mvcc_db_total_size_in_bytes

# Growth rate over 7 days (bytes per day)
deriv(etcd_mvcc_db_total_size_in_bytes[7d]) * 86400

I run a 3-node OpenShift training cluster where all nodes are combined
control-plane and workers, and I use a single capacity-workshop namespace
for all student workloads. As I scale to more students and cohorts, when
does it make more sense to keep growing this single cluster versus deploying
separate per-student clusters? What are the operational trade-offs?

I connected my student cluster to an RHACM hub in Module 5 and I'm
building a Grafana capacity dashboard on the hub. Write three PromQL
queries for my capacity-workshop namespace:
1. CPU request overcommit ratio per namespace across all clusters
2. Memory utilisation versus allocated capacity per cluster
3. The number of pods per node to identify density hotspots

Use the cluster label so each cluster appears separately in the dashboard.

Write the PromQL query that calculates Pod Velocity — the number of new
pod deployments per week — across all namespaces for the past 90 days.
This is the foundation of our capacity forecasting model from Module 2.

My capacity-workshop cluster has 3 nodes, each currently running around
65 pods. The Pod Velocity from my Module 2 Prometheus dashboard is
approximately 14 new pods per week. The default maxPods per node is 250.

How many weeks until I hit maxPods and need to add a worker node?
Show your calculation step by step, then write a PromQL expression that
computes this runway automatically from live cluster data.

Current available capacity:
  3 nodes × 250 maxPods = 750 total pod slots
  Currently used: 3 × 65 = 195 pods
  Available: 750 - 195 = 555 pod slots

At 14 new pods/week:
  Weeks until full: 555 ÷ 14 = 39.6 weeks (~40 weeks)

PromQL for live runway (in weeks):
  (
    sum(kube_node_status_allocatable{resource="pods"})
    - sum(kube_pod_info{node!=""})
  ) / <pod_velocity_per_week>

Generate a Grafana panel JSON snippet that shows "days until maxPods is
reached" as a single-stat panel per node. This should read from Prometheus
and update automatically. Use a green/yellow/red threshold:
- green: > 60 days
- yellow: 30-60 days
- red: < 30 days

Assume Pod Velocity of 14 pods/week and maxPods of 250.

My production cluster has a CPU request overcommit ratio of 2.3× (meaning
applications have requested 2.3× the actual allocatable CPU). During Black
Friday, we expect a 10× traffic spike over baseline.

What are the risks of running at 2.3× overcommit during a 10× traffic event,
and how much additional node capacity should I plan to provision before the
event? Use the capacity planning framework from the answer.

Based on the following capacity data from our capacity-workshop OpenShift
cluster, write a one-paragraph executive summary suitable for a quarterly
budget request:

- Current cluster: 3 combined control-plane/worker nodes on AWS (m7a.2xlarge)
- Active workloads: capacity-workshop namespace with ~195 pods running
- Pod Velocity: 14 new pods/week (growing 8% month-over-month)
- Capacity runway: ~40 weeks at current growth before hitting maxPods
- Action required: provision additional worker nodes before week 32 (safety buffer)
- Estimated cost: $3,200/month per additional m7a.2xlarge node on AWS

Write this for an audience of finance and business leadership, not engineers.
Avoid technical jargon. Focus on cost, risk, and timeline.

Look at my current cluster state. Which three namespaces have the highest
ratio of CPU requests versus actual CPU usage over the past 24 hours?

Based on current pod counts across all nodes and the deployment velocity
you can observe in cluster events, estimate how many weeks of node capacity
remain before I need to add a worker node.

What deployments and pods are running in the capacity-workshop namespace?
Are any of them in a non-Running state?

How many nodes does this cluster have? What is the total allocatable CPU and
memory across all nodes? Are any nodes in a NotReady or degraded state?

Based on the nodes in this cluster, estimate how much additional workload capacity
is available. What fraction of allocatable CPU and memory is currently requested?