Operator Release Guide¶

Requires: Kubernetes or OpenShift cluster

This document is the single authoritative runbook for releasing the Jupyter Notebook Validator Operator to OperatorHub and community operator catalogs. Follow the numbered steps in order.

Version History and Submission Status¶

Version	OCP Stream	Upgrade Strategy	OperatorHub Status
v1.0.5	4.18+	`replaces: v1.0.4`	Submitted
v1.0.6	4.18+	`replaces: v1.0.5`	Submitted
v1.0.7	4.18	`replaces: v1.0.6`	Submitted (merged upstream)
v1.0.8	4.19	`olm.skipRange: >=1.0.2 <1.0.8`	community-operators-prod PR #9442 / community-operators PR #7940 - pending maintainer merge
v1.0.9	4.20	`olm.skipRange: >=1.0.2 <1.0.9`	Released
v1.0.10	4.22	`olm.skipRange: >=1.0.2 <1.0.10`	Current

OCP-stream convention: v1.0.7 → OCP 4.18 | v1.0.8 → OCP 4.19 | v1.0.9 → OCP 4.20 | v1.0.10 → OCP 4.22

Upgrade strategy change at v1.0.8: spec.replaces was replaced with olm.skipRange because v1.0.7 was submitted with a flat bundle structure (no manifests/ subdirectory), making it invisible to the operatorcert check_replaces_availability check. See Known Pipeline Pitfalls for details.

Metadata Standards¶

All releases MUST use the following consistent metadata.

Provider and Maintainer¶

provider:
  name: Decision Crafters
  url: https://www.decisioncrafters.com/

maintainers:
  - name: Tosin Akinosho
    email: takinosh@redhat.com

Git Author (required for DCO)¶

git config user.email "takinosh@redhat.com"
git config user.name "Tosin Akinosho"

DCO Sign-off¶

All commits to community-operator repos MUST include a DCO sign-off:

git commit -s -m "operator jupyter-notebook-validator-operator (v1.0.X)"

The sign-off line must match the author email:

Signed-off-by: Tosin Akinosho <takinosh@redhat.com>

Icon¶

Use the standard Decision Crafters icon from v1.0.3-ocp4.19. Do not use personal GitHub avatars. The base64data: value begins with:

iVBORw0KGgoAAAANSUhEUgAAAIAAAACACAIAAABMXPacAAAABG...

Pre-Release Checklist¶

Before generating the bundle or opening PRs, verify:

provider.name is Decision Crafters
provider.url is https://www.decisioncrafters.com/
maintainers[0].email is takinosh@redhat.com
maintainers[0].name is Tosin Akinosho
containerImage annotation matches the image tag being released
Icon matches the standard icon (from v1.0.3-ocp4.19)
Git author email is takinosh@redhat.com
All commits include DCO sign-off (-s flag)

Verification commands:

CSV=bundle/manifests/jupyter-notebook-validator-operator.clusterserviceversion.yaml

# Provider
grep -A2 "provider:" $CSV

# Maintainers
grep -A2 "maintainers:" $CSV

# containerImage annotation
grep "containerImage:" $CSV | head -1

# Icon (first 50 chars)
grep "base64data:" $CSV | sed 's/.*base64data: //' | cut -c1-50

Release Runbook¶

Replace VERSION with the new version number (e.g. 1.0.9) and OCP_STREAM with the target OCP minor version (e.g. 4.20) throughout.

Step 1 - Create the release branch¶

VERSION=1.0.9
OCP_STREAM=4.20

git checkout main
git pull origin main
git checkout -b release-${OCP_STREAM}
git push -u origin release-${OCP_STREAM}

Step 2 - Bump versions¶

# Makefile
sed -i "s/^VERSION ?= .*/VERSION ?= ${VERSION}/" Makefile

# Helm chart
CHART=helm/jupyter-notebook-validator-operator/Chart.yaml
sed -i "s/^version:.*/version: ${VERSION}/" $CHART
sed -i "s/^appVersion:.*/appVersion: \"${VERSION}\"/" $CHART

# Verify
grep "^VERSION" Makefile
grep "^version\|^appVersion" $CHART

Step 3 - Regenerate manifests and DeepCopy code¶

make manifests generate
git diff --stat   # verify only expected generated files changed

Step 4 - Generate the OLM bundle¶

make bundle

After make bundle, manually correct the CSV and bundle metadata. Do NOT use spec.replaces - use olm.skipRange instead. See Known Pipeline Pitfalls for the reason.

CSV=bundle/manifests/jupyter-notebook-validator-operator.clusterserviceversion.yaml

# Remove spec.replaces if make bundle generated it
sed -i '/^  replaces:/d' $CSV

# Add olm.skipRange annotation to metadata.annotations
# (adjust lower bound to the first ever published version: 1.0.2)
python3 - <<'EOF'
import re, sys

path = "bundle/manifests/jupyter-notebook-validator-operator.clusterserviceversion.yaml"
version = "${VERSION}"   # replace with actual VERSION value

with open(path) as f:
    content = f.read()

skip_line = f'    olm.skipRange: ">=1.0.2 <{version}"\n'
# Insert after 'metadata:\n  annotations:\n'
content = re.sub(
    r'(metadata:\n  annotations:\n)',
    r'\1' + skip_line,
    content,
    count=1
)
with open(path, "w") as f:
    f.write(content)
print("olm.skipRange written")
EOF

# Update containerImage annotation (must match the image tag you will push)
sed -i "s|containerImage:.*quay.io/takinosh/jupyter-notebook-validator-operator:.*|containerImage: quay.io/takinosh/jupyter-notebook-validator-operator:${VERSION}|" $CSV

# Update bundle/metadata/annotations.yaml - set channel and OCP range
# Edit manually or with sed:
sed -i "s|operators.operatorframework.io.bundle.channels.v1:.*|operators.operatorframework.io.bundle.channels.v1: stable|" \
    bundle/metadata/annotations.yaml
sed -i "s|com.redhat.openshift.versions:.*|com.redhat.openshift.versions: v${OCP_STREAM}-v4.22|" \
    bundle/metadata/annotations.yaml

Verify the key fields after editing:

grep -E "skipRange|containerImage:|com.redhat.openshift|channels" $CSV bundle/metadata/annotations.yaml
# spec.replaces must NOT appear:
grep "spec.replaces\|^  replaces:" $CSV && echo "ERROR: replaces still present" || echo "OK: no replaces"

Step 4.5 - Update the FBC catalog¶

Add the new version to catalog/catalog.yaml. Two sections need updating:

In the stable channel block (schema: olm.channel), append a new entry:

  - name: jupyter-notebook-validator-operator.v${VERSION}
    skipRange: ">=1.0.2 <${VERSION}"

Add a new olm.bundle block at the end of the file (copy from the previous version and update name, version, image, containerImage, createdAt, and com.redhat.openshift.versions):

---
schema: olm.bundle
name: jupyter-notebook-validator-operator.v${VERSION}
package: jupyter-notebook-validator-operator
image: quay.io/takinosh/jupyter-notebook-validator-operator-bundle:v${VERSION}
properties:
  - type: olm.package
    value:
      packageName: jupyter-notebook-validator-operator
      version: ${VERSION}
  - type: olm.gvk
    value:
      group: mlops.mlops.dev
      kind: NotebookValidationJob
      version: v1alpha1
  - type: olm.csv.metadata
    value:
      annotations:
        capabilities: Seamless Upgrades
        categories: Developer Tools, AI/Machine Learning
        certified: "false"
        containerImage: quay.io/takinosh/jupyter-notebook-validator-operator:${VERSION}
        createdAt: "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
        description: Automates Jupyter Notebook validation in MLOps workflows
        repository: https://github.com/tosin2013/jupyter-notebook-validator-operator
        support: Community
        com.redhat.openshift.versions: "v${OCP_STREAM}-v4.22"
        operatorframework.io/suggested-namespace: jupyter-notebook-validator-operator-system

Then validate the catalog:

make catalog-validate

All checks must pass before proceeding.

Step 5 - Validate the bundle¶

operator-sdk bundle validate ./bundle \
  --select-optional suite=operatorframework

All checks must pass with no errors before proceeding.

Step 6 - Build and push images¶

IMG=quay.io/takinosh/jupyter-notebook-validator-operator:${VERSION}
BUNDLE_IMG=quay.io/takinosh/jupyter-notebook-validator-operator-bundle:v${VERSION}

# Operator image
make docker-build docker-push IMG=${IMG}

# Bundle image
make bundle-build bundle-push BUNDLE_IMG=${BUNDLE_IMG}

# Verify images are accessible
docker pull ${IMG}
docker pull ${BUNDLE_IMG}

Step 7 - Tag the release and push¶

git add Makefile bundle/ helm/ config/
git commit -s -m "release: v${VERSION} for OCP ${OCP_STREAM}"
git push origin release-${OCP_STREAM}

# Tag triggers release.yml CI which builds and pushes images and creates
# the GitHub Release
git tag v${VERSION}
git push origin v${VERSION}

Monitor the Release workflow in GitHub Actions to confirm images are built and the GitHub Release is created.

Step 8 - Submit to OperatorHub (automated)¶

Use scripts/submit-to-operatorhub.sh to automate fork sync, branch creation, bundle copy, DCO commit, and PR opening for both upstream repos.

Dry run first to verify what will happen:

scripts/submit-to-operatorhub.sh ${VERSION} --dry-run

Submit to both repos:

scripts/submit-to-operatorhub.sh ${VERSION} --target both

Or submit to each repo individually:

# community-operators-prod first
scripts/submit-to-operatorhub.sh ${VERSION} --target community-operators-prod

# After the prod PR merges, submit to community-operators
scripts/submit-to-operatorhub.sh ${VERSION} --target community-operators

The script is idempotent: running it twice will detect existing branches and PRs and skip them. It validates prerequisites (gh auth, git DCO config), bundle metadata (provider, maintainer, containerImage), and checks for existing open PRs before submitting.

OperatorHub requires exactly one commit per PR. If you need to amend after pushing, squash before force-pushing - never add a second commit:
git add -A
git commit --amend -s --no-edit
git push --force origin add-jupyter-notebook-validator-operator-${VERSION}
If you already have multiple commits, squash them first:
git reset --soft HEAD~N   # N = number of commits to collapse
git commit -s -m "operator jupyter-notebook-validator-operator (${VERSION})"
git push --force origin add-jupyter-notebook-validator-operator-${VERSION}

Wait for each PR to pass all automated checks before proceeding.

File Based Catalog (FBC) Submission¶

Status: FBC is the recommended format for new OperatorHub submissions. The legacy bundle-directory approach (Step 8 above) remains available during the transition period while upstream repos complete their FBC migration.

Background¶

File Based Catalog (FBC) replaces the legacy SQLite-based index images with a declarative YAML catalog. The project already maintains an in-repo FBC at catalog/catalog.yaml with olm.package, olm.channel, and olm.bundle entries for each published version. CI validates this file on every PR with opm validate catalog/.

How FBC Works¶

The catalog file (catalog/catalog.yaml) contains three schema types:

olm.package - declares the operator package name, default channel, and description.
olm.channel - defines upgrade channels (stable, alpha) with entries listing each version and its upgrade path (skipRange or replaces).
olm.bundle - describes a specific version's bundle image, GVKs, package version, and CSV metadata annotations.

When submitting to OperatorHub via FBC, you submit the catalog tree instead of individual operators/<name>/<version>/ bundle directories.

Adding a New Version to the FBC Catalog¶

After building and pushing the bundle image (Steps 4-6):

# Option A: Use opm render to generate the bundle entry automatically
BUNDLE_IMG=quay.io/takinosh/jupyter-notebook-validator-operator-bundle:v${VERSION}
make catalog-render BUNDLE_IMG=${BUNDLE_IMG}

# Review the appended entries in catalog/catalog.yaml
# Then validate:
make catalog-validate

# Option B: Manually add entries (copy from previous version and update)
# See Step 4.5 above for the manual YAML format

After rendering or manually adding the entry, update the stable channel block to include the new version with its skipRange.

FBC Submission to Upstream Repos¶

When the upstream OperatorHub repos (community-operators-prod, community-operators) support FBC submissions, the PR structure changes:

Place the catalog entries under catalogs/ in the upstream repo (instead of operators/<name>/<version>/).
The PR title format remains: operator jupyter-notebook-validator-operator (VERSION).
One catalog entry per PR.

See the FBC Onboarding Guide for the latest upstream requirements.

Makefile Targets¶

Target	Description
`make catalog-render BUNDLE_IMG=...`	Render a bundle image into FBC YAML (appends to `catalog/catalog.yaml`)
`make catalog-validate`	Validate the FBC catalog directory with `opm validate`
`make catalog-build`	Build the catalog container image from the FBC directory
`make catalog-push`	Push the catalog container image

OperatorHub Submission Backlog¶

Current status as of April 2026:

Version	community-operators-prod	community-operators
v1.0.7	Merged ✓	Merged ✓
v1.0.8	PR #9442 - CI passing, awaiting merge	PR #7940 - CI passing, awaiting merge
v1.0.9	Submitted	Submitted
v1.0.10	Planned	Planned

Related issues: - #22 - Resolve OperatorHub submission backlog - #39 - Automate OperatorHub bundle submission script

Known Pipeline Pitfalls¶

These issues were encountered during the v1.0.8 submission and are documented here to prevent repeating them.

Pitfall 1 - `check_replaces_availability` fails for flat-structure bundles¶

Symptom: The operatorcert static test reports KeyError: '<version>' inside check_replaces_availability, even though the prior version exists in the upstream repo.

Root cause: The Bundle.probe() function in the operatorcert framework requires a manifests/ subdirectory to consider a directory a valid bundle. v1.0.7 was submitted to both upstream repos with manifests placed directly in the version root (no manifests/ subdir). As a result all_bundles() silently skips v1.0.7, and any bundle that sets spec.replaces: v1.0.7 gets a KeyError during the check.

Fix: Use olm.skipRange in the CSV metadata.annotations instead of spec.replaces in the CSV spec:

metadata:
  annotations:
    olm.skipRange: ">=1.0.2 <1.0.8"

The check_replaces_availability function returns immediately when spec.replaces is absent, so the broken lookup never happens. OLM still honours the skipRange for upgrade path decisions.

Future mitigation: Always ensure bundle directories have the correct nested structure (manifests/ and metadata/ subdirectories). Validate locally with:

operator-sdk bundle validate ./bundle

Pitfall 2 - Channel trap: `stable,alpha` with a `stable`-only `replaces` target¶

Symptom: check_replaces_availability fails with KeyError even when spec.replaces points to a version that appears to exist.

Root cause: The check runs once for every channel the submitted bundle is declared in. If the bundle is in stable,alpha and the replaces target (e.g. v1.0.7) is only in stable, the check fails when it processes the alpha channel because v1.0.7 is not visible there.

Fix: Submit to stable only unless there is an explicit reason for alpha. Set channels in bundle/metadata/annotations.yaml:

operators.operatorframework.io.bundle.channels.v1: stable
operators.operatorframework.io.bundle.channel.default.v1: stable

Do not set alpha unless you are intentionally maintaining an alpha channel with a full, unbroken upgrade chain from the oldest alpha bundle.

Pitfall 3 - Transient `ppc64le` IIB build failure¶

Symptom: The add-bundle-to-index task in the operator-hosted-pipeline fails with IIB build failed and Reason: Failed to build the container image on the arch ppc64le. All static and certification tasks passed.

Root cause: Red Hat's IIB (Index Image Builder) service occasionally encounters infrastructure failures on ppc64le build nodes. The failure is unrelated to the bundle content.

Fix: Retry the pipeline. Because the GitHub token used for fork operations may not have write access to comment on the upstream PR, the easiest retry method is a no-op force-push:

cd ~/forks/community-operators-prod
git commit --amend --no-edit
git push --force origin add-jupyter-notebook-validator-operator-${VERSION}

This triggers a new pipeline run without changing any bundle content. If the pipeline keeps failing on IIB for more than 2–3 retries, post a comment on the upstream PR asking a maintainer to run /pipeline restart operator-hosted-pipeline.

Troubleshooting¶

DCO Check Failed¶

# Rebase and add sign-off to the last commit
git rebase HEAD~1 --signoff
git push --force-with-lease origin your-branch

If multiple commits need sign-off:

git rebase -i HEAD~N   # mark commits as 'edit'
# For each commit:
git commit --amend --signoff --no-edit
git rebase --continue

containerImage Annotation Mismatch¶

The containerImage annotation in the CSV must exactly match the pushed image tag:

metadata:
  annotations:
    containerImage: quay.io/takinosh/jupyter-notebook-validator-operator:1.0.9

Verify with:

grep "containerImage:" bundle/manifests/*.clusterserviceversion.yaml | head -1

Multiple Bundles in community-operators-prod¶

If you see "The PR affects more than one bundle", you included more than one version directory. Create a separate PR for each version. Only one operators/<name>/<version>/ directory per PR is allowed.

Bundle Validation Errors¶

Run the full scorecard locally before opening the PR:

operator-sdk bundle validate ./bundle \
  --select-optional suite=operatorframework

operator-sdk scorecard bundle \
  --config tests/scorecard/config.yaml \
  --wait-time 120s

Release Scripts¶

# Full pre-submission test (builds, validates, runs scorecard)
./scripts/full-pre-submission-test.sh

# Local E2E test
./scripts/local-e2e-test.sh [tier1|tier2|tier3|metrics|webhook|all|basic]

# Pre-submission validation only
./scripts/pre-submission-validate.sh