Jupyter Notebook Validator Operator - Architecture Overview

Executive Summary

The Jupyter Notebook Validator Operator is a Kubernetes-native tool designed to automate the validation of Jupyter Notebooks within MLOps workflows. This document provides a high-level overview of the architectural decisions documented in our ADRs.

Quick Reference

• Language: Go 1.22+
• Framework: Operator SDK v1.37.0+
• Primary Platform: OpenShift 4.20+ (certified: 4.20/4.21/4.22 per ADR-056), Kubernetes 1.31-1.35
• CRD: NotebookValidationJob (mlops.dev/v1alpha1)
• Deployment: OLM Bundle (primary), Helm Chart (secondary), Raw Manifests (tertiary)
• Distribution: OpenShift OperatorHub, Red Hat Catalog, OperatorHub.io, Artifact Hub

Architectural Decisions

1. Technology Stack (ADR 001)

Decision: Operator SDK v1.32.0+ with Go Modules

Why: - Long-term API stability - Native OpenShift integration - Comprehensive tooling and community support - Built-in controller-runtime

Impact: Standardized project structure, reduced boilerplate, seamless OLM integration

2. Platform Support Strategy (ADR 002, ADR 006)

Decision: Phased rollout with hybrid support model

Phases: 1. Phase 1: OpenShift 4.20 foundation (released) 2. Phase 2: OpenShift 4.21 and 4.22 expansion (current) 3. Phase 3 (Months 7-9): Kubernetes 1.31+ community support

Support Tiers: - Tier 1: OpenShift 4.20-4.22 (certified, full support per ADR-056) - Tier 2: Kubernetes 1.31-1.35 (community, best-effort)

Why: Balances enterprise needs with community adoption, manages risk through incremental expansion

Impact: Clear roadmap, manageable testing complexity, broad platform coverage

3. API Design (ADR 003)

Decision: Multi-version CRD with conversion webhooks

API Versioning: - v1alpha1: Initial experimental version - v1beta1: Stable API (future) - v1: Production-ready (future)

Key Features: - OpenAPI v3 schema validation - Status subresource separation - Server-side defaulting - Conversion webhooks for smooth upgrades

Why: Enables API evolution without breaking users, follows Kubernetes best practices

Impact: Smooth upgrade path, strong validation, backward compatibility

4. Deployment & Packaging (ADR 004, ADR 007)

Decision: Multi-format packaging with phased distribution

Packaging Formats: 1. OLM Bundle: Primary for OpenShift, automatic upgrades 2. Helm Chart: Secondary for Kubernetes, templating support 3. Raw Manifests: Tertiary for GitOps/CI/CD, maximum flexibility

Distribution Channels (by phase):

Phase	OpenShift OperatorHub	Red Hat Catalog	OperatorHub.io	Artifact Hub
1	✅ Community	❌	❌	❌
2	✅ Community	✅ Certified	❌	❌
3	✅ Community	✅ Certified	✅ Community	✅ Helm

Why: Best-in-class UX for each platform, maximum discoverability, broad adoption

Impact: Maintenance burden across formats, but superior user experience

5. Security & RBAC (ADR 005)

Decision: Least-privilege RBAC with hybrid deployment model

Service Accounts: - Operator SA: jupyter-notebook-validator-operator (cluster or namespace-scoped) - Validation Pod SA: jupyter-notebook-validator-runner (minimal permissions)

Deployment Modes: - Cluster-Wide: Operator watches all namespaces (platform teams) - Namespace-Scoped: Operator watches single namespace (self-service) - Multi-Namespace: Operator watches specific namespaces (advanced)

Security Features: - Non-root containers - Read-only root filesystem - Dropped capabilities - OpenShift SCC compliance

Why: Follows security best practices, supports multiple deployment scenarios, audit-friendly

Impact: Flexible deployment, enhanced security posture, compliance-ready

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Kubernetes/OpenShift Cluster              │
│                                                              │
│  ┌────────────────────────────────────────────────────┐    │
│  │  Operator Namespace                                 │    │
│  │                                                     │    │
│  │  ┌──────────────────────────────────────────┐     │    │
│  │  │  Jupyter Notebook Validator Operator     │     │    │
│  │  │  (Deployment)                            │     │    │
│  │  │                                          │     │    │
│  │  │  - Watches NotebookValidationJob CRs    │     │    │
│  │  │  - Reconciles desired state             │     │    │
│  │  │  - Creates validation pods              │     │    │
│  │  │  - Updates CR status                    │     │    │
│  │  └──────────────────────────────────────────┘     │    │
│  └────────────────────────────────────────────────────┘    │
│                                                              │
│  ┌────────────────────────────────────────────────────┐    │
│  │  User Namespace                                     │    │
│  │                                                     │    │
│  │  ┌──────────────────────────────────────────┐     │    │
│  │  │  NotebookValidationJob CR                │     │    │
│  │  │  (Custom Resource)                       │     │    │
│  │  │                                          │     │    │
│  │  │  spec:                                   │     │    │
│  │  │    notebook:                             │     │    │
│  │  │      git: {url, ref, path}              │     │    │
│  │  │    podConfig: {image, resources}        │     │    │
│  │  └──────────────────────────────────────────┘     │    │
│  │                                                     │    │
│  │  ┌──────────────────────────────────────────┐     │    │
│  │  │  Validation Pod (ephemeral)              │     │    │
│  │  │                                          │     │    │
│  │  │  1. Clone Git repository                │     │    │
│  │  │  2. Execute notebook cells              │     │    │
│  │  │  3. Compare with golden (optional)      │     │    │
│  │  │  4. Report results                      │     │    │
│  │  └──────────────────────────────────────────┘     │    │
│  └────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

Validation Workflow

1. User creates NotebookValidationJob CR

   apiVersion: mlops.dev/v1alpha1
   kind: NotebookValidationJob
   metadata:
     name: validate-my-notebook
   spec:
     notebook:
       git:
         url: https://github.com/org/repo.git
         ref: main
         path: notebooks/analysis.ipynb
   

2. Operator reconciles CR

3. Validates CR spec
4. Fetches notebook from Git

5. Creates validation pod with appropriate configuration

6. Validation pod executes

7. Clones Git repository
8. Executes notebook cells sequentially
9. Captures outputs and errors

10. Compares with golden notebook (if specified)

11. Operator updates CR status

    status:
      phase: Succeeded  # or Failed
      results:
        - cellIndex: 0
          status: Success
        - cellIndex: 1
          status: Success
    

Development Roadmap

Phase 1: OpenShift 4.20 Foundation

Goals: - ✅ Core operator functionality - ✅ Production-ready on OpenShift 4.20+ - ✅ OLM bundle published to community catalog - ✅ Comprehensive test suite

Deliverables: - Operator v0.1.0 - CRD v1alpha1 - Documentation and examples - CI/CD pipeline

Phase 2: OpenShift Expansion (Months 4-6)

Goals: - ✅ Support OpenShift 4.20, 4.21, and 4.22 - ✅ Red Hat certification - ✅ Multi-version testing - ✅ API stability improvements

Deliverables: - Operator v0.2.0 - Certified operator bundle - Version compatibility matrix - Upgrade guides

Phase 3: Kubernetes Community (Months 7-9)

Goals: - ✅ Support Kubernetes 1.31-1.35 - ✅ Helm chart distribution - ✅ Community adoption - ✅ Multiple distribution channels

Deliverables: - Operator v0.3.0 - Helm chart on Artifact Hub - OperatorHub.io listing - Community documentation

Testing Strategy

Test Pyramid

                    ┌─────────────┐
                    │   E2E Tests │  (Multi-version, multi-platform)
                    └─────────────┘
                  ┌───────────────────┐
                  │ Integration Tests │  (envtest, CRD validation)
                  └───────────────────┘
              ┌─────────────────────────────┐
              │       Unit Tests            │  (>80% coverage)
              └─────────────────────────────┘

Test Matrix (Phase 3)

Test Type	OpenShift 4.20	OpenShift 4.21	OpenShift 4.22	K8s 1.31-1.35
Unit	✅	✅	✅	✅
Integration	✅	✅	✅	✅
E2E	✅	✅	✅	✅

Key Metrics

Quality Metrics

• Test coverage: >80%
• E2E test pass rate: >95%
• Mean time to recovery: <4 hours
• Bug escape rate: <5%

Adoption Metrics

• Active installations
• GitHub stars and forks
• Community contributions
• Support ticket volume

Performance Metrics

• Reconciliation duration: <5s
• Validation job success rate: >90%
• Resource utilization: <200Mi memory, <200m CPU

Next Steps

1. Initialize Project (Week 1)

   operator-sdk init --domain mlops.dev --repo github.com/your-org/jupyter-notebook-validator-operator
   operator-sdk create api --group mlops --version v1alpha1 --kind NotebookValidationJob --resource --controller
   

2. Implement Core Features (Weeks 2-8)

3. CRD types and validation
4. Controller reconciliation logic
5. Git integration
6. Pod orchestration

7. Status reporting

8. Build Test Suite (Weeks 9-10)

9. Unit tests
10. Integration tests

11. E2E tests on OpenShift 4.20+

12. Create OLM Bundle (Week 11)

13. ClusterServiceVersion
14. Bundle manifests

15. Submit to community catalog

16. Documentation (Week 12)

17. Installation guide
18. User guide
19. API reference
20. Troubleshooting

References

ADRs

• ADR 001: Operator Framework and SDK Version
• ADR 002: Platform Version Support Strategy
• ADR 003: CRD Schema Design and Versioning
• ADR 004: Deployment and Packaging Strategy
• ADR 005: RBAC and Service Account Model
• ADR 006: Version Support Roadmap and Testing
• ADR 007: Distribution and Catalog Strategy

External Resources

• Operator SDK Documentation
• Kubernetes Operator Pattern
• OpenShift Documentation
• OLM Documentation

Questions?

For questions about the architecture or ADRs: - Open an issue in the GitHub repository - Review the detailed ADRs in docs/adrs/ - Contact the platform team

---

Last Updated: 2025-11-07 Version: 1.0 Status: Living Document