Skip to content

ADR-023: Strategy for Source-to-Image (S2I) Build Integration on OpenShift

Status

Superseded

Date

2025-01-08 Archived: 2025-11-10

Superseded By

  • ADR-027: S2I Build Strategy for Git Integration (accepted implementation)
  • ADR-031: Tekton Build Strategy (current primary build method)

Context

The Jupyter Notebook Validator Operator needs to execute notebooks that may have custom dependencies. On OpenShift, Security Context Constraints (SCC) prevent containers from running as specific UIDs and writing to certain directories, causing runtime pip install failures.

Current Problem

  • Standard Jupyter images expect to run as UID 1000 with /home/jovyan as the home directory
  • OpenShift's restricted-v2 SCC assigns random UIDs (e.g., 1000920000) to containers
  • When pip tries to install packages to /home/jovyan/.local, it fails with "Permission denied" errors
  • The operator currently attempts runtime pip installation, which fails on OpenShift

Industry Patterns

Major cloud ML platforms solve this by pre-building container images with dependencies before execution: - Azure ML: Environment class automatically builds Docker images from requirements.txt - AWS SageMaker: Bring Your Own Container (BYOC) pattern - Google Vertex AI: Recommends custom containers for production (faster startup, better security)

OpenShift Capabilities

OpenShift provides native Source-to-Image (S2I) capability that can: - Automatically build container images from source code - Detect and install dependencies from requirements.txt - Create immutable, reproducible images - Integrate with OpenShift's image registry and build system

Decision

Adopt OpenShift's Source-to-Image (S2I) as the primary build mechanism for creating notebook validation container images with custom dependencies.

Implementation Approach

When a NotebookValidationJob is created with an optional buildConfig specification:

  1. Detect OpenShift by checking for the build.openshift.io API group
  2. Create an S2I BuildConfig that references the notebook's Git repository
  3. Use a Jupyter-compatible S2I builder image as the base for dependency installation
  4. Trigger the build and wait for its completion before launching the validation pod
  5. Use the resulting image for the validation pod rather than installing dependencies at runtime

CRD Schema Extension

apiVersion: mlops.redhat.com/v1alpha1
kind: NotebookValidationJob
spec:
  podConfig:
    containerImage: "quay.io/jupyter/minimal-notebook:latest"  # Optional: use pre-built image
    buildConfig:  # Optional: trigger S2I build
      enabled: true
      strategy: "s2i"  # Default strategy
      baseImage: "quay.io/jupyter/minimal-notebook:latest"
      autoGenerateRequirements: false  # See ADR-024
      requirementsFile: "requirements.txt"  # Path in git repo

Opt-In Behavior

The buildConfig field is optional, allowing users to: - Option 1: Provide a pre-built image via containerImage (no build) - Option 2: Enable buildConfig to trigger automatic S2I builds - Option 3: Use default image with runtime pip installation (current behavior, may fail on OpenShift)

Consequences

Positive Consequences

  1. Eliminates runtime pip failures on OpenShift by moving dependency installation to build phase
  2. Immutable, reproducible images: Dependencies are baked in and cannot drift at runtime
  3. Faster notebook validation: No per-job installation overhead
  4. Improved security posture: Dependencies installed at build time under controlled contexts
  5. Alignment with major ML platforms: Follows Azure ML, SageMaker, Vertex AI patterns
  6. Leverages native OpenShift tooling: No external CI/CD dependencies needed
  7. Image caching: Built artifacts can be reused across multiple validation jobs
  8. Better error handling: Build failures are clearer than runtime permission errors

Negative Consequences

  1. OpenShift coupling: Ties the solution to OpenShift's S2I API, reducing portability to vanilla Kubernetes
  2. Prerequisite requirements: Requires OpenShift cluster with BuildConfig APIs and registry access
  3. Additional operator complexity: Build orchestration logic must be maintained and tested
  4. Longer time-to-first-validation: Initial builds add latency before notebook execution
  5. Registry storage usage: Builds and pushed images consume registry storage quotas
  6. Learning curve: Users must understand S2I concepts and configuration

Risks and Mitigations

Risk Impact Mitigation
Build failures harder to debug than runtime pip errors Medium Stream build logs through operator; surface clear error messages; link to troubleshooting docs
S2I builder images become outdated or incompatible Medium Define and version S2I builder image requirements; update in lockstep with Jupyter base images
Network policies block registry access during build/push High Document required network and registry permissions; provide guidance to cluster administrators
Users unaware of fallback options if S2I unavailable Low Document fallback to user-supplied pre-built images via optional buildConfig field
Build queue delays in multi-tenant clusters Medium Implement build timeout and retry logic; document expected build times

Alternatives Considered

1. Kaniko-based builds inside Kubernetes pods

Rejected: Requires privileged containers or complex RBAC configurations, which conflicts with security goals.

2. Buildah via local pod

Rejected: Similar security concerns as Kaniko; requires elevated privileges.

3. External CI pipelines (Tekton, GitHub Actions)

Rejected for core: Adds external dependencies and complexity. Documented as community contribution in ADR-025.

4. Runtime pip installation with custom base images (current approach)

Rejected: Fails on OpenShift due to SCC restrictions; poor user experience.

5. Require users to always provide pre-built images

Rejected: Poor user experience; high barrier to entry for notebook validation.

Implementation Tasks

  1. CRD Schema Updates
  2. Add buildConfig field to PodConfig struct
  3. Add validation for buildConfig fields

  4. Update CRD documentation and examples

  5. Platform Detection

  6. Extend pkg/platform/detector.go to detect OpenShift
  7. Check for build.openshift.io API group

  8. Check for image.openshift.io API group

  9. Build Orchestration

  10. Implement S2I BuildConfig creation logic
  11. Add build status monitoring and waiting
  12. Stream build logs to operator events

  13. Handle build failures with clear error messages

  14. Controller Integration

  15. Update reconciliation loop to check for buildConfig
  16. Trigger builds before creating validation pods

  17. Use built image reference for validation pod

  18. Documentation

  19. Create S2I prerequisites guide
  20. Document BuildConfig configuration options
  21. Add troubleshooting guide for build failures

  22. Provide example NotebookValidationJob manifests

  23. Testing

  24. Unit tests for build orchestration logic
  25. Integration tests on OpenShift cluster
  26. E2E tests covering successful and failed builds
  27. Performance tests for build caching
  • ADR-008: Notebook Testing Strategy and Complexity Levels
  • ADR-011: Error Handling and Retry Strategy
  • ADR-019: RBAC and Pod Security Policies for Notebook Secret Access
  • ADR-021: OpenShift-Native Dashboard Strategy
  • ADR-024: Fallback Strategy for Notebooks Missing requirements.txt (companion ADR)
  • ADR-025: Community-Contributed Build Methods and Extension Framework (companion ADR)

References