ADR-027: S2I Build Strategy for Git Integration¶

Status: Superseded Date: 2025-01-08 Updated: 2025-11-10 Related: ADR-009 (Secret Management), ADR-023 (S2I Build - Archived), ADR-028 (Tekton Task Strategy), ADR-031 (Tekton Build - Primary)

Status¶

Superseded

Context¶

During testing on OpenShift 4.18, we encountered a security constraint issue where the alpine/git:latest image used for git-clone init containers runs as root, violating OpenShift's runAsNonRoot security context requirement.

The Problem¶

Error: container has runAsNonRoot and image will run as root 
(pod: "test-basic-math-validation_default(...)", container: git-clone)

Initial Solution (Quick Fix)¶

We initially fixed this by switching to bitnami/git:latest, which supports running as a non-root user. This works but has limitations: - Git clone happens at runtime (slower pod startup) - Network dependency at pod creation time - Init container overhead - Dependencies installed at runtime via pip

The Insight 💡¶

User's Key Observation: "This is a perfect use case for the build config or tekton!"

This insight is absolutely correct and represents a fundamental shift in thinking:

Instead of working around the git image issue, we should eliminate the need for a git-clone init container entirely by using S2I/Tekton builds.

Decision¶

We will support both approaches with S2I/Tekton builds as the recommended production approach:

Approach 1: Init Container (Development/Quick Tests)¶

Use bitnami/git:latest for git-clone init container
Quick iteration, simple debugging
Good for development and ad-hoc testing
When to use: Development, quick tests, debugging

Approach 2: S2I/Tekton Build (Production/CI/CD) 🎯 RECOMMENDED¶

Build custom image with notebook repository baked in
No init container needed
All dependencies pre-installed
Faster pod startup, better security
When to use: Production, CI/CD pipelines, performance-critical workloads

Rationale¶

Why S2I/Tekton is Superior¶

1. Performance¶

Init Container Approach:
├── Pod Creation: 0s
├── Init Container (git-clone): 10-30s
│   ├── Pull bitnami/git image: 5-10s
│   ├── Clone repository: 5-15s
│   └── Verify notebook: 1-2s
├── Main Container Start: 0s
├── Install dependencies (pip): 30-120s
└── Execute notebook: Variable
Total Startup: 40-150s

S2I Build Approach:
├── Build (one-time): 5-10 minutes
│   ├── Clone repository
│   ├── Install dependencies
│   └── Create image
├── Pod Creation: 0s
├── Pull built image: 5-10s
└── Execute notebook: Variable
Total Startup (after build): 5-10s

Result: 4-15x faster pod startup after initial build!

2. Security¶

Init Container:
- Runtime dependency installation (pip install)
- Network access required at pod creation
- Dependencies not vetted/scanned
- Git credentials in pod environment

S2I Build:
- Dependencies installed at build time
- Image can be scanned for vulnerabilities
- No runtime network dependency
- Git credentials only in build environment
- Immutable artifact

3. Reproducibility¶

Init Container:
- Git ref can change (branch moves)
- Dependency versions can change (pip install latest)
- Network issues can cause failures
- Different results on different runs

S2I Build:
- Image is immutable
- Same image = same result
- Can be tagged and versioned
- Cacheable and reusable

4. OpenShift Native¶

S2I is an OpenShift-native feature:
- Integrated with OpenShift Build API
- Automatic image stream management
- Built-in security scanning
- RBAC integration
- Audit trail

Why Keep Init Container Approach?¶

Development Velocity: Quick iteration without build overhead
Debugging: Easier to debug git clone issues
Flexibility: Can test different branches/commits quickly
Fallback: Works when S2I/Tekton not available

Implementation¶

Development Best Practice: CRD Research First¶

Key Learning: Always research the actual CRDs on the target cluster before implementing build/pipeline logic.

# Research OpenShift Build API
oc api-resources | grep build
oc explain buildconfig.spec --recursive
oc explain buildconfig.spec.triggers
oc explain build.status

# Research Tekton Pipeline API
oc api-resources | grep -E "tekton|pipeline"
oc explain pipelinerun --recursive
oc explain pipeline --recursive

Why This Matters: 1. Discover Auto-Trigger Mechanisms: Found ConfigChangeBuildTriggerType for automatic build triggering 2. Understand API Capabilities: Learn what the platform provides natively 3. Avoid Reinventing: Don't implement what the platform already does 4. Correct API Usage: Use proper subresources and trigger mechanisms

S2I Build Workflow¶

apiVersion: mlops.mlops.dev/v1alpha1
kind: NotebookValidationJob
metadata:
  name: production-validation
spec:
  notebook:
    git:
      url: "https://github.com/org/notebooks.git"
      ref: "main"
    path: "notebooks/model-validation.ipynb"

  podConfig:
    buildConfig:
      enabled: true                    # Enable S2I build
      strategy: "s2i"                  # Use S2I strategy
      baseImage: "quay.io/jupyter/minimal-notebook:latest"
      requirementsFile: "requirements.txt"
      timeout: "15m"

    containerImage: "quay.io/jupyter/minimal-notebook:latest"  # Fallback

Controller Logic¶

func (r *NotebookValidationJobReconciler) reconcileValidation(ctx context.Context, job *mlopsv1alpha1.NotebookValidationJob) (ctrl.Result, error) {
    containerImage := job.Spec.PodConfig.ContainerImage

    // Check if build is enabled
    if isBuildEnabled(job) {
        logger.Info("Build integration enabled, handling build workflow")

        // Handle S2I/Tekton build
        builtImage, err := r.handleBuildIntegration(ctx, job)
        if err != nil {
            logger.Error(err, "Build failed, falling back to container image")
            // Fall back to init container approach
        } else {
            logger.Info("Build completed, using built image", "image", builtImage)
            containerImage = builtImage
            // NO INIT CONTAINER NEEDED - notebook is in the image!
        }
    }

    // Create validation pod
    pod, err := r.createValidationPod(ctx, job, containerImage)
    // ...
}

Automatic Build Triggering with ConfigChange¶

Problem: Builds created without triggers stay in "New" status and require manual triggering.

Solution: Add ConfigChangeBuildTriggerType to BuildConfig triggers.

// pkg/build/s2i_strategy.go - BuildConfig with auto-trigger
bc := &buildv1.BuildConfig{
    ObjectMeta: metav1.ObjectMeta{
        Name:      buildName,
        Namespace: job.Namespace,
    },
    Spec: buildv1.BuildConfigSpec{
        CommonSpec: buildv1.CommonSpec{
            Source:   buildSource,
            Strategy: buildStrategy,
            Output:   buildOutput,
        },
        // Add ConfigChange trigger to automatically start build on creation
        Triggers: []buildv1.BuildTriggerPolicy{
            {
                Type: buildv1.ConfigChangeBuildTriggerType,
            },
        },
    },
}

Result: Build automatically starts when BuildConfig is created - no manual oc start-build needed!

From CRD Research:

$ oc explain buildconfig.spec.triggers
...
- ConfigChange ConfigChangeBuildTriggerType will trigger a build on an
  initial build config creation

Build Strategy Detection¶

// pkg/build/s2i_strategy.go
func (s *S2IStrategy) Detect(ctx context.Context, client client.Client) (bool, error) {
    // Check if OpenShift Build API is available
    buildConfig := &buildv1.BuildConfig{}
    err := client.Get(ctx, types.NamespacedName{Name: "test", Namespace: "default"}, buildConfig)

    if err != nil {
        if errors.IsNotFound(err) {
            return true, nil  // API exists, resource doesn't
        }
        if runtime.IsNotRegisteredError(err) {
            return false, nil  // API doesn't exist
        }
        return false, err
    }

    return true, nil
}

Consequences¶

Positive¶

Performance: Dramatically faster pod startup for production workloads
Security: Better security posture with pre-built, scanned images
Reproducibility: Immutable images ensure consistent results
OpenShift Native: Leverages platform capabilities
Flexibility: Supports both approaches based on use case

Negative¶

Complexity: More complex workflow with build management
Build Time: Initial build takes 5-10 minutes
Storage: Built images consume registry storage
Learning Curve: Users need to understand S2I/Tekton

Neutral¶

Dual Approach: Need to maintain both code paths
Documentation: Need clear guidance on when to use each approach
Testing: Need to test both workflows

Migration Path¶

Phase 1: Fix Init Container (✅ Complete)¶

Switch to bitnami/git:latest
Ensure OpenShift compatibility
Document approach

Phase 2: Enable S2I Builds (✅ Complete)¶

Implement build integration in controller
Add S2I strategy detection
Create sample configurations
Register OpenShift Build API types in scheme

Phase 3: CRD Research and Auto-Trigger (✅ Complete)¶

Research OpenShift Build CRDs on cluster
Discover ConfigChange trigger mechanism
Implement automatic build triggering
Add ConfigChangeBuildTriggerType to BuildConfig
Test auto-trigger functionality

Phase 4: Test and Validate (⏳ In Progress)¶

Test init container approach with bitnami/git
Test S2I build workflow with auto-trigger
Verify builds start automatically
Performance benchmarking

Phase 5: Documentation and Best Practices (⏳ Pending)¶

Document both approaches
Provide decision matrix
Create migration guide
Update samples
Document CRD research best practices

Phase 6: Tekton Integration (⏳ Pending)¶

Research Tekton Pipeline CRDs on cluster
Implement Tekton strategy with auto-trigger
Support custom pipelines
Advanced build scenarios

Alternatives Considered¶

Alternative 1: Always Use Init Container¶

Rejected: Poor performance, runtime dependencies, security concerns

Alternative 2: Always Use S2I Build¶

Rejected: Too slow for development, overkill for simple tests

Alternative 3: Custom Git Image¶

Rejected: Maintenance burden, reinventing the wheel

Alternative 4: Pre-built Image Library¶

Considered for Future: Maintain library of pre-built images tagged by repo+commit

References¶

OpenShift S2I Documentation
Bitnami Git Container
ADR-005: OpenShift Compatibility
ADR-009: Git Integration with Credentials
Phase 4.5: S2I Build Integration
docs/GIT-CLONE-APPROACHES.md

Notes¶

This ADR captures a key architectural insight: instead of working around platform constraints, leverage platform capabilities. The S2I build approach transforms a problem (git-clone security) into an opportunity (better performance and security through pre-built images).

The user's observation that "this is a perfect use case for build config" represents exactly the kind of pragmatic, platform-aware thinking that leads to better architecture.

Decision Date¶

2025-11-09

Decision Makers¶

Development Team
User (key insight)

Review Date¶

2025-12-09 (30 days)