ADR 019: RBAC & Pod Security Policies for Notebook Secret Access¶

Status¶

Accepted

Context¶

ADR-014 through ADR-018 established credential injection patterns and rotation policies. This ADR addresses access control and security hardening for notebook secret access.

Problem¶

Notebooks execute arbitrary user code, which creates security risks: - Unauthorized secret access: Notebooks could access secrets they shouldn't - Privilege escalation: Compromised notebooks could escalate privileges - Lateral movement: Compromised notebooks could access other namespaces - No audit trail: No visibility into which notebooks accessed which secrets

Security Principles¶

Least Privilege: Grant minimum permissions required
Defense in Depth: Multiple layers of security controls
Separation of Concerns: Isolate workloads by namespace
Audit Trail: Log all secret access
Pod Security Standards: Enforce baseline security policies

Decision¶

We implement RBAC policies and Pod Security Standards to enforce least-privilege access to secrets.

RBAC Strategy¶

Principle: Each notebook namespace has its own ServiceAccount with access only to secrets in that namespace.

Architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Namespace: notebook-validation            │
│                                                              │
│  ┌────────────────────────────────────────────────────┐    │
│  │  ServiceAccount: notebook-validator-sa             │    │
│  │  - Can read secrets in this namespace only         │    │
│  │  - Cannot read secrets in other namespaces         │    │
│  │  - Cannot create/update/delete secrets             │    │
│  └────────────────────────────────────────────────────┘    │
│                          │                                   │
│                          │ RBAC Role binding                 │
│                          ▼                                   │
│  ┌────────────────────────────────────────────────────┐    │
│  │  Role: notebook-secret-reader                      │    │
│  │  - get, list secrets                               │    │
│  │  - Only in this namespace                          │    │
│  └────────────────────────────────────────────────────┘    │
│                          │                                   │
│                          │ Mounts secrets                    │
│                          ▼                                   │
│  ┌────────────────────────────────────────────────────┐    │
│  │  NotebookValidationJob Pod                         │    │
│  │  - Uses notebook-validator-sa                      │    │
│  │  - Can only access secrets in this namespace       │    │
│  └────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

RBAC Resources¶

1. ServiceAccount (per namespace):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: notebook-validator-sa
  namespace: notebook-validation
automountServiceAccountToken: true

2. Role (namespace-scoped):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: notebook-secret-reader
  namespace: notebook-validation
rules:
  # Allow reading secrets (for env var injection)
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  # Allow reading configmaps (for configuration)
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]

3. RoleBinding (binds ServiceAccount to Role):

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: notebook-validator-secret-reader
  namespace: notebook-validation
subjects:
  - kind: ServiceAccount
    name: notebook-validator-sa
    namespace: notebook-validation
roleRef:
  kind: Role
  name: notebook-secret-reader
  apiGroup: rbac.authorization.k8s.io

4. NotebookValidationJob (uses ServiceAccount):

apiVersion: mlops.mlops.dev/v1alpha1
kind: NotebookValidationJob
metadata:
  name: s3-pipeline-validation
  namespace: notebook-validation
spec:
  notebook:
    git:
      url: https://github.com/example/notebooks.git
      ref: main
      path: s3-pipeline.ipynb
  podConfig:
    containerImage: quay.io/jupyter/scipy-notebook:latest
    serviceAccountName: notebook-validator-sa  # Uses least-privilege SA
    env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: aws-credentials
            key: access-key-id

Pod Security Standards¶

Principle: Enforce baseline security policies for all notebook pods.

Policy: Use Restricted Pod Security Standard (most secure)

Namespace Label:

apiVersion: v1
kind: Namespace
metadata:
  name: notebook-validation
  labels:
    # Enforce Restricted Pod Security Standard
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Restricted Policy Requirements: - ✅ Run as non-root user - ✅ Drop all capabilities - ✅ No privilege escalation - ✅ Read-only root filesystem (where possible) - ✅ Seccomp profile (RuntimeDefault) - ✅ No host namespaces (hostNetwork, hostPID, hostIPC)

Pod Security Context (enforced by operator):

spec:
  podConfig:
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      fsGroup: 1000
      seccompProfile:
        type: RuntimeDefault
    containers:
      - name: notebook-validator
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
          readOnlyRootFilesystem: false  # Notebooks need to write to /tmp

Network Policies (Optional)¶

Principle: Restrict network access to only required services.

Example:

kindmet   spec

href="#__codelineno-7-1">apiVersion: networking.k8s.io/v1 class="p">: NetworkPolicy adata: name: notebook-validator-netpol namespace: notebook-validation class="p">: podSelector: matchLabels: app: notebook-validator policyTypes: - Ingress - Egress ingress: [] # No ingress (notebooks don't receive traffic) egress: # Allow DNS - to: - namespaceSelector: matchLabels: name: kube-system ports: - protocol: UDP port: 53 # Allow HTTPS to external services (S3, APIs) - to: - namespaceSelector: {} ports: - protocol: TCP port: 443 # Allow database access - to: - podSelector: matchLabels: app: postgres ports: - protocol: TCP port: 5432

Audit Logging¶

Principle: Log all secret access for audit trail.

Kubernetes Audit Policy:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log secret access
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
    namespaces: ["notebook-validation"]
  # Log pod creation (to track which notebooks ran)
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]
    namespaces: ["notebook-validation"]
    verbs: ["create"]

Audit Log Query (find secret access):

# Find all secret access in last 24 hours
kubectl logs -n kube-system kube-apiserver-* | \
  jq 'select(.objectRef.resource=="secrets" and .objectRef.namespace=="notebook-validation")'

Consequences¶

Positive¶

Least Privilege: Notebooks can only access secrets in their namespace
Defense in Depth: Multiple security layers (RBAC, PSS, NetworkPolicy)
Audit Trail: All secret access logged
Compliance: Meets SOC 2, PCI-DSS requirements
Reduced Attack Surface: Restricted capabilities, no privilege escalation

Negative¶

Operational Overhead: More RBAC resources to manage
Complexity: More complex security model
Potential Breakage: Restrictive policies may break some notebooks

Neutral¶

No Code Changes: Operator enforces security policies automatically
Optional NetworkPolicy: Users can add NetworkPolicy if needed

Implementation¶

Phase 1: RBAC Setup¶

Create ServiceAccount template
Create Role template
Create RoleBinding template
Document RBAC setup

Phase 2: Pod Security Standards¶

Update operator to enforce Restricted PSS
Add securityContext to Pod spec
Test with Restricted PSS

Phase 3: Audit Logging¶

Configure Kubernetes audit policy
Create audit log queries
Document audit procedures

Phase 4: Network Policies (Optional)¶

Create NetworkPolicy templates
Test network isolation
Document NetworkPolicy setup

Security Checklist¶

Before deploying notebooks: - [ ] ServiceAccount created with least-privilege RBAC - [ ] Namespace labeled with Pod Security Standard (Restricted) - [ ] Secrets created with appropriate access controls - [ ] Audit logging enabled - [ ] NetworkPolicy configured (optional) - [ ] Security context enforced (runAsNonRoot, drop capabilities)

Troubleshooting¶

Issue: Pod fails with "forbidden: violates PodSecurity"¶

Symptoms: Pod creation fails with PSS violation

Diagnosis:

kubectl describe pod <pod-name> -n notebook-validation

Common Causes: - Running as root (runAsNonRoot: false) - Privilege escalation allowed - Capabilities not dropped

Solution: Update Pod securityContext to meet Restricted PSS

Issue: "forbidden: User cannot get resource 'secrets'"¶

Symptoms: Pod fails to mount secret