Customizable diagnostic pod template for security-restricted environments

[pebrc]

Problem

In security-hardened Kubernetes clusters, the eck-diagnostics pod often fails to schedule or run because the hardcoded pod template does not meet cluster policies. Common blockers include:

• Kyverno / OPA Gatekeeper policies that require dropping all Linux capabilities, restricting seccompProfile, or enforcing readOnlyRootFilesystem
• Network policies that block egress/ingress unless pods carry specific labels or annotations
• Resource quotas that enforce minimum resource requests above the hardcoded 20Mi memory / 100m CPU (see Allow configuring resources in diagnostic pod template #390)
• Pod security standards (Restricted profile) that reject containers without a locked-down securityContext

The --image-pull-secrets flag (shipped in 1.10.0) solved the air-gap case, but users hitting policy enforcement still have to fork the tool and recompile with a modified job.tpl.yml — that's not a sustainable workaround.

What Users Need to Customize

Field	Why
securityContext (container + pod)	Drop capabilities, set runAsNonRoot, readOnlyRootFilesystem, seccompProfile
metadata.labels	Match network policy selectors that gate pod-to-pod communication
metadata.annotations	Sidecar injection control, monitoring opt-out, policy exceptions
resources.requests / resources.limits	Meet minimum resource quotas (#390)
Pod name pattern	Some clusters enforce naming conventions
serviceAccountName	Use a pre-provisioned SA with narrower RBAC

Approach Options

Option A: --pod-template flag (user-provided template file)

Allow users to supply their own complete pod template that overrides the built-in one:

eck-diagnostics --pod-template ./my-template.yaml ...

The tool would inject its runtime values (image, ES password env, volume mounts, command) into the user-provided template using the existing Go template variables ({{ .PodName }}, {{ .DiagnosticImage }}, etc.).

Pros: Maximum flexibility — users can customize anything. Solves all current and future customization requests in one shot.
Cons: Users must maintain a template that tracks internal changes (new template variables, command args). Breaking changes to the template contract are hard to communicate. Higher support burden.

Option B: --pod-template-patch flag (strategic merge patch)

Allow users to supply a partial YAML that is strategically merged into the built-in template:

eck-diagnostics --pod-template-patch ./patch.yaml ...

# patch.yaml
metadata:
  labels:
    network-policy: allow-es
spec:
  serviceAccountName: diagnostics-sa
  containers:
    - name: main  # matches by name
      securityContext:
        capabilities:
          drop: [ALL]
        runAsNonRoot: true
        readOnlyRootFilesystem: true
      resources:
        requests:
          memory: 64Mi
          cpu: 200m

Pros: Users only specify what they need to change. The base template evolves independently — patches survive template updates as long as the patched fields still exist.
Cons: Strategic merge patch semantics can be surprising (list replacement vs merge). Slightly more complex to implement.

Option C: Individual flags for common fields

Add targeted flags for the most common customization needs:

eck-diagnostics \
  --pod-labels "network-policy=allow-es,team=platform" \
  --pod-annotations "sidecar.istio.io/inject=false" \
  --security-context-drop-all \
  --memory-request 64Mi \
  --cpu-request 200m \
  --service-account diagnostics-sa \
  ...

Pros: Self-documenting, easy to use, no external files needed. Each flag can be validated independently.
Cons: Flag explosion over time. Will never cover every policy requirement — always one more field someone needs. Does not solve the general case.

Recommendation

Option B (strategic merge patch) provides the best balance. It solves the general case without requiring users to track template internals, and avoids flag proliferation. Option C flags could be added later for the most common fields (--pod-labels, --security-context-drop-all) as convenience shortcuts.

Relationship to Other Issues

• Subsumes Allow configuring resources in diagnostic pod template #390 (resource limits customization)
• Complements --image-pull-secrets (already shipped in 1.10.0)
• Complements Provide options to run Elasticsearch and Kibana from ECK-Diagnostic without elevated permissions #349 (API-only collection mode for restricted RBAC)