Skip to content

Flagd sidecar injection causes evaluation errors on pod creation/termination #806

@Y0sh1dk

Description

@Y0sh1dk

Bug

Injected Flagd container is added as an additional container via spec.containers rather than as a native Kubernetes sidecar (spec.initContainers with restartPolicy: Always) which was made stable in v1.33 and was first enabled by default in v1.29. This results in flag evaluation failures on pod creation/termination due to no ordering guarantees between the containers.

How to reproduce

  1. Apply minimal FeatureFlag, FeatureFlagSource and ReplicaSet resources
apiVersion: core.openfeature.dev/v1beta1
kind: FeatureFlag
metadata:
  name: test
  namespace: default
spec:
  flagSpec:
    flags:
      test-flag:
        state: ENABLED
        variants:
          "on": true
          "off": false
        defaultVariant: "on"
---
apiVersion: core.openfeature.dev/v1beta1
kind: FeatureFlagSource
metadata:
  name: test
  namespace: default
spec:
  sources:
    - source: default/test
      provider: kubernetes
---
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: flag-evaluator
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flag-evaluator
  template:
    metadata:
      labels:
        app: flag-evaluator
      annotations:
        openfeature.dev/enabled: "true"
        openfeature.dev/featureflagsource: "default/test"
    spec:
      containers:
        - name: test
          image: curlimages/curl:latest
          command:
            - /bin/sh
            - -c
            - |
              trap 'echo "SIGTERM received, sleeping 5s..."; sleep 5; exit 0' TERM

              while true; do
                curl -sf -w "\n" \
                  -X POST \
                  -H "Content-Type: application/json" \
                  -d '{"flagKey": "test-flag", "context": {}}' \
                  "http://${FLAGD_HOST}:${FLAGD_PORT}/schema.v1.Service/ResolveBoolean" \
                  || echo "FLAG EVALUATION FAILED (exit $?)"
                sleep 1
              done &
              wait
          env:
            - name: FLAGD_HOST
              value: "localhost"
  1. Main container errors evaluating flags on pod startup as the Flagd container is not ready yet

This scenario is rare as majority of workloads are going to take longer to startup than Flagd.

FLAG EVALUATION FAILED (exit 7)

FLAG EVALUATION FAILED (exit 7)
{"level":"info","ts":"2026-03-09T05:04:47.137Z","caller":"cmd/start.go:133","msg":"flagd version: v0.12.4 (cb2b8eeb9c5496272b1f22d63f8eaa5d220707f5), built at: 2025-06-02","component":"start"}
{"level":"info","ts":"2026-03-09T05:04:47.138Z","caller":"flag-sync/sync_service.go:94","msg":"starting flag sync service on port 8015","component":"FlagSyncService"}
{"level":"info","ts":"2026-03-09T05:04:47.139Z","caller":"kubernetes/kubernetes_sync.go:90","msg":"starting kubernetes sync notifier for resource: default/test","component":"sync","sync":"kubernetes"}
{"level":"info","ts":"2026-03-09T05:04:47.140Z","caller":"flag-evaluation/connect_service.go:229","msg":"Flag IResolver listening at [::]:8013","component":"service"}
{"level":"info","ts":"2026-03-09T05:04:47.140Z","caller":"flag-evaluation/connect_service.go:249","msg":"metrics and probes listening at 8014","component":"service"}
{"level":"info","ts":"2026-03-09T05:04:47.140Z","caller":"ofrep/ofrep_service.go:58","msg":"ofrep service listening at 8016","component":"OFREPService"}
{"level":"info","ts":"2026-03-09T05:04:47.147Z","caller":"kubernetes/kubernetes_sync.go:204","msg":"kube sync notifier event: add: default test","component":"sync","sync":"kubernetes"}
{"value":true, "reason":"STATIC", "variant":"on", "metadata":{}}
{"value":true, "reason":"STATIC", "variant":"on", "metadata":{}}
...
  1. Terminate pod and notice Flagd container terminates instantly, causing evaluation errors in the main container

This scenario is common for workloads that require some time to shutdown gracefully.

...
{"value":true, "reason":"STATIC", "variant":"on", "metadata":{}}
{"value":true, "reason":"STATIC", "variant":"on", "metadata":{}}
{"level":"info","ts":"2026-03-09T05:04:56.934Z","caller":"ofrep/ofrep_service.go:69","msg":"shutting down ofrep service","component":"OFREPService"}
SIGTERM received, sleeping 5s...
{"level":"info","ts":"2026-03-09T05:04:56.934Z","caller":"flag-sync/sync_service.go:163","msg":"shutting down gRPC sync service","component":"FlagSyncService"}
{"level":"info","ts":"2026-03-09T05:04:56.934Z","caller":"runtime/runtime.go:91","msg":"Shutting down server...","component":"runtime"}
{"level":"info","ts":"2026-03-09T05:04:56.934Z","caller":"runtime/runtime.go:93","msg":"Server successfully shutdown.","component":"runtime"}

FLAG EVALUATION FAILED (exit 7)

FLAG EVALUATION FAILED (exit 7)

FLAG EVALUATION FAILED (exit 7)

Expected Behaviour

Flagd is injected as a native Kubernetes sidecar to guarantee startup/shutdown ordering.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions