Upgrading from version 3.28.3 to any higher version results in "tls: failed to verify certificate: x509" when starting or restarting pods

[hannes-weizmann]

Expected Behavior

I am performing the upgrade following the guide "Upgrading an installation that uses manifests and the Kubernetes API datastore". calicoctl is not installed, and I am not using Istio. The Calico pods start successfully. After that, pods should start normally.

Current Behavior

Upgrade performed via curl and kubectl apply --server-side --force-conflicts -f upgrade.yaml. The Calico pods start successfully and are Running and Ready. Deploying a new pod or restarting an existing one results in the following error:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7793eab191474b0fc110efb60e94a220074e1d5ce174853eb1befecca05f1487": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate signed by unknown authority

Possible Solution

I have checked the ca.crt in the Calico pods and in the Calico kubeconfig. They match the fingerprint of /etc/kubernetes/pki/ca.crt.
I have tried upgrading to different Calico versions: 3.28.5, 3.29.x, 3.30.x
→ all of them lead to the same result.

Steps to Reproduce (for bugs)

1. Update Calico from v3.28.3
2. Restart a pod

Context

I want to upgrade my Calico. But any newer version results in the Error above. A downgrade to 3.28.3 fixes the issue.

Your Environment

• Calico version: 3.28.3
• Orchestrator version (e.g. kubernetes, openshift, etc.): kubelet-version: 1.34.4
• Operating System and version: Rocky Linux 9.7

[lucastigera]

Hi @hannes-weizmann! Thanks for raising this.

Could you confirm which dataplane you are using (eBPF or iptables)? Also, could you share your FelixConfiguration?

kubectl get felixconfig -o yaml

[hannes-weizmann]

Hi @lucastigera
We are using the iptables dataplane. bpfEnabled is not configured in the FelixConfiguration.

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
name: default
spec:
bpfConnectTimeLoadBalancing: TCP
bpfHostNetworkedNATWithoutCTLB: Enabled
bpfLogLevel: ""
floatingIPs: Disabled
logSeverityScreen: Info
reportingInterval: 0s

[mazdakn]

@hannes-weizmann usually this error is a manifestation of other issues, like bad certs handed by kubernetes, or timing issue.
Can you check if the time between API server and nodes are in sync? Also are the certs valid?

[hannes-weizmann]

@mazdakn Time appears to be in sync across all nodes. Certificates are also valid.
What i recently found out is, that the issue only occurs when i upgrade the image docker.io/calico/cni (install-cni).
Chaning the version of calico-node and calico-kube-controller works fine.

[caseydavenport]

Interesting. So I looked into this and the kubeconfig generation code in cni-plugin/pkg/install/install.go is actually identical between v3.28.3 and v3.28.5. The only change to cni-plugin/ between those versions is a Go/k8s dependency bump - Go 1.22 → 1.23.8 and client-go v0.28 → v0.29. Go 1.23 tightened x509 certificate verification in a few ways (release notes, GODEBUG reference) that could cause this depending on how your cluster's CA cert was generated.

Can you check openssl x509 -in /etc/kubernetes/pki/ca.crt -text -noout and look at the Signature Algorithm and Serial Number fields? If the serial is negative or the signature algorithm is SHA-1, that would explain the Go 1.23 breakage.

[hannes-weizmann]

@caseydavenport This is what I see in my ca.crt:

Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption

So the CA itself is not using SHA-1 and the serial is not negative.