bug: AIC broken until restart after failover test

[martin-schulze-e2m]

Current Behavior

We did a K8s failover test where some nodes of the cluster were shut down to simulate a partial outage and some (but not all) apisix routes did not function afterwards.
Further investigation turned up error messages in the ingress-controller logs and after restarting the pod, the routes started working again.

Expected Behavior

After the system has settled, the ingress-controllelr should be able to resume normal operation.

Error Logs

These are the logs filtered for "error", newest lines on top (the failover test started around 15:05 UTC):

# the next 3 lines repeated each minute until restart
2026-03-19T15:15:57.300Z	DEBUG	provider	apisix/provider.go:306	handled ADC execution errors	{"status_record": null, "status_update": {}}
2026-03-19T15:15:57.300Z	INFO	provider.client	client/client.go:182	no GatewayProxy configs provided
2026-03-19T15:15:57.300Z	INFO	provider.client	client/client.go:177	syncing all resources
# repeated in bursts until restart
2026-03-19T15:12:57.575Z	ERROR	controllers.GatewayProxy	controller/utils.go:1278	failed to list resource	{"error": "Index with name field:serviceRefs does not exist"}
# next two lines repeated until restart
2026-03-19T15:12:57.517Z	ERROR	controller-runtime	controller/controller.go:347	Reconciler error	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "GatewayProxy": {"name":"apisix-config","namespace":"apisix"}, "namespace": "apisix", "name": "apisix-config", "reconcileID": "d3b4ea4f-c502-42b2-8b75-da07b9a7ab62", "error": "Index with name field:ingressClassParametersRef does not exist"}
2026-03-19T15:12:57.517Z	ERROR	controllers.GatewayProxy	controller/gatewayproxy_controller.go:172	failed to list IngressClassList	{"error": "Index with name field:ingressClassParametersRef does not exist"}
# repeated 10s of times per millisecond until 2026-03-19T15:12:57.518Z
2026-03-19T15:12:57.509Z	ERROR	controllers.GatewayProxy	controller/utils.go:1278	failed to list resource	{"error": "Index with name field:secretRefs does not exist"}
# repeated 10s of times per millisecond, note the different field name (sevrice vs secret)
2026-03-19T15:12:57.410Z	ERROR	controllers.GatewayProxy	controller/utils.go:1278	failed to list resource	{"error": "Index with name field:serviceRefs does not exist"}
# this line is from the apisix pod instead of inngress-controller, repeated until 15:12:53
2026/03/19 15:12:33 [error] 51#51: *257113185 recv() failed (111: Connection refused), context: ngx.timer, client: 10.62.14.1, server: 0.0.0.0:9080
# repeated 7x until 15:12:20.830650
E0319 15:11:52.704232       1 leaderelection.go:436] error retrieving resource lock apisix/apisix-ingress-controller-leader: Get "https://10.63.0.1:443/apis/coordination.k8s.io/v1/namespaces/apisix/leases/apisix-ingress-controller-leader?timeout=10s": dial tcp 10.63.0.1:443: connect: connection refused
2026-03-19T15:10:41.225Z	INFO	setup	manager/run.go:283	failed to get Kubernetes server version	{"error": "Get \"https://10.63.0.1:443/version?timeout=32s\": dial tcp 10.63.0.1:443: connect: no route to host"}
# message repeated many times for different kinds
2026-03-19T15:09:42.798Z	INFO	controller-runtime.api-detection	utils/k8s.go:65	group/version not available in cluster	{"kind": "Ingress", "group": "networking.k8s.io", "version": "v1", "groupVersion": "networking.k8s.io/v1", "error": "Get \"https://10.63.0.1:443/apis/networking.k8s.io/v1?timeout=32s\": dial tcp 10.63.0.1:443: connect: no route to host"}
2026-03-19T15:09:39.725Z	INFO	setup	manager/run.go:283	failed to get Kubernetes server version	{"error": "Get \"https://10.63.0.1:443/version?timeout=32s\": dial tcp 10.63.0.1:443: connect: no route to host"}
2026-03-19T15:09:28.772Z	INFO	controller-runtime.api-detection	utils/k8s.go:65	group/version not available in cluster	{"kind": "ApisixRoute", "group": "apisix.apache.org", "version": "v2", "groupVersion": "apisix.apache.org/v2", "error": "Get \"https://10.63.0.1:443/apis/apisix.apache.org/v2?timeout=32s\": context deadline exceeded - error from a previous attempt: http2: client connection lost"}
2026-03-19T15:08:56.770Z	INFO	controller-runtime.api-detection	utils/k8s.go:65	group/version not available in cluster	{"kind": "Ingress", "group": "networking.k8s.io", "version": "v1", "groupVersion": "networking.k8s.io/v1", "error": "Get \"https://10.63.0.1:443/apis/networking.k8s.io/v1?timeout=32s\": context deadline exceeded"}
Error: leader election lost
Error: leader election lost
E0319 15:08:24.254648       1 leaderelection.go:436] error retrieving resource lock apisix/apisix-ingress-controller-leader: Get "https://10.63.0.1:443/apis/coordination.k8s.io/v1/namespaces/apisix/leases/apisix-ingress-controller-leader?timeout=10s": context deadline exceeded
2026-03-19T15:07:32.132Z	DEBUG	provider	apisix/provider.go:306	handled ADC execution errors
2026-03-19T15:06:33.149Z	DEBUG	provider	apisix/provider.go:306	handled ADC execution errors	{"status_record": {}, "status_update": {}}
2026-03-19T15:06:32.131Z	ERROR	provider	apisix/provider.go:282	failed to sync	{"error": "failed to sync 1 configs: GatewayProxy/apisix/apisix-config"}
2026-03-19T15:06:32.131Z	DEBUG	provider	apisix/provider.go:306	handled ADC execution errors	{"status_record": {"GatewayProxy/apisix/apisix-config":{"Errors":[{"Name":"GatewayProxy/apisix/apisix-config","FailedErrors":[{"Err":"socket hang up","ServerAddr":"http://10.62.17.169:9180","FailedStatuses":[{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-03-19T15:06:32.13Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}}]}]}]}}, "status_update": {"ApisixGlobalRule/apisix/opentelemetry":["ServerAddr: http://10.62.17.169:9180, Err: socket hang up"], ... <redacted many more>}}
2026-03-19T15:06:32.131Z	ERROR	provider.client	client/client.go:210	failed to sync resources	{"name": "GatewayProxy/apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/apisix/apisix-config: [ServerAddr: http://10.62.17.169:9180, Err: socket hang up]]"}
2026-03-19T15:06:32.131Z	ERROR	provider.client	client/client.go:269	failed to execute adc command	{"config": {"name":"GatewayProxy/apisix/apisix-config","serverAddrs":["http://10.62.17.169:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/apisix/apisix-config: [ServerAddr: http://10.62.17.169:9180, Err: socket hang up]"}
2026-03-19T15:06:32.131Z	ERROR	provider.executor	client/executor.go:142	failed to run http sync for server	{"server": "http://10.62.17.169:9180", "error": "ServerAddr: http://10.62.17.169:9180, Err: socket hang up"}
2026-03-19T15:06:32.131Z	ERROR	provider.executor	client/executor.go:328	ADC Server sync failed	{"result": {"status":"all_failed","total_resources":1,"success_count":0,"failed_count":1,"success":[],"failed":[{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-03-19T15:06:32.13Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket hang up","response":{"status":0,"headers":null}}]}, "error": "ADC Server sync failed: socket hang up"}
2026-03-19T15:05:32.132Z	DEBUG	provider	apisix/provider.go:306	handled ADC execution errors	{"status_record": {}, "status_update": {}}

Steps to Reproduce

We don't know yet what exactly triggered this. This is our best guess so far:

1. install apisix helm chart on k8s
2. shut down some cluster nodes (including some control plane nodes?)
3. ???

Environment

• APISIX Ingress controller version (run apisix-ingress-controller version --long): apache/apisix-ingress-controller:2.0.1 (docker image)
• Kubernetes cluster version (run kubectl version): v1.29.15

[Baoyuantop]

Thanks for the detailed logs. From the log timeline, the failure window shows kube-apiserver connection failures, leader election loss, and API discovery failures, followed by persistent Index with name field:XXX does not exist errors.

Based on the code, AIC checks API resource availability via discovery during startup to decide whether to register field indexes. If the API server is unreachable at that point, index registration gets skipped. Once the cluster recovers, the indexes are never retroactively registered, causing continuous Reconcile failures until restart.

To help narrow down the root cause further, could you provide:

1. Whether the ingress-controller pod was recreated during/after failover (and if so, whether the API server was still unstable at that time)
2. Full startup logs showing api-detection / Skipping indexer setup lines
3. Whether any CRD/APIService availability changed during failover
4. A minimal reproduction procedure if possible

[martin-schulze-e2m]

Sorry for the late response. I had another look at our logs but I am afraid they are rather patchy due to the failover as well. This is the output of a kubectl get events --follow job around the time of the failover:

2026-03-19 15:47:59.978	apisix            35m                    Normal    LeaderElection           Lease/apisix-ingress-controller-leader           apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader
2026-03-19 15:47:59.977	apisix            38m                    Warning   Unhealthy                Pod/apisix-ingress-controller-6788f78f4c-gd95r   Readiness probe failed: Get "http://10.62.17.29:8081/readyz": read tcp 10.62.17.1:52512->10.62.17.29:8081: read: connection reset by peer
2026-03-19 15:47:59.977	apisix            38m                    Normal    Killing                  Pod/apisix-ingress-controller-6788f78f4c-gd95r   Container manager failed liveness probe, will be restarted
2026-03-19 15:47:59.976	apisix            38m (x4 over 39m)      Warning   Unhealthy                Pod/apisix-ingress-controller-6788f78f4c-gd95r   Liveness probe failed: Get "http://10.62.17.29:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2026-03-19 15:47:59.976	apisix            37m (x13 over 39m)     Warning   Unhealthy                Pod/apisix-ingress-controller-6788f78f4c-gd95r   Readiness probe failed: Get "http://10.62.17.29:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2026-03-19 15:47:59.975	apisix            38m (x2 over 39m)      Normal    Started                  Pod/apisix-ingress-controller-6788f78f4c-gd95r   Started container manager
2026-03-19 15:47:59.974	apisix            38m (x2 over 39m)      Normal    Created                  Pod/apisix-ingress-controller-6788f78f4c-gd95r   Created container: manager
2026-03-19 15:47:59.974	apisix            38m (x2 over 39m)      Normal    Pulled                   Pod/apisix-ingress-controller-6788f78f4c-gd95r   Container image "apache/apisix-ingress-controller:2.0.1" already present on machine
2026-03-19 15:47:59.973	apisix            35m                    Normal    EnsuredLoadBalancer      Service/apisix-gateway                        Ensured load balancer
2026-03-19 15:47:59.972	apisix            35m                    Normal    EnsuringLoadBalancer     Service/apisix-gateway                        Ensuring load balancer
2026-03-19 15:13:15.845	apisix            18s                      Normal    LeaderElection           Lease/apisix-ingress-controller-leader           apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader
2026-03-19 15:13:15.844	apisix            3m38s                    Warning   Unhealthy                Pod/apisix-ingress-controller-6788f78f4c-gd95r   Readiness probe failed: Get "http://10.62.17.29:8081/readyz": read tcp 10.62.17.1:52512->10.62.17.29:8081: read: connection reset by peer
2026-03-19 15:13:15.843	apisix            3m38s                    Normal    Killing                  Pod/apisix-ingress-controller-6788f78f4c-gd95r   Container manager failed liveness probe, will be restarted
2026-03-19 15:13:15.842	apisix            3m18s (x4 over 4m18s)    Warning   Unhealthy                Pod/apisix-ingress-controller-6788f78f4c-gd95r   Liveness probe failed: Get "http://10.62.17.29:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2026-03-19 15:13:15.841	apisix            2m58s (x13 over 4m38s)   Warning   Unhealthy                Pod/apisix-ingress-controller-6788f78f4c-gd95r   Readiness probe failed: Get "http://10.62.17.29:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2026-03-19 15:13:15.840	apisix            3m37s (x2 over 4m51s)   Normal    Started                  Pod/apisix-ingress-controller-6788f78f4c-gd95r   Started container manager
2026-03-19 15:13:15.821	apisix            3m37s (x2 over 4m51s)   Normal    Created                  Pod/apisix-ingress-controller-6788f78f4c-gd95r   Created container: manager
2026-03-19 15:13:15.789	apisix            3m37s (x2 over 4m51s)   Normal    Pulled                   Pod/apisix-ingress-controller-6788f78f4c-gd95r   Container image "apache/apisix-ingress-controller:2.0.1" already present on machine
2026-03-19 15:13:15.789	apisix            23s                Normal    EnsuredLoadBalancer      Service/apisix-gateway                Ensured load balancer
2026-03-19 15:13:15.788	apisix            23s                Normal    EnsuringLoadBalancer     Service/apisix-gateway                Ensuring load balancer

Unfortunately, it seems the events on CRDs are not displayed (or there were none).

So to answer your questions:

1. our kube_pod_container_status_restarts_total metric says pod apisix-ingress-controller-6788f78f4c-gd95 restarted once at 15:13 I can't say for sure if the API was down at this time but I'd assume as much.
2. here are more logs from the ingress controller, this time excluding error messages:

2026-03-19T15:18:25.204Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:17:57.300Z	INFO	provider.client	client/client.go:182	no GatewayProxy configs provided
2026-03-19T15:17:57.300Z	INFO	provider.client	client/client.go:177	syncing all resources
2026-03-19T15:16:57.301Z	INFO	provider.client	client/client.go:182	no GatewayProxy configs provided
2026-03-19T15:16:57.300Z	INFO	provider.client	client/client.go:177	syncing all resources
2026-03-19T15:15:57.300Z	INFO	provider.client	client/client.go:182	no GatewayProxy configs provided
2026-03-19T15:15:57.300Z	INFO	provider.client	client/client.go:177	syncing all resources
2026-03-19T15:15:41.364Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:14:57.301Z	INFO	provider.client	client/client.go:182	no GatewayProxy configs provided
2026-03-19T15:14:57.301Z	INFO	provider.client	client/client.go:177	syncing all resources
2026-03-19T15:14:19.442Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:13:57.301Z	INFO	provider.client	client/client.go:182	no GatewayProxy configs provided
2026-03-19T15:13:57.301Z	INFO	provider.client	client/client.go:177	syncing all resources
2026-03-19T15:13:38.481Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:13:18.000Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:13:07.759Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:13:02.638Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:13:00.077Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:58.796Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:58.156Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.835Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.674Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.594Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.553Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.532Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.522Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.517Z	DEBUG	controllers.GatewayProxy	controller/utils.go:1473	to process provider endpoints by provider.service	{"service": {"name":"apisix-admin","namespace":"apisix"}}
2026-03-19T15:12:57.516Z	INFO	controller-runtime	controller/controller.go:248	Starting workers	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "worker count": 1}
2026-03-19T15:12:57.516Z	INFO	controller-runtime	controller/controller.go:239	Starting Controller	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy"}
2026-03-19T15:12:57.300Z	INFO	controller-runtime	controller/controller.go:204	Starting EventSource	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1alpha1.GatewayProxy"}
2026-03-19T15:12:57.300Z	INFO	controller-runtime	controller/controller.go:204	Starting EventSource	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.Secret"}
2026-03-19T15:12:57.300Z	INFO	controller-runtime	controller/controller.go:204	Starting EventSource	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.Service"}
2026-03-19T15:12:57.300Z	INFO	controller-runtime	controller/controller.go:204	Starting EventSource	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.EndpointSlice"}
2026-03-19T15:12:57.300Z	INFO	controller-runtime	controller/controller.go:204	Starting EventSource	{"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.IngressClass"}
2026-03-19T15:12:57.300Z	INFO	status.updater	status/updater.go:131	started status update handler
2026-03-19T15:12:57.300Z	INFO	provider	apisix/provider.go:256	Ready detected, starting sync loop
2026-03-19T15:12:57.300Z	INFO	provider	apisix/provider.go:254	starting provider, waiting for readiness
2026-03-19T15:12:57.300Z	INFO	readiness	readiness/manager.go:139	readiness manager started
2026-03-19T15:12:57.300Z	DEBUG	controller-runtime.events	recorder/recorder.go:104	apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"apisix","name":"apisix-ingress-controller-leader","uid":"769a49a6-01d2-42c5-abcf-64a92892c597","apiVersion":"coordination.k8s.io/v1","resourceVersion":"247335807"}, "reason": "LeaderElection"}
I0319 15:12:57.300009       1 leaderelection.go:271] successfully acquired lease apisix/apisix-ingress-controller-leader
I0319 15:11:52.703743       1 leaderelection.go:257] attempting to acquire leader lease apisix/apisix-ingress-controller-leader...
2026-03-19T15:11:52.703Z	INFO	controller-runtime.controller-runtime.certwatcher	certwatcher/certwatcher.go:133	Starting certificate poll+watcher	{"interval": "10s"}
2026-03-19T15:11:52.703Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:242	Serving webhook server	{"host": "", "port": 9443}
2026-03-19T15:11:52.703Z	INFO	controller-runtime.controller-runtime.certwatcher	certwatcher/certwatcher.go:211	Updated current TLS certificate
2026-03-19T15:11:52.703Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:191	Starting webhook server
2026-03-19T15:11:52.703Z	INFO	controller-runtime.controller-runtime.metrics	server/server.go:247	Serving metrics server	{"bindAddress": ":8080", "secure": false}
2026-03-19T15:11:52.703Z	INFO	controller-runtime	manager/server.go:83	starting server	{"name": "health probe", "addr": "[::]:8081"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.metrics	server/server.go:208	Starting metrics server
2026-03-19T15:11:52.702Z	INFO	setup	manager/run.go:264	starting controller manager
2026-03-19T15:11:52.702Z	INFO	setup	manager/run.go:258	setting up ready checks
2026-03-19T15:11:52.702Z	INFO	setup	manager/run.go:252	setting up health checks
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-apisix-apache-org-v1alpha1-consumer"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "apisix.apache.org/v1alpha1, Kind=Consumer", "path": "/validate-apisix-apache-org-v1alpha1-consumer"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-apisix-apache-org-v2-apisixroute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "apisix.apache.org/v2, Kind=ApisixRoute", "path": "/validate-apisix-apache-org-v2-apisixroute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-apisix-apache-org-v2-apisixtls"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "apisix.apache.org/v2, Kind=ApisixTls", "path": "/validate-apisix-apache-org-v2-apisixtls"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-apisix-apache-org-v2-apisixconsumer"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "apisix.apache.org/v2, Kind=ApisixConsumer", "path": "/validate-apisix-apache-org-v2-apisixconsumer"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-gateway-networking-k8s-io-v1alpha2-udproute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute", "path": "/validate-gateway-networking-k8s-io-v1alpha2-udproute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-gateway-networking-k8s-io-v1alpha2-tcproute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute", "path": "/validate-gateway-networking-k8s-io-v1alpha2-tcproute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-gateway-networking-k8s-io-v1-grpcroute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "gateway.networking.k8s.io/v1, Kind=GRPCRoute", "path": "/validate-gateway-networking-k8s-io-v1-grpcroute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-gateway-networking-k8s-io-v1-httproute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "gateway.networking.k8s.io/v1, Kind=HTTPRoute", "path": "/validate-gateway-networking-k8s-io-v1-httproute"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-apisix-apache-org-v1alpha1-gatewayproxy"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "apisix.apache.org/v1alpha1, Kind=GatewayProxy", "path": "/validate-apisix-apache-org-v1alpha1-gatewayproxy"}
2026-03-19T15:11:52.702Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-gateway-networking-k8s-io-v1-gateway"}
2026-03-19T15:11:52.701Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "gateway.networking.k8s.io/v1, Kind=Gateway", "path": "/validate-gateway-networking-k8s-io-v1-gateway"}
2026-03-19T15:11:52.701Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-networking-k8s-io-v1-ingressclass"}
2026-03-19T15:11:52.701Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "networking.k8s.io/v1, Kind=IngressClass", "path": "/validate-networking-k8s-io-v1-ingressclass"}
2026-03-19T15:11:52.701Z	INFO	controller-runtime.controller-runtime.webhook	webhook/server.go:183	Registering webhook	{"path": "/validate-networking-k8s-io-v1-ingress"}
2026-03-19T15:11:52.700Z	INFO	controller-runtime.controller-runtime.builder	builder/webhook.go:226	Registering a validating webhook	{"GVK": "networking.k8s.io/v1, Kind=Ingress", "path": "/validate-networking-k8s-io-v1-ingress"}
2026-03-19T15:11:52.700Z	INFO	setup	manager/run.go:243	setting up webhooks
2026-03-19T15:11:52.700Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixGlobalRule"}
2026-03-19T15:11:52.699Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "networking.k8s.io/v1, Kind=Ingress"}
2026-03-19T15:11:52.699Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "networking.k8s.io/v1, Kind=IngressClass"}
2026-03-19T15:11:52.699Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixUpstream"}
2026-03-19T15:11:52.698Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixTls"}
2026-03-19T15:11:52.698Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixPluginConfig"}
2026-03-19T15:11:52.697Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixConsumer"}
2026-03-19T15:11:52.697Z	INFO	setup.apiv2	manager/controllers.go:291	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixRoute"}
2026-03-19T15:11:52.695Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixPluginConfig"}
2026-03-19T15:11:52.695Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixRoute"}
2026-03-19T15:11:52.694Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixConsumer"}
2026-03-19T15:11:52.694Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "networking.k8s.io/v1, Kind=Ingress"}
2026-03-19T15:11:52.694Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "networking.k8s.io/v1, Kind=IngressClass"}
2026-03-19T15:11:52.693Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixGlobalRule"}
2026-03-19T15:11:52.693Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixTls"}
2026-03-19T15:11:52.692Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"}
2026-03-19T15:11:52.692Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:11:52.692Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:11:52.691Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:11:52.691Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:11:52.690Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=Gateway"}
2026-03-19T15:11:52.690Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"}
2026-03-19T15:11:52.689Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=Consumer"}
2026-03-19T15:11:52.689Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:11:52.688Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:11:52.688Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:11:52.687Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:11:52.687Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=Gateway"}
2026-03-19T15:11:52.686Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"}
2026-03-19T15:11:52.685Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"}
2026-03-19T15:11:52.685Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=Consumer"}
2026-03-19T15:11:52.684Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=BackendTrafficPolicy"}
2026-03-19T15:11:52.684Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=GatewayProxy"}
2026-03-19T15:11:52.683Z	INFO	setup	manager/run.go:227	setting up controllers
2026-03-19T15:11:52.683Z	INFO	setup	manager/run.go:223	CRD ReferenceGrants is not installed	{"err": "failed to get server groups: Get \"https://10.63.0.1:443/api\": dial tcp 10.63.0.1:443: connect: connection refused"}
2026-03-19T15:11:52.683Z	INFO	setup	manager/run.go:216	check ReferenceGrants is enabled
2026-03-19T15:11:52.683Z	INFO	provider.client	client/client.go:63	ADC client initialized
2026-03-19T15:11:52.683Z	INFO	controller-runtime.readiness	manager/controllers.go:384	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v1alpha1, Kind=Consumer"}
2026-03-19T15:11:52.682Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"}
2026-03-19T15:11:52.681Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:11:51.661Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:11:48.585Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:11:45.515Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:11:45.514Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixUpstream"}
2026-03-19T15:11:45.514Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixConsumer"}
2026-03-19T15:11:45.513Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixTls"}
2026-03-19T15:11:42.441Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixPluginConfig"}
2026-03-19T15:11:39.370Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixGlobalRule"}
2026-03-19T15:11:39.370Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixRoute"}
2026-03-19T15:11:39.369Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "networking.k8s.io/v1, Kind=Ingress"}
2026-03-19T15:11:38.173Z	INFO	root/root.go:125	controller start configuration	{"config": {"log_level":"debug","controller_name":"apisix.apache.org/apisix-ingress-controller","leader_election_id":"apisix-ingress-controller-leader","metrics_addr":":8080","server_addr":":9092","enable_server":false,"enable_http2":false,"probe_addr":":8081","secure_metrics":false,"leader_election":{"lease_duration":"30s","renew_deadline":"20s","retry_period":"2s"},"exec_adc_timeout":"15s","provider":{"type":"apisix-standalone","sync_period":"1m0s","init_sync_delay":"20m0s"},"webhook":{"enable":true,"tls_cert_file":"tls.crt","tls_key_file":"tls.key","tls_cert_dir":"/certs","port":9443},"disable_gateway_api":false}}
2026-03-19T15:11:36.299Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixTls"}
2026-03-19T15:11:36.298Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixPluginConfig"}
2026-03-19T15:11:36.298Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixRoute"}
2026-03-19T15:11:36.297Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v2, Kind=ApisixConsumer"}
2026-03-19T15:11:35.279Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "networking.k8s.io/v1, Kind=Ingress"}
2026-03-19T15:11:35.278Z	INFO	controller-runtime.indexer.apiv2	indexer/indexer.go:93	Skipping indexer setup, API not found in cluster	{"api": "networking.k8s.io/v1, Kind=IngressClass"}
2026-03-19T15:11:35.278Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"}
2026-03-19T15:11:35.277Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=Consumer"}
2026-03-19T15:11:33.227Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"}
2026-03-19T15:11:33.226Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:11:33.225Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:11:32.909Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:11:29.833Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:11:26.761Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=Gateway"}
2026-03-19T15:11:24.841Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"}
2026-03-19T15:11:21.769Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:11:18.698Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:11:18.697Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:11:15.626Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:11:15.625Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=Gateway"}
2026-03-19T15:11:12.553Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"}
2026-03-19T15:11:09.485Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=GatewayProxy"}
2026-03-19T15:11:06.411Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=Consumer"}
2026-03-19T15:11:06.410Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=BackendTrafficPolicy"}
2026-03-19T15:11:06.409Z	INFO	setup	manager/run.go:227	setting up controllers
2026-03-19T15:11:06.409Z	INFO	setup	manager/run.go:223	CRD ReferenceGrants is not installed	{"err": "failed to get server groups: Get \"https://10.63.0.1:443/api\": dial tcp 10.63.0.1:443: connect: no route to host"}
2026-03-19T15:11:04.745Z	INFO	setup	manager/run.go:216	check ReferenceGrants is enabled
2026-03-19T15:11:04.745Z	INFO	provider.client	client/client.go:63	ADC client initialized
2026-03-19T15:11:04.745Z	INFO	controller-runtime.readiness	manager/controllers.go:384	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v1alpha1, Kind=Consumer"}
2026-03-19T15:11:03.337Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"}
2026-03-19T15:11:01.673Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:10:59.785Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:10:58.601Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:10:57.097Z	INFO	controller-runtime.readiness	manager/controllers.go:324	Skipping readiness registration, API not found	{"gvk": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:10:55.113Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixUpstream"}
2026-03-19T15:10:52.041Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixConsumer"}
2026-03-19T15:10:50.729Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixTls"}
2026-03-19T15:10:48.777Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixPluginConfig"}
2026-03-19T15:10:47.657Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixGlobalRule"}
2026-03-19T15:10:44.585Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "apisix.apache.org/v2, Kind=ApisixRoute"}
2026-03-19T15:10:42.253Z	INFO	controller-runtime.readiness	manager/controllers.go:353	Skipping readiness registration, API not found	{"gvk": "networking.k8s.io/v1, Kind=Ingress"}
2026-03-19T15:10:38.185Z	INFO	root/root.go:125	controller start configuration	{"config": {"log_level":"debug","controller_name":"apisix.apache.org/apisix-ingress-controller","leader_election_id":"apisix-ingress-controller-leader","metrics_addr":":8080","server_addr":":9092","enable_server":false,"enable_http2":false,"probe_addr":":8081","secure_metrics":false,"leader_election":{"lease_duration":"30s","renew_deadline":"20s","retry_period":"2s"},"exec_adc_timeout":"15s","provider":{"type":"apisix-standalone","sync_period":"1m0s","init_sync_delay":"20m0s"},"webhook":{"enable":true,"tls_cert_file":"tls.crt","tls_key_file":"tls.key","tls_cert_dir":"/certs","port":9443},"disable_gateway_api":false}}
2026-03-19T15:10:35.369Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:10:32.297Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:10:31.753Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:10:29.229Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:10:29.101Z	INFO	setup.gatewayapi	manager/controllers.go:214	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=Gateway"}
2026-03-19T15:10:26.729Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"}
2026-03-19T15:10:23.661Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"}
2026-03-19T15:10:22.957Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"}
2026-03-19T15:10:20.585Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"}
2026-03-19T15:10:17.513Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=Gateway"}
2026-03-19T15:10:16.745Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"}
2026-03-19T15:10:13.673Z	INFO	controller-runtime.indexer.gatewayapi	indexer/indexer.go:116	Skipping indexer setup, API not found in cluster	{"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"}
2026-03-19T15:10:11.369Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=GatewayProxy"}
2026-03-19T15:10:10.541Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=Consumer"}
2026-03-19T15:10:07.469Z	INFO	controller-runtime.indexer.apiv1alpha1	indexer/indexer.go:71	Skipping indexer setup, API not found in cluster	{"api": "apisix.apache.org/v1alpha1, Kind=BackendTrafficPolicy"}

3. I am not sure how to check that. We had several services behind ApisixRoutes that restarted due to the failover.
4. I don't think this will be possible unless we plan to do another failover test.

[Baoyuantop]

Thank you for providing the information. We will conduct further investigation.