Calico drop udp coredns response in 3.31.3

[hengqiali]

Recently, during an upgrade from Calico 3.28.1 to Calico 3.31.3, I encountered a Calico bug introduced by #8934 and confidential-containers/cloud-api-adaptor#2015 .

Calico added a new feature in version 3.29 (refer to above link) that disables connection tracking for UDP flows. Additionally, since earlier versions, there has been a default rule that drops any UDP packets with destination port 4789 originating from pods.

This leads to an issue: when a client node sends a UDP 53 request to CoreDNS, if the client's source port is SNATed by the client-side's vxlan.calico to 4789, then the response from CoreDNS is directly dropped on the CoreDNS node.

In the older version, UDP connection tracking was still in place, so this issue didn’t occur, that is, udp packets with dst 4789 from pods will not be dropped and instead is processed by udp conntrack successfully.

In the calico new version >3.29, this problem affects almost all UDP requests from pods and from nodes to pods with 4789 udp port in production.

Do you think what's the best way to fix the issue?

My think is

1. Remove the iptable rule that drop udp pkts with dst 4789 port
2. Revert the MR from the above link
3. Make calico vxlan SNAT the source port to any port but not 4789..

Thanks

[hengqiali]

We meet the same root cause again after applying allowVXLANPacketsFromWorkloads=true , issue  occurs on client node side when it receives the coredns reponse. Because such response packet's source ip is coredns ip, dst ip is node vxlan.calico ip, but calico only allow packets whose dst ip is node vxlan.calico with allowed-nodes ip list.. Then it drops the packets using following rules:

-> iptables-save | grep -i 4789
-A cali-INPUT -p udp -m comment --comment "cali:J76FwxInZIsk7uKY" -m comment --comment "Allow IPv4 VXLAN packets from allowed hosts" -m multiport --dports 4789 -m set --match-set cali40all-vxlan-net src -m addrtype --dst-type LOCAL -j ACCEPT
-A cali-INPUT -p udp -m comment --comment "cali:EDCNTTxYfggApx8C" -m comment --comment "Drop IPv4 VXLAN packets from non-allowed hosts" -m multiport --dports 4789 -m addrtype --dst-type LOCAL -j DROP
-A cali-OUTPUT -p udp -m comment --comment "cali:L6ph6wPpifwelglW" -m comment --comment "Allow IPv4 VXLAN packets to other allowed hosts" -m multiport --dports 4789 -m addrtype --src-type LOCAL -m set --match-set cali40all-vxlan-net dst -j ACCEPT
-A cali-pi-_jeTzeKlg2jMNl31LJr- -p udp -m comment --comment "cali:JhEJ0Yezq1gmGxT6" -m set --match-set cali40s:SFT7hgZgXfXtrTuyJkautS2 src -m multiport --dports 4789,51820 -j MARK --set-xmark 0x10000/0x10000

[caseydavenport]

discussed in Slack - fix is probably to adjust the VXLAN tunnel MASQ rule to use a port-range to avoid SNATing to the VXLAN port.

[hengqiali]

Thanks @caseydavenport for concern.
Expected to a quick fix, I think all versions > 3.29 need this fix!