networkPlugin cni failed to set up pod "svclb-traefik-xxx_kube-system" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container"xxxx"

[ruoyu1]

K3s agent node in respberryPi reports as follows:

(combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3d21e12828cba1bc8d090e94f3310e59e95a16689c51f951c9fdc612153024af" network for pod "svclb-traefik-642c96ae-b7ggt": networkPlugin cni failed to set up pod "svclb-traefik-642c96ae-b7ggt_kube-system" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "3d21e12828cba1bc8d090e94f3310e59e95a16689c51f951c9fdc612153024af"

K3s version: v1.27.16+k3s1
agent node OS:

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# cat /proc/version
Linux version 6.1.21-v8+ (dom@buildbot) (aarch64-linux-gnu-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023

other clues:
cni0 has not been created

$ ip -d addr show cni0
Device "cni0" does not exist.
$ ip -d addr show flannel.1
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 26:56:bd:b5:2e:28 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    vxlan id 1 local 192.168.14.100 dev wlan0 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 10.42.1.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet 169.254.154.242/16 brd 169.254.255.255 scope global noprefixroute flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::2456:bdff:feb5:2e28/64 scope link 
       valid_lft forever preferred_lft forever

what I have tried:

1. use iptables

sudo systemctl stop ntf & systemctl  disable ntf 
sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo reboot

2. create cni0 manually

ip link add cni0 type bridge
ip link set dev cni0 up
ip addr add 10.42.1.1/24 dev cni0
# ip -d addr show cni0
7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 3e:b5:70:b8:e1:5b brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    bridge forward_delay 1500 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 bridge_id 8000.0:0:0:0:0:0 designated_root 8000.0:0:0:0:0:0 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  186.00 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 10.42.1.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::3cb5:70ff:feb8:e15b/64 scope link 
       valid_lft forever preferred_lft forever

3. pull klipper-lb image manually....

expect someone to help me, i have been trapped 2 days~~~~

[brandond]

1. 1.27 is end of life. Please upgrade.
2. Don't try to create the CNI interfaces manually. These are managed by flannel.
3. Please check the docs for raspberry pi setup prerequisites. I suspect you are missing the vxlan kernel module.