Continuous repeated CIDRNotAvailable event for one node in cluster.

[tjf]

Environmental Info:
K3s Version:

k3s version v1.26.4+k3s1 (8d0255af)
go version go1.19.8

Node(s) CPU architecture, OS, and Version:

Linux node1 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

Cluster Configuration:

Total of 3 servers using the embedded etcd. I am using Calico 3.25 instead of Flannel.

Describe the bug:

While the node is working fine -- all network functionality, pod scheduling, service routing, etc -- it is constantly logging CIDRNotAvailable events. I can't find any mention in the k3s logs of any network issues that could be causing this. If I didn't see this in the kubectl describe node output, I'd have no idea there was a problem, honestly.

Events:
  Type    Reason            Age                      From           Message
  ----    ------            ----                     ----           -------
  Normal  CIDRNotAvailable  107s (x1025 over 3d14h)  cidrAllocator  Node node1 status is now: CIDRNotAvailable

Steps To Reproduce:

Not sure, but if someone could point me in the direction of troubleshooting further, I'd appreciate it.

Expected behavior:

I expect this event not to be logged :-)

Additional info:
Here's the systemd unit file ExecStart entry, it's the same on all the nodes:

ExecStart=/usr/local/bin/k3s \
    server \
	'--disable-network-policy' \
	'--flannel-backend=none' \
	'--token=<redacted>' \
	'--disable=servicelb' \
	'--cluster-cidr=192.168.20.0/23' \
	'--cluster-dns=172.20.0.10' \
	'--service-cidr=172.20.0.0/24' \
	'--server' \
	'https://192.168.45.52:6443' \

Also -- I opened an issue with Project Calico, but they don't think this is being caused by the calico software. See projectcalico/calico#7722

Thanks :-)

[brandond]

	'--cluster-cidr=192.168.20.0/23' \
	'--cluster-dns=172.20.0.10' \
	'--service-cidr=172.20.0.0/24' \

Why the tiny CIDRs? The default node CIDR mask is /24, so with a /23 cluster-cidr you only have enough addresses for two nodes in your cluster. You've also only allocated yourself enough space for 255 ClusterIP services.

You should rebuild your cluster with more reasonable CIDR blocks.

[tjf]

Just adding another comment in the event you missed my edit above.

[brandond]

The k3s stub ccm doesn't handle node cidrs, so I think you should be ok.

[tjf]

Adding --kube-cloud-controller-manager-arg=allocate-node-cidrs=false to the command line seems to have squashed it!

Update: Bah, no it didn't. I'll dig into the code and see if I can hunt this down.

[BhautikChudasama]

@tjf how did you fixed this issue?

[BhautikChudasama]

@tjf This is the part of --kube-controller-manager-arg not --kube-cloud-controller-manager-arg
Full example

--kube-controller-manager-arg=allocate-node-cidrs=false

[tjf]

@BhautikChudasama thanks for replying! I never did end up fixing this as it was just on a personal cluster and it didn't matter THAT much. But, seeing how I messed up the args, I may give this another shot. Good job!