pod stuck in UnexpectedAdmissionError state after host reboot

[gfrankliu]

When the host computer reboots, most of the pods in the k3s cluster comes up fine, except one of our GPU pods from a deployment. We most always observe the old pods stuck in UnexpectedAdmissionError state, and then replacement pods are starting in good Running state. Those "ghost" pods in UnexpectedAdmissionError state seems don't do anything and don't cause any harm. I can always manually kubectl delete them. Has anyone else seen the similar issue? Is there a way for k3s auto clean those pods?

[gfrankliu]

It seems when we do rolling restart the deployment ("kubectl rollout restart"), k3s left behind lots of old replicasets.

$ kubectl get deployment | grep triton
triton-server       8/8     8            8           108d
$ kubectl get rs | grep triton
triton-server-544f5758d4       0         0         0       6d19h
triton-server-56dc8d5d59       0         0         0       38h
triton-server-6456c67f8b       0         0         0       2d20h
triton-server-6679c965cf       0         0         0       2d20h
triton-server-685c976cb4       0         0         0       14d
triton-server-68877964d5       8         8         8       20h
triton-server-78b98fd66d       0         0         0       27d
triton-server-78bbdf6dc8       0         0         0       26d
triton-server-7b5dcc4c8f       0         0         0       2d20h
triton-server-7d7bd8fcc8       0         0         0       2d20h
triton-server-856cc54bb4       0         0         0       2d20h

The UnexpectedAdmissionError pods seem to come from the old replicasets:

$ kubectl get pod | grep triton
triton-server-68877964d5-267cw                 1/1     Running                    0              20h
triton-server-68877964d5-26btz                 1/1     Running                    0              20h
triton-server-68877964d5-2nspq                 1/1     Running                    0              20h
triton-server-68877964d5-5c2s7                 1/1     Running                    0              20h
triton-server-68877964d5-n4kbj                 1/1     Running                    0              20h
triton-server-68877964d5-pvlfd                 1/1     Running                    0              20h
triton-server-68877964d5-vm8zn                 1/1     Running                    0              20h
triton-server-68877964d5-zr8tk                 1/1     Running                    0              20h
triton-server-78bbdf6dc8-6qbk5                 0/1     UnexpectedAdmissionError   0              26d
triton-server-78bbdf6dc8-bcvwc                 0/1     UnexpectedAdmissionError   0              26d
triton-server-78bbdf6dc8-flfs5                 0/1     UnexpectedAdmissionError   0              26d
triton-server-78bbdf6dc8-mfczv                 0/1     UnexpectedAdmissionError   0              26d
triton-server-78bbdf6dc8-nb5hm                 0/1     UnexpectedAdmissionError   0              26d
triton-server-78bbdf6dc8-qj74h                 0/1     UnexpectedAdmissionError   0              26d
triton-server-78bbdf6dc8-rxnmk                 0/1     UnexpectedAdmissionError   0              26d
triton-server-78bbdf6dc8-sd2w8                 0/1     UnexpectedAdmissionError   0              26d

$ kubectl describe pod/triton-server-78bbdf6dc8-sd2w8 
...
Status:              Failed
Reason:              UnexpectedAdmissionError
Message:             Pod was rejected: Allocate failed due to no healthy devices present; cannot allocate unhealthy devices nvidia.com/mig-1g.6gb, which is unexpected
....

If I manually delete the old unused replicasets, those ghost pods will go away:

$ kubectl delete rs/triton-server-78bbdf6dc8
replicaset.apps "triton-server-78bbdf6dc8" deleted
$ kubectl get pod | grep triton
triton-server-68877964d5-267cw                 1/1     Running     0              20h
triton-server-68877964d5-26btz                 1/1     Running     0              20h
triton-server-68877964d5-2nspq                 1/1     Running     0              20h
triton-server-68877964d5-5c2s7                 1/1     Running     0              20h
triton-server-68877964d5-n4kbj                 1/1     Running     0              20h
triton-server-68877964d5-pvlfd                 1/1     Running     0              20h
triton-server-68877964d5-vm8zn                 1/1     Running     0              20h
triton-server-68877964d5-zr8tk                 1/1     Running     0              20h
$ 

Shouldn't k3s auto cleanup those old replicasets and pods? (that's why I initially opened this as a bug in k3s)