Difference between tsh kube and tsh proxy kube

[Ezzahhh]

Not sure if it was appropriate to create an issue instead of a discussion: #49622

I'm observing an interesting difference when connecting through Teleport with tsh kube and tsh proxy kube when in conjunction with the k9s utility. Specifically, I get a lot of context deadline exceeded especially with the subjectaccessreviews call when going through tsh kube login and k9s often does not connect properly (even when increasing timeout). However, when using tsh proxy kube I do not have any such issue at all.

I see logs like this in my teleport-agent and nothing appears awry but k9s reports context deadline exceeded:

2024-12-12T08:41:14Z INFO [KUBERNETE] Round trip: POST https://100.64.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews, code: 201, duration: 42.182415ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport.cluster.local pid:7.1 reverseproxy/reverse_proxy.go:223

I do not experience any issues with kubectl with tsh kube. Is this an expected difference? I would not usually expect going through proxy and not to have any difference with tooling like k9s.

[Ezzahhh]

I would just add that enabling debug logs on the cluster side did not reveal that much. When I observe hangs or disconnects in k9s there are no accompanying logs in the proxy service. If k9s does eventually connect after a bunch of retries, I do se the round trip messages as shown in the original post although sometimes with higher durations (couple hundred ms). This behaviour is observable in two clusters that are connected to Teleport (the only two joined so far during testing).