Multiplexing and reverse tunnel

[evrynet1]

Teleport version: 14.3.3

I have just enabled multiplexing and when adding new SSH nodes (resources) reverse tunnels are being used which is much slower than the usual connection. How can I switch back to normal connections?

P.S.: still would like to use the advantages of multiplexing, the legacy listeners are just there for backward compatibility

teleport.yaml:

version: v3
teleport:
nodename: 198-74-48-226.ip.linodeusercontent.com
data_dir: /var/lib/teleport
join_params:
token_name:
method: token
proxy_server: gw.domain.com:443
log:
output: stderr
severity: INFO
format:
output: text
ca_pin:
diag_addr: ""
auth_service:
enabled: "no"
ssh_service:
enabled: "yes"
labels:
teleport.internal/resource-id:
commands:

• name: hostname
  command: [hostname]
  period: 1m0s
  proxy_service:
  enabled: "no"
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme: {}
  Thank you

Thank you

[evrynet1]

If I change this to auth_server with the following client teleport.yaml it shows the following after connecting:

Jan 25 21:16:50 localhost teleport: 2024-01-25T21:16:50Z INFO [NODE:1] Connected to cluster over tunnel connection, ignoring public_addr setting. pid:1261.1 service/service.go:2555

teleport:
nodename: 198-74-48-226.ip.linodeusercontent.com
auth_token:
ca_pin:

auth_servers:

• gw.domain.com:443
  ssh_service:
  enabled: yes
  public_addr: 127.0.0.1
  commands:
  • name: uptime
    command: [uptime]
    period: 30m0s
    labels:
    client: "Test"

[zmb3]

You are joining via the proxy server's reverse tunnel because you are specifying proxy_server in your agent config:

version: v3
teleport:
  proxy_server: proxy.example.com:443

If you want to dial agents directly instead of via reverse tunnel, you should configure your agents to connect to the auth server, not the proxy server.

version: v3
teleport:
  auth_server: auth.example.com:3025

As a side note, you included a join token in your message above, which is considered a secret and should not be shared. I'd recommend rotating it now that it's been exposed.

[evrynet1]

See my previous response please. I have also tried it with auth_server as well. I thought the idey behind multiplexing is that the connections are wrapped in TLS and multiplexed on one service port. The auth server is specified with 443 port in the agent config but it is still tunneling the connection.

[root@test ~]# curl https://gw.domain.com/webapi/ping | jq '.proxy'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 778 100 778 0 0 1208 0 --:--:-- --:--:-- --:--:-- 1208
{
"kube": {
"enabled": true,
"listen_addr": "0.0.0.0:443"
},
"ssh": {
"listen_addr": "0.0.0.0:443",
"tunnel_listen_addr": "0.0.0.0:443",
"web_listen_addr": "0.0.0.0:443",
"public_addr": "gw.domain.com:443"
},
"db": {
"postgres_listen_addr": "0.0.0.0:443",
"mysql_listen_addr": "0.0.0.0:443"
},
"tls_routing_enabled": true,
"assist_enabled": false
}
[root@bq ~]#

Jan 25 21:16:50 localhost teleport: 2024-01-25T21:16:50Z INFO [NODE:1] Connected to cluster over tunnel connection, ignoring public_addr setting. pid:1261.1 service/service.go:2555

teleport:
nodename: 198-74-48-226.ip.linodeusercontent.com
auth_token:
ca_pin:

auth_servers:

gw.domain.com:443
ssh_service:
enabled: yes
public_addr: 127.0.0.1
commands:
name: uptime
command: [uptime]
period: 30m0s
labels:
client: "Test"

[zmb3]

Take a closer look at my answer.

• You tried it with auth_servers, not auth_server.
• You still specified a proxy address on port 443, not an auth server address on port 3025.

Multiplexing is a feature of the proxy. All traffic on the proxy server is multiplexed on a single port. If you want direct dial, agents must connect to auth, not proxy, and auth doesn't listen on port 443.

[evrynet1]

ah I see so it is still not possible to put both auth and proxy behind like CF right? which only supports 443/80 ports? is it possible to get auth listen on port 443 as well?

[zmb3]

It's possible, but you'd need a separate load balancer for auth. Both auth and proxy can't listen on the same port on the same host.

[evrynet1]

how about same host, different IPs?

[webvictim]

Give it a try!

[evrynet1]

Can you double check my configs please?

[root@eu ~]# netstat -ntpln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:1167            0.0.0.0:*               LISTEN      25000/cdp           
tcp        0      0 192.168.1.2:443       0.0.0.0:*               LISTEN      20271/teleport      
tcp        0      0 192.168.1.3:443      0.0.0.0:*               LISTEN      20271/teleport      
tcp6       0      0 :::3022                 :::*                    LISTEN      20271/teleport      
[root@eu ~]# 

version: v2
teleport:
  nodename: eu.domain.com
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: []
  diag_addr: ""
auth_service:
  enabled: "yes"
  proxy_listener_mode: multiplex
  listen_addr: 192.168.1.3:443
  cluster_name: eu.domain.com
  client_idle_timeout: 15m
  keep_alive_interval: 5s
  keep_alive_count_max: 3
  authentication:
    type: local
    second_factor: on
    webauthn:
      rp_id: gw.domain.com
ssh_service:
  enabled: "yes"
  labels:
    env: private
  commands:
  - name: uptime
    command: [uptime]
    period: 30m0s
proxy_service:
  enabled: "yes"
  tunnel_listen_addr: 192.168.1.2:443
  web_listen_addr: 192.168.1.2:443
  public_addr: ["gw.domain.com:443","domain.com:443","eu.domain.com:443"]
  https_keypairs:
    - key_file: /etc/letsencrypt/live/domain.com/privkey.pem
      cert_file: /etc/letsencrypt/live/domain.com/fullchain.pem
  acme:
    enabled: "no"
    email: test@test.com

Client side:

version: v3
teleport:
  nodename: test.test.com
  auth_token: <redacted>
  ca_pin:
  - 
  auth_server: da.domain.com:443
ssh_service:
      enabled: yes
      public_addr: 127.0.0.1
      commands:
      - name: uptime
        command: [uptime]
        period: 30m0s
      labels:
        client: "name Name"

Is that supposed to be correct? da.domain.com is pointed to 192.168.1.3

[webvictim]

I fixed the formatting of your configs for you. In future you can use three backticks (```) to define code blocks so they're readable.

The configs look fine. Setting tunnel_listen_addr on the proxy_service is likely doing nothing and could be removed.

[evrynet1]

@greedy52 could you see this thread please? I found an older one here where websockets and ALPN routing were tested against well-known proxies like CloudFlare and as I understand this should be compatible now as per 36343? Could you advise here please?

#36343

Both proxy and auth are on the same node behind CF (without tunnel).

[greedy52]

Can the auth server work behind CF without using CF tunnel right now or after your change is released?

My change only works on Proxy. I am not sure if CF or CF tunnel will work for Auth as Teleport needs mTLS (I don't have a lot of experience with CF).

Either way, I don't think it's a good idea to expose Auth to the world. If the SSH nodes are outside the same network, they should connect to the Proxy and utilize reverse tunnels.

How bad is the latency?

[evrynet1]

@greedy52 I did some testing today and it seems the latency more or less the same so ok but now you mentioned exposing auth to the world is not a good idea so I need some clarification here if I may. I currently have SSH nodes connected 2 different ways as seen on this screenshot:

"NetD-Test" teleport.yaml (v2) has the auth_servers set to domain:3025 (without CF)...does it mean it is connecting directly to the auth server?

The other teleport.yaml (v2) has the auth_servers set to domain:433 (with CF) and a reversel tunnel connection is setup, that is connecting through proxy is that right? Thank you!

[greedy52]

"NetD-Test" teleport.yaml (v2) has the auth_servers set to domain:3025 (without CF)...does it mean it is connecting directly to the auth server?
The other teleport.yaml (v2) has the auth_servers set to domain:433 (with CF) and a reversel tunnel connection is setup, that is connecting through proxy is that right? Thank you!

Yes and yes. 98.xxx.xxx.xxx:3022 is the public address for direct connection from client, and <- tunnel means it always goes through Proxy's reverse tunnel.

If teleport.yaml is v2, it will first try a "direct" connect to auth_servers. When fails, it will try the first address of auth_servers as Proxy using reverse tunnel.

If using teleport.yaml v3, the service will only try "direct" connect to auth_servers. And you would want to explicitly specify proxy_server to connect to Proxy using reverse tunnel.

[evrynet1]

@greedy52 ok last confirmation then so "direct" connect to auth_servers should be avoided as its directly connecting/exposing auth server and proxy connection using reverse tunnels is the recommended/preferred way?

[greedy52]

so "direct" connect to auth_servers should be avoided as its directly connecting/exposing auth server and proxy connection using reverse tunnels is the recommended/preferred way?

Good question. I think it depends on the networking configurations.

Proxy connection using reverse tunnels is recommended when the SSH nodes/Teleport agents are not in the same local network. Or if you only want outgoing connections from the SSH nodes/Teleport agents.

[webvictim]

Did you write this with AI? There is no reverse_tunnel_enabled option in Teleport config files.

[webvictim]

The SSH ControlMaster multiplexing settings have no relation to Teleport's TLS multiplexing.

[webvictim]

As above, the SSH ControlMaster multiplexing settings have no relation to Teleport's TLS multiplexing.

Please don't post AI-generated garbage on discussion posts like this, especially without checking whether it's relevant, useful and most of all, correct. At best it's a waste of everybody's time and at worst, it's confusing.