Using teleport with AWX

[jmarler]

Has anyone been able to figure out how to use teleport with AWX? What I mean by that is:

• Provide a dynamic inventory to AWX from teleport (The ansible-teleport python script)
• Use teleport ssh proxy to execute ansible commands from AWX to teleport servers

I have teleport providing dynamic inventory to ansible on my local system, with ansible configured to use teleport's ssh proxy, and that works perfectly. What I can't figure out is how to get that working inside AWX. I know that I need to get teleport installed in various containers for AWX, I need teleport service accounts setup, and AWX has to be configured to use teleport when connecting by ssh to inventory hosts, and the dynamic inventory plugin has to be able to connect to teleport via that service account automatically.

Has anyone done that? The AWX documentation is pretty sparse on adding extra tools to the k8s images it deploys.

[jervine]

I've been trying to get AWX to run templates using teleport as the SSH provider as well:
Is it possible to have Ansible AWX work with teleport?

I have got a tbot deployment that runs on a kubernetes cluster and generates a machineid identity as a kubernetes secret (and also tested with a directory output as well). I would like to have Ansible AWX run playbooks using the identity that tbot generates. From the documentation on the website, Ansible (not taking AWX into account yet) can make use of the machineid ssh_config to connect to remote hosts. This seems to be a decent start.

AWX spins up a new 'execution environment' each time a template (playbook) is executed, and the playbook is executed from this environment. For teleport, one challenge is that the fresh execution environment should have the teleport credentials available. This can probably be resolved by mounting the secret into the execution environment (although it's not clear if this is possible).

The machineid output as a kubernetes secret does not contain the ssh_config details though, and this would be problematic if trying to configure the AWX execution environment use the custom teleport ssh_config file. This might necessitate the use of a directory output.

An option here could be to run tbot outside of kubernetes (or as DaemonSet with a hostDir volume) on each worker node that the AWX environment can run on. This would allow the machineid output to point to a node filesystem directory which is then imported (path exposed to isloated jobs) within AWX to the execution environment. Neither of these strike me as ideal.

Has anyone got teleport working with AWX?

[AlexisDuf]

Hi @jervine

I've succeeded to use Teleport with AWX without any problems.
I use a kubernetes cluster for my AWX, and for teleport credential I use the machine ID and tbot command.
I created a custom Container group with an initContainer for getting my ssh config from tbot by using directory output method.
As I'm using a volume mount I'm able to get the credentials for my ansible runner.
You need to build a custom execution environment as you need tbot cli in the ansible runner for ssh connection'

Please find below my container group pod spec:

apiVersion: v1
kind: Pod
metadata:
  namespace: awx
spec:
  serviceAccountName: tbot
  automountServiceAccountToken: false
  initContainers:
    - name: tbot
      image: public.ecr.aws/gravitational/teleport:14.2.2
      command:
        - /bin/sh
        - -c
        - "tbot start -c /config/tbot.yaml --oneshot && chown -R 1000 /runner/awx && chmod -R 700 /runner/awx"
      env:
        # POD_NAMESPACE is required for the kubernetes_secret` destination
        # type to work correctly.
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        # KUBERNETES_TOKEN_PATH specifies the path to the service account
        # JWT to use for joining.
        # This path is based on the configuration of the volume and
        # volumeMount.
        - name: KUBERNETES_TOKEN_PATH
          value: /var/run/secrets/tokens/join-sa-token
        # TELEPORT_ANONYMOUS_TELEMETRY enables the submission of anonymous
        # usage telemetry.  This helps us shape the future development of
        # `tbot`. You can disable this by omitting this.
        - name: TELEPORT_ANONYMOUS_TELEMETRY
          value: "1"
      volumeMounts:
        - mountPath: /config
          name: config
        - name: tsh
          mountPath: /runner/awx
        - mountPath: /var/run/secrets/tokens
          name: join-sa-token
      securityContext:
          fsGroup: 1000
  containers:
    - image: quay.io/ansible/awx-ee:latest
      name: worker
      env:
        - name: ANSIBLE_CONFIG
          value: /runner/.ansible/ansible.cfg
      args:
        - ansible-runner
        - worker
        - '--private-data-dir=/runner'
      resources:
        requests:
          cpu: 250m
          memory: 100Mi
      volumeMounts:
        - name: tsh
          mountPath: /runner/awx
      securityContext:
        runAsUser: 1000          
  volumes:
    - name: config
      configMap:
        name: tbot-config
    - name: tsh
      emptyDir: {}
    - name: join-sa-token
      projected:
        sources:
          - serviceAccountToken:
              path: join-sa-token
              # 600 seconds is the minimum that Kubernetes supports. We
              # recommend this value is used.
              expirationSeconds: 600
              # `example.teleport.sh` must be replaced with the name of
              # your Teleport cluster.
              audience: xxxxxxxx

[UPDATE - 09.01.2024]
I'm currently facing to an issue when I'm using many hosts (more than 4), so if you limit the number of hosts it works without problem.
At a period of time the runner is not able to connect on the VMs using MachineID. I created a case to teleport, waiting for their feedbacks

[mfominov]

@AlexisDuf how u add init container?
I can't find docs in AWX Operator about init container.

[AlexisDuf]

Hello @mfominov

You can setup the init container from the UI of AWX in Administration > Instance Groups and you select an existing one or you create a new one.
You just need to tick "Customize pod specification" then change the pod spec and adding the initContainer

Hope it helps

[mfominov]

@AlexisDuf Thx)