Running Tethys on Kubernetes

[gronka]

Hi,

I'd like to run Tethys on Kubernetes. In order to do this, I need to pass the LoadBalancer IP to the Django ALLOWED_HOSTS variable. It looks like this could be done in docker/run.sh if we lookup a new environment variable such as K8S_LOADBALANCER_NAME=TETHYS_LB_SERVICE_HOST, to poitn to an environment variable automatically passed by k8s to out container if our load balancer has the name tethys-lb.

I was told other people are already running Tethys on k8s, so I'm wondering how they're currently achieving this.

Thanks

[sdc50]

@swainn do you have any advice here? I don't understand the details of running Tethys on Kubernetes well enough to understand why this is needed. If we need to add something to run.sh to accommodate Kubernetes then we need to make sure that it is only used when it's applicable.

[swainn]

Hi, yes, we are using k8s for our deployments at Aquaveo. I've asked @romer8 to comment.

[romer8]

Hi @gronka, If the IP of the LoadBalancer needs to be added to the ALLOWED_HOSTS variable in the portal config this can also be done by creating a salt state file that sets the ALLOWED_HOSTS in the portal config through a env variable (using the tethys settings --set command). The run.sh will run all the salt state files that are in the top.sls file (you might need to create a new custom top.sls file that includes this new salt state file that sets the ALLOWED_HOSTS). Finally, you can build the docker image from the Dockerfile and use it in the k8 pod that will have tethys running. The k8 pod definition should let you specify the image and the env variables. I have a couple of questions:

1. Are planning to use vanilla k8?
2. Do you plan to use Helm for the deployment?
3. Which cloud service do you plan to user if any?

Documentation about salt files: https://docs.tethysplatform.org/en/stable/installation/production/docker/salt_state.html

[gronka]

Interesting process. Thank you for that description and the link.

1. The goal is to have it running on Azure KS. Although atm we also have multiple testing environments. I think maintaining compatibility with vanilla k8s is probably good.
2. I don't know. Do you recommend it?
3. Azure KS

[romer8]

@gronka. If the goal is to have the deployment to run in Azure KS. I think you can use two different approaches:

1. if you plan to use vanilla k8 for the deployment (with all the .yamls) I think a good tool might be kustomize which allows to work with Kubernetes manifests, natively supported in kubectl from Kubernetes v1.14 onwards. It follows a "patch" approach, allowing you to declaratively modify existing manifests rather than using templates. This works great if you usually patch the k8 deployment.
2. Helm charts: It consists of charts, which are pre-configured templates for Kubernetes manifests, and releases, which are instances of those charts deployed to Kubernetes clusters. The great power of hel charts is that it allows to to parameterize Kubernetes manifests, allowing for dynamic configuration and easy customization.

Both should work well on Azure KS, and even with terraform. It depends the route you are trying to go.

Some examples that I have worked on:

• terraform for a helm tethys chart in aws: https://github.com/CIROH-UA/tethysportal-ciroh/blob/main/terraform/modules/ciroh_portal_dev/tethys_portal.tf
• helm chart for a tethys portal: https://github.com/CIROH-UA/tethysportal-ciroh/tree/main/charts/ciroh

Lamentably I have not used Azure KS, but the helm chart should help as good guide if you decide to use helm for k8 deployment

[sdc50]

In my UBI Tethys Docker repo I add a variable ADDITIONAL_SALT_SCRIPTS. The top.sls file is generated at runtime, so it can read the ADDTIONAL_SALT_SCRIPTS variable and add those to the top.sls. Unfortunately, I haven't yet integrated that back into the original Tethys repo.

Also, this approach would still require you to generate a new docker image. I think you are trying to test k8s with the original docker image, right?

[romer8]

This seems like a really neat solution

[gronka]

I understand now. Thanks. I got it running with the method I described, but this is better.

I've been building a Docker image since last week. So, if I still say "original docker image" I mean a docker image based on open source Tethys.

[romer8]

@gronka if that is the case, that you want to use the original Tethys image without creating one. I think creating a job.yaml that runs the tethys settings command to set up the ALLLOWED_HOSTS to include the load balancer IP can also be a solution. I might be similar to adding additional salt script through a run time generated top.sls, the only difference should be that the settings re setup when the run.sh has finished instead of doing it as an extra salt script.

[gronka]

I tried this and it works (well, I'm new to Tethys, so I don't know what all I should test yet). So, it was very simple - I mistakenly thought ALLOWED_HOSTS had to be set during the docker build step.

[gronka]

The solution to the ALLOWED_HOSTS question is here: #1018 (comment)

Next, I think I should convert the HEALTHCHECK in the Dockerfile to a Kubernetes liveness probe. @romer8 Have you done this? It would save me some time if you have. If not, I can share what I come up with.

[gronka]

I created a script that can be used a livenessProbe, and put it in this PR:
#1038

[gronka]

For posterity, I've been running tethys with the following config. The image used here is 1 year old - my testing has worked with an image I've built myself from the main branch.

I haven't tested this exact config - just gathering something to post.

Thanks for everyone's help.

There is an issue where supervisord fails to start nginx, so I'll have to SSH into the tethys pod to manually start tethys. I hope this is due to not using a domain name.

$pod=$1
kubectl exec -n tethys-blue --stdin --tty $pod -- /bin/bash
service nginx start

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tethys
  namespace: tethys
  labels:
    name: tethys
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tethys
  template:
    metadata:
      labels:
        app: tethys
    spec:
      containers:
      - name: tethys
        image: tethysplatform/tethys-core:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
        # these variables are injected by Kubernetes. They don't need to be used, but someone might find them useful
        env : 
        - name: K8S_HOST_IP
          valueFrom: 
            fieldRef:
              fieldPath: status.hostIP
        - name: K8S_POD_IP
          valueFrom: 
            fieldRef:
              fieldPath: status.podIP
        - name: ALLOWED_HOSTS
          value: "\"[localhost, 127.0.0.1, {Domain name or IP}]\""
        - name: CLIENT_MAX_BODY_SIZE
          value: 75M
        - name: NGINX_PORT
          value: "80"
        - name: TETHYS_DB_HOST
          value: tethys-pg
        - name: TETHYS_DB_NAME
          value: tethys_platform
        - name: TETHYS_DB_USERNAME
          value: tethys_default
        - name: TETHYS_DB_PASSWORD
          value: pass
        - name: TETHYS_DB_SUPERUSER
          value: tethys_super
        - name: TETHYS_DB_SUPERUSER_PASS
          value: pass
        - name: TETHYS_PORT
          value: "8000"
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - '-c'
            - /usr/lib/tethys/liveness-probe.sh
---
apiVersion: v1
kind: PersistentVolume
metadata:
  # pv's don't have a namespace
  name: tethys-postgres-data
  labels:
    type: local
spec:
  storageClassName: manual
  capacity: 
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/volume/tethys"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tethys-postgres-data-claim
  namespace: tethys
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: tethys-pg-config
  namespace: tethys
  labels:
    app: tethys-pg
data:
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: pass 
  POSTGRES_HOST_AUTH_METHOD: trust
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tethys-pg
  namespace: tethys
  labels:
    app: tethys-pg
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tethys-pg
  template:
    metadata:
      labels:
        #run: tethys-pg
        app: tethys-pg
    spec:
      containers:
      - name: postgres
        image: postgres:14.4
        imagePullPolicy: "IfNotPresent"
        ports:
          - containerPort: 5432
            name: tethys-pg
        envFrom:
          - configMapRef:
              name: tethys-pg-config
        volumeMounts:
          - mountPath: /var/lib/postgresql/data
            name: postgresdb-teth
      volumes:
        - name: postgresdb-teth
          persistentVolumeClaim:
            claimName: tethys-postgres-data-claim
---
apiVersion: v1
kind: Service
metadata:
  name: tethys-pg
  namespace: tethys
  labels:
    app: tethys-pg
spec:
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    app: tethys-pg

[swainn]

Thanks @gronka !