WHO Owner

Owned by the Global Tuberculosis Programme, GTB, Geneva Switzerland. References: Carl-Michael Nathanson.

Tbsequencing backend

The backend of the tbsequencing portal is based on the Django Rest Framework. All infrastructure required for running the backend is defined under the main repository.

The deployment workflows includes the following:

1. Collecting the static files and copying them into an S3 bucket
2. Building the backend docker image and pushing at ECR
3. Running the migration(s)
4. Forcing deployment of the ECS fargate task to use latest docker image

The backend will be accessible by adding the suffix "/admin/" to your chosen domain address.

Authentication is handled by OIDC connected to an Entra ID tenant.

Authentication configuration

The system has been integrated to work with Entra ID (formerly Azure AD) authentication, using azure/msal-react on the frontend and django_auth_adfs on the backend.

msal-react uses ID tokens whereas django_auth_adfs uses Access tokens. Reference: https://oauth.net/id-tokens-vs-access-tokens/

To configure the application on Entra ID, we will need to register two apps, one for the frontend and one for the backend.

Backend application

For the backend Entra ID application, set the Redirect URI in a Web Scope to https://domain-address/api/v1/oauth2/callback

⚠️For testing and development purposes only, the Redirect URI can be defined under http scheme, only with domain name localhost.⚠️

Expose an API that both Admins and users can consent to.

Retrieve the Tenant ID, the Application ID and create a client secret so that you can fill the following keys in the AWS Secrets that was created by the main repository

Key name	Notes
ADFS_TENANT_ID	Tenant identifier
ADFS_CLIENT_ID	Application (client) ID of the backend in the Entra ID app declaration
ADFS_CLIENT_SECRET	Client Secret value, requested earlier on the Entra ID backend app declaration, under the Certificates & secrets tab

Access to the admin django panel

The Django admin panel will be served at https://domain-address/admin/

Each users that must be granted access to the admin panel must be assigned this role under the "Enterprise applications" view, which is available on the landing panel of the "Azure AD" service.

Under "Enterprise applications", select the "Users and groups" panel. There select "Add User/Group" and assign the users to the admins groups.

Finally, the Entra ID admin must grant consent for the permission in the same "Enterprise application" menu.

Development set up

Project setup

There are 2 main setup options to make project run on your machine, which one do you need usually depends on your purposes.

Also, it is possible to run a project in both setups.

Local setup

Local setup is good for development and debugging purposes.

To run the webservice locally, you need python >=3.10, pip and pipenv installed. Also, Postgres database connection is required.

If you're going to work with submission flow (upload FASTQ files or use S3 storage backend), you will need AWS connection, with at least 1 S3 bucket read/write access.

You can spin up local database using docker-compose, by default it will be available at localhost:5433/postgres with username:password postgres:postgres

docker-compose up -d db

1. Initialize virtual environment and install packages. --dev flag will install packages, required to run tests and pre-commit hooks:

   pipenv install --dev
   

2. If you are going to make changes to the repo, install pre-commit hooks into it:

   pipenv run pre-commit install
   

   You can exclude pipenv run from commands, after activating pipenv shell. To activate it, run pipenv activate from project root.

3. Copy .env.dist into .env, and update necessary variables.

   More on env vars and configuration in Configuration section.

4. Run database migrations.

   migrate command will apply any pending migrations to the database.

   postmigrate command will perform necessary updates in database, such as updating frontend domain URL, setting custom postgres SEARCH PATH, filling genphen tables with initial data, and, for non-production environment, creating sample users.

   pipenv run python manage.py migrate
   pipenv run python manage.py postmigrate
   

   More on database change management and migrations in Database change management section.

5. Run local server.

   By default, it resides on http://localhost:8000

   pipenv run python manage.py runserver
   

6. When needed, you can run tests:

    pipenv run pytest
   

   The tests will utilize current database connection. More on testing in Testing stage.

docker-compose setup

docker-compose setup can be used by dependent projects, such as frontend, to quick-start backend application with all necessary dependencies, including Swagger UI for interactive API schema view.

Unlike local setup, docker-compose setup runs web application in production mode, means serving static files through nginx, and Django through WSGI-server gunicorn.

To spin up docker-compose setup, run command:

docker-compose up -d --build

This will update and start all services.

After services were launched, migrate database to the latest state:

docker-compose exec web python manage.py migrate
docker-compose exec web python manage.py postmigrate

This will apply any necessary migrations, as well as fill database with initial data.

docker-compose services include:

Postgres database

Exposed on localhost:5433/postgres with postgres:postgres credentials by default.

Can be used as standalone database service for local setup as well.

Mailcatcher

Intercepts all Django's outgoing emails, and shows them in a web interface.

Web interface is on localhost:1080, SMTP server is on localhost:1025.

Nginx proxy

Proxies Django webapp and serves its static files. Exposed on localhost:8000.

Django webapp

Exposed on localhost:8001, but access through nginx should be preferred over it.

Swagger UI

Interactive API schema exploration. Exposed on localhost:2080. Pre-pointed to backend API schema that is currently located inside project.

Configuration

Django application is configured via environment variables. They can be passed directly or defined in .env file inside project root for local setup, and .dc.env for docker-compose setup.

.dc.env already have some variables pre-defined, according to docker-compose.yml settings. You can change them as needed, for instance if you want to connect different database to a docker-compose setup.

Reference file with all available variables and their descriptions located at project root and called .env.dist.

To configure local setup, copy .env.dist file to .env, and change all necessary variables.

Making changes

To make changes to the project, you will need to deploy a local setup.

Standard approach to make changes:

• Make any necessary changes.

• Run makemigrations command to generate migrations if there are any changes to managed models:

  pipenv run python manage.py makemigrations
  

• Run migrate command to apply new migrations (if any):

  pipenv run python manage.py migrate
  

• Run tests. Write additional tests if needed.

• Update API schema.

• Commit changes, fixing pre-commit checks suggestions.

Testing

In order to launch tests against application, you will need local setup configured. After you have your local backend app working, from project root run the command to launch tests:

pipenv run pytest

Updating OpenAPI schema

If your changes affected API, regenerate OpenAPI schema to automatically reflect the changes. Run the command from project root:

pipenv run python manage.py generateschema --format openapi-json --file api/static/api/schema.json

After that, you can review changes (with text editor or Swagger UI). If all good, feel free to commit the schema changes alongside with code changes.

Managing database changes

See Database change management section on how to deal with database changes.

Connecting a local Django web container with a remote database

Use an ssh tunnel to allow the local container to connect to the remote database. Use the following options:

-g (Allows remote hosts to connect to local forwarded ports.)
-L (Specifies that connections to the given TCP port or Unix socket on the local (client) host are to be forwarded to the given host and port, or Unix socket, on the remote side.)

ssh -g -L ${local_port}:${remote_database_host_name}:${remote_port} ec2-user@${ec2_hostname/ip_address}

If you use the default neworking mode in docker compose (i.e. bridge), you must the correct local ip address for the database host. It cannot be localhost (unless you change the networking mode to host). Usually it's 172.17.0.1.

Modify the following environment variable in .dc.env. You can fetch from secret manager the master username/master password.

DB_HOST=172.17.0.1
DB_PORT=5432
DB_USER=tbkbmasteruser
DB_PASSWORD=averysecretpassword
DB_NAME=tbkbdb

Only ever connect to the staging/test remote database. Connecting via SSH to the production database should be impossible anyway.