Skip to content

Commit

Permalink
More CI docs (#554)
Browse files Browse the repository at this point in the history 
* More CI docs

* Better document setting up CI
  • Loading branch information
@ErikSchierboom
ErikSchierboom authored Aug 8, 2024
1 parent 6ca912f commit aec4d35
Showing 1 changed file with 284 additions and 10 deletions.
294 changes: 284 additions & 10 deletions building/tracks/new/setup-continuous-integration.md
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,290 @@
# Set up Continuous Integration

Setting up Continuous Integration (CI) for your track is very important, as it helps to automatically catch mistakes.
Setting up Continuous Integration (CI) for your track is very important, as it helps automatically catch mistakes.

Our tracks all use [GitHub Actions](https://docs.github.com/en/actions) to run their CI.
GitHub actions uses the concept of _workflows_, which are scripts that are run automatically whenever a specific event occurs (e.g. pushing a commit).
## GitHub Actions

Each workflow corresponds to a file in `.github/workflows`.
Each new track repository comes pre-loaded with three workflows:
Our tracks (and other repositories) use [GitHub Actions](https://docs.github.com/en/actions) to run their CI.
GitHub Actions uses the concept of _workflows_, which are scripts that run automatically whenever a specific event occurs (e.g. pushing a commit).

- `test.yml`: this workflow should run the tests for each exercise the track has implemented
- `configlet.yml`: this workflow runs the [configlet tool](/docs/building/configlet), which checks if a track's (configuration) files are properly structured - both syntactically and semantically.
- `sync-labels.yml`: this workflow automatically syncs the repository's labels from a `labels.yml` file
Each GitHub Actions workflow is defined in a `.yml` file in the `.github/workflows` directory.
For information on workflows, check the following docs:

Of these three workflows, only the first workflow will need some manual work.
To find out what needs to happen, please check the `test.yml` file's contents, which has TODO comments to help you.
- [Workflow syntax](https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions)
- [Choosing when your workflow runs](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow)
- [Choosing where your workflow runs](https://docs.github.com/en/actions/writing-workflows/choosing-where-your-workflow-runs)
- [Choose what your workflow does](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does)
- [Writing workflows](https://docs.github.com/en/actions/writing-workflows)
- [Best practices](/docs/building/github/gha-best-practices)

## Pre-defined workflows

A track repository contains several pre-defined workflows:

- `configlet.yml`: runs the [configlet tool](/docs/building/configlet), which checks if a track's (configuration) files are properly structured - both syntactically and semantically
- `no-important-files-changed.yml`: checks if pull requests would cause all existing solutions of one or more changes exercises to be re-run
- `sync-labels.yml`: automatically syncs the repository's labels from a `labels.yml` file
- `test.yml`: verify the track's exercises

Of these workflows, _only_ the `test.yml` workflow requires manual work.
The other workflows should not be changed (we keep them up-to-date automatically).

## Test workflow

The test workflow should verify the track's exercises.
The workflow itself should not do much, except for:

- Checking out the code (already implemented)
- Installing dependencies (e.g. installing an SDK, optional)
- Running the script to verify the exercises (already implemented)

### Verify exercises script

As mentioned, the exercises are verified via a script, namely the `bin/verify-exercises` (bash) script.
This script is _almost_ done, and does the following:

- Loops over all exercise directories
- For each exercise directory, it then:
- Copies the example/exemplar solution to the (stub) solution files (already implemented)
- Calls the `unskip_tests` function in which you can unskip tests in your test files (optional)
- Calls the `run_tests` function in which you should run the tests (required)

The `run_tests` and `unskip_tests` functions are the only things that you need to implement.

### Unskipping tests

If your track supports skipping tests, we must ensure that no tests are skipped when verifying an exercise's example/exemplar solution.
In general, there are two ways in which tracks support "unskipping" tests:

1. Removing annotations/code/text from the test files.
For example, changing `test.skip` to `test`.
2. Providing an environment variable.
For example, setting `SKIP_TESTS=false`.

If skipping tests is file-based (the first option mentioned above), edit the `unskip_tests` function to modify the test files (the existing code already handles the looping over the test files).

```exercism/note
The `unskip_test` function runs on a copy of an exercise directory, so feel free to modify the files as you see fit.
```

If unskipping tests requires an environment variable to be set, make sure that it is set in the `run_tests` function.

### Running tests

The `run_tests` function is responsible for running the tests of an exercise.
When the function is called, the example/exemplar files will already have been copied to (stub) solution files, so you only need to call the right command to run the tests.

The function must return a zero as the exit code if all tests pass, otherwise return a non-zero exit code.

```exercism/note
The `run_tests` function runs on a copy of an exercise directory, so feel free to modify the files as you see fit.
```

### Example: Arturo track

This is what the [`bin/verify-exercises` file](https://github.com/exercism/arturo/blob/79560f853f5cb8e2f3f0a07cbb8fcce8438ee996/bin/verify-exercises) looks file for the Arturo track:

```bash
#!/usr/bin/env bash

# Synopsis:
# Test the track's exercises.

# Example: verify all exercises
# ./bin/verify-exercises

# Example: verify single exercise
# ./bin/verify-exercises two-fer

set -eo pipefail

required_tool() {
command -v "${1}" >/dev/null 2>&1 ||
die "${1} is required but not installed. Please install it and make sure it's in your PATH."
}

required_tool jq

copy_example_or_examplar_to_solution() {
jq -c '[.files.solution, .files.exemplar // .files.example] | transpose | map({src: .[1], dst: .[0]}) | .[]' .meta/config.json | while read -r src_and_dst; do
cp "$(echo "${src_and_dst}" | jq -r '.src')" "$(echo "${src_and_dst}" | jq -r '.dst')"
done
}

unskip_tests() {
jq -r '.files.test[]' .meta/config.json | while read -r test_file; do
sed -i 's/test.skip/test/g' "${test_file}"
done
}

run_tests() {
arturo tester.art
}

verify_exercise() {
local dir
local slug
local tmp_dir

dir=$(realpath "${1}")
slug=$(basename "${dir}")
tmp_dir=$(mktemp -d -t "exercism-verify-${slug}-XXXXX")

echo "Verifying ${slug} exercise..."

(
cp -r "${dir}/." "${tmp_dir}"
cd "${tmp_dir}"

copy_example_or_examplar_to_solution
unskip_tests
run_tests
)
}

exercise_slug="${1:-*}"

shopt -s nullglob
for exercise_dir in ./exercises/{concept,practice}/${exercise_slug}/; do
if [ -d "${exercise_dir}" ]; then
verify_exercise "${exercise_dir}"
fi
done
```

It uses `sed` to unskip tests:

```bash
sed -i 's/test.skip/test/g' "${test_file}"
```

and runs the tests via the `arturo` command:

```bash
arturo tester.art
```

## Implement the test workflow

The goal of the test workflow (defined in `.github/workflows/test.yml`) is to automatically verify that the track's exercises are in proper shape.
The workflow is setup to run automatically (in GitHub Actions terminology: is _triggered_) when a push is made to the `main` branch or to a pull request's branch.

There are three options when implementing this workflow:

### Option 1: install track-specific tooling (e.g. an SDK) in the GitHub Actions runner instance

In this approach, any track-specific tooling (e.g. an SDK) is installed directly in the GitHub Actions runner instance.
Once done, you then run the `bin/verify-exercises` script (which assumes the track tooling is installed).

For an example, see the [Arturo track's `test.yml` workflow](https://github.com/exercism/arturo/blob/79560f853f5cb8e2f3f0a07cbb8fcce8438ee996/.github/workflows/test.yml):

```yml
name: Test

on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:

jobs:
ci:
runs-on: ubuntu-22.04

steps:
- name: Checkout repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332

- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install libgtk-3-dev libwebkit2gtk-4.0-dev libmpfr-dev
- name: Install Arturo
run: bin/install-arturo
env:
GH_TOKEN: ${{ github.token }}

- name: Verify all exercises
run: bin/verify-exercises
```
#### Option 2: running the verify exercises script within test runner Docker image
In this option, we're using the fact that each track must have a test runner which has all dependencies already installed
To enable this option, we need to set the workflow's container to the test runner:
```yml
container:
image: exercism/vimscript-test-runner
```
This will then automatically pull the test runner Docker image when the workflow executes, and run the `verify-exercises` script within that Docker container.

```exercism/note
The main benefit of this approach is that it better mimics how tests are being run in production (on the website).
With the approach, it is less likely that things will fail in production that passed in CI.
The downside of this approach is that it likely is slower, due to having to pull the Docker image and the overhead of Docker.
```

For an example, see the [vimscript track's `test.yml` workflow](https://github.com/exercism/vimscript/blob/e599cd6e02cbcab2c38c5112caed8bef6cdb3c38/.github/workflows/test.yml).

```yml
name: Verify Exercises
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
jobs:
ci:
runs-on: ubuntu-24.04
container:
image: exercism/vimscript-test-runner
steps:
- name: Checkout repository
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332
- name: Verify all exercises
run: bin/verify-exercises
```

### Option 3: download the test runner Docker image and change verify exercises script

In this option, we're using the fact that each track must have a test runner which already knows how to verify exercises.
To enable this option, we first need to download (pull) the track's test runner Docker image and then run the `bin/verify-exercises` script, which is modified to use the test runner Docker image to run the tests.

```exercism/note
The main benefit of this approach is that it best mimics how tests are being run in production (on the website).
With the approach, it is less likely that things will fail in production that passed in CI.
The downside of this approach is that it likely is slower, due to having to pull the Docker image and the overhead of Docker.
```

For an example, see the [Standard ML track's `test.yml` workflow](https://github.com/exercism/sml/blob/e63e93ee50d8d7f0944ff4b7ad385819b86e1693/.github/workflows/ci.yml).

```yml
name: sml / ci
on:
pull_request:
push:
branches: [main]
workflow_dispatch:
jobs:
ci:
runs-on: ubuntu-22.04
steps:
- name: Checkout code
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332
- run: docker pull exercism/sml-test-runner
- name: Run tests for all exercises
run: sh ./bin/test
```

0 comments on commit aec4d35

Please sign in to comment.