Independent workflows with their own branch protection checks

[gustaff-weldon]

TLDR:

According to docs:

If a workflow is skipped due to path filtering, branch filtering or a commit message, then checks associated with that workflow will remain in a "Pending" state.

If a job in a workflow is skipped due to a conditional, it will report its status as "Success". For more information see Skipping workflow runs and Using conditions to control job execution.

This is inconsistent and when the suggested solution (creating a duplicate always-successful workflow) is applied it is unclear which task wins if both workflows will trigger (for PRs touching code across multiple projects).

Why branch protection checks for workflows that do not run are not skipped?

How can we achieve truly independent workflows with their own related-checks?

A bit of background

We are using monorepo and would like to run separate workflows for projects that are independent from others.
Eg. given:

- api-docs
- infra
- frontend
- service-1
- service-2

We would like to have a separate project workflow eg. for api-docs, infra etc...

.github/workflows/api-docs.yml

name: api-docs

on:
  push:
    paths:
      - api-docs/**
  jobs:
    ....
   api-docs-workflow-results:
     needs:
        - api-docs-build
        - api-docs-test
        - ....

.github/workflows/infra.yml

name: infra

on:
  push:
    paths:
      - infra/**
  jobs:
    ....
   infra-workflow-results:
     needs:
        - infra-validate
        - infra-lint
        - ....

and so on, for other projects.

As you can see, each workflow will trigger only if certain project folders have been touched.

The problem

We would like to ensure PRs touching specific folders are not merged if relevant checks have not passed.

We have added branch protection rules and marked infra-workflow-results and api-docs-workflow-results as required for successful PR merge.

The issue here is the checks seem to be global and will be applied to all PRs, even if they do not trigger workflows that contain the step. And if the PR does not trigger a workflow, the check will be pending.

This problem is described in https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks and the suggestion is to add dummy workflows:

.github/workflows/api-docs-dummy.yml

on:
  push:
    paths-ignore:
      - api-docs/**

jobs:
  api-docs-workflow-results:
    steps:
      - run: 'echo "No build required"'

This kinda works until developer will change more than one project (eg. both api-docs and infra) in one PR.

In such case both dummy and regular workflow triggers and then it is unclear which of the api-docs-workflow-results tasks is used for the GH check (imagine the actual api-docs build fails, but the always-successful one does not).

We would like to have a clear separation between those builds, perhaps skipping the checks if the workflow they come from did not trigger.

Perhaps there's another way.

[gustaff-weldon]

@jenschelkopf does this fall under your umbrella maybe?

[henrygriffiths]

Hi @gustaff-weldon,

Admittedly, these solutions are a bit clunky, and it doesn't solve the root of the problem, but hopefully either one (or maybe even both) are useful for resolving the issue.

For several of the checks we use in my org, we rely on conditionals based on file paths for some singular checks within workflows (aka don't bother to run linting if only specific files were changed, but always run a build, etc). Here's an abbreviated version of the command we run:
curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/jq -r '.[] | .filename' | grep '^api-docs/' | wc -l
However, as the github.event.pull_request.number is only available when the check is triggered by a pull request check, additional logic would have to be put in place to grab the PR number. For checks on push running curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }}/commits/${{ github.event.head_commit.id }}/pulls | jq -r '.[0].number' gets the/a pr number.

Solution 1:

Using the 'feature' of skipped checks counting as passing, you could use that as follows (based on the workflows you provided above):
.github/workflows/api-docs.yml

name: api-docs

on:
  push:
jobs:
  api-docs_filepath_check:
    outputs:
      count: ${{ steps.set_count.outputs.count }}
    steps:
      - name: Get PR Num
        run: echo "pr_num=$(curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }}/commits/${{ github.event.head_commit.id }}/pulls | jq -r '.[0].number')" >> $GITHUB_ENV
      - name: Get Files (PR)
        if: env.pr_num != 'null'
        run: echo "count=$(curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }}/pulls/${{ env.pr_num }}/files | jq -r '.[] | .filename' | grep '^api-docs/' | wc -l)" >> $GITHUB_ENV
      - name: Get Files (Commit)
        if: env.pr_num == 'null'
        run: echo "count=$(curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }}/commits/${{ github.event.head_commit.id }} | jq -r '.files.[] | .filename' | grep '^api-docs/' | wc -l)" >> $GITHUB_ENV
      - run: echo '::set-output name=count::${{ env.count }}'
  api-docs:
    needs: [api-docs_filepath_check]
    if: needs.api-docs_filepath_check.outputs.count > 0
    steps:
      - run: echo "Found api docs changes"

    # add the following to subsequent jobs
    needs: [api-docs]

  api-docs-workflow-results:
    needs:
      - api-docs
      - api-docs-build
      - api-docs-test
      - ....

... and so on for each workflow.

Notes:

• I have not noticed any issue with the pr number being grabbed based on the first result from the pulls api endpoint, but it may cause issues if multiple PRs are open for the same commit or branch.
• The files api endpoint has a limit of 300 files (3000 with pagination, which would result in a more complicated command).
• Once files in a specific project/folder have been changed, said checks will always run on the PR, regardless if subsequent commits change these files, which may result in performance 'issues'.

Solution 2

Alternatively, using the ability to post custom status checks:
.github/workflows/api-docs.yml

name: api-docs

on:
  push:
    paths:
      - api-docs/**
jobs:
  # jobs to run if api docs changed

.github/workflows/api-docs-dummy.yml

name: api-docs-dummy

on:
  push:
    paths-ignore:
      - api-docs/**

jobs:
  api-docs-dummy:
    steps:
      - run: echo 'Api Docs Not Changed'

.github/workflows/api-docs-status.yml

name: api-docs-status

on:
  workflow_run:
    workflows: ['api-docs', 'api-docs-dummy']
    types:
      - completed
jobs:
  post-status:
    steps:
      - name: Post Status (Real)
        if: github.event.workflow_run.name == 'api-docs'
        run: curl -X POST -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }}/statuses/${{ github.event.workflow_run.head_commit.id }} -d '{"state": "${{ github.event.workflow_run.conclusion }}", "context": "Workflow / api-docs", "target_url": "${{ github.event.workflow_run.html_url }}"
      - name: Post Status (Copy)
        if: github.event.workflow_run.name == 'api-docs-dummy'
        run: curl -X POST -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }}/statuses/${{ github.event.workflow_run.head_commit.id }} -d '{"state": "$(if [[ $(for i in $(curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" -s https://api.github.com/repos/${{ github.repository }}/commits/${{ github.event.workflow_run.head_commit.id }} | jq -r '.parents[] | .sha') ; do curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" -s https://api.github.com/repos/${{ github.repository }}/commits/$i/statuses | jq -r '[.[] | select(.context == "Workflow / api-docs")][0].state' ; done | grep -v 'success' | wc -l) -eq 0 ]] ; then echo "success" ; else echo "failure" ; fi)", "context": "Workflow / api-docs", "target_url": "${{ github.event.workflow_run.html_url }}"

... and so on for each workflow.
Notes:

• The status workflow could be concatenated down using a bit more logic so you don't need 3 workflows per project/watched folder, just using the github.event.workflow_run.name in the context field and jq command, and endsWith(github.event.workflow_run.name, '-dummy') in the if statements.
• The status workflow relies off of being triggered off of either the dummy workflow (required) or the actual workflow.
  • If using the actual workflow, the output of the whole workflow is copied - this is the equivalent of making every job in the workflow required (so if some jobs fail, but PRs are allowed to be merged regardless, this will break that)
  • If using the dummy workflow, the status from the previous commit is copied (or, in the case of a merge commit, any non success status from any parent commit is a failure, otherwise success)
• The status relies on a 'bump start' the first time it is deployed, since it grabs the status off of a previous commit: the status needs to be either manually posted to the previous commit, or a file from each one of the projects needs to be changed so the real status workflow runs.
• A race condition can occur, where in a second commit is pushed and has its checks completed (such as the dummy workflow running), therefore resulting in no status found, which results in a failure status being posted.
  • Using the concurrency groups feature may resolve this, depending on overall check setup

As I haven't extensively tested these solutions, and I don't know what your whole check suite looks like, I have included some caveats in the notes below each solution. I personally would use the first solution, as it's a bit more simple, and checks could easily be set to run if the PR includes over 300 (or 3000, with pagination) changed files, but given the requirement of adding a job to every subsequent job's needs section, it is a bit clunky depending on the complexity of the workflow.

Hopefully this is useful!