Github 3.14.1 compatiblity issue - unable to register runners

[kb-med]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.3

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Create Github Enterprise instance with version 3.14.1
2. Install gha-runner-scale-set-controller-0.9.3
3. Generate token on Github for authentication
4. Configure secret with token
5. Install gha-runner-scale-set-0.9.3

Describe the bug

Arc controller cannot register runner with Github Enterprise instance with following errors

On existing instance we experienced these errors

2024-10-07T08:07:29Z INFO listener-app app initialized 2024-10-07T08:07:29Z INFO listener-app Starting listener 2024-10-07T08:07:29Z INFO listener-app refreshing token {"githubConfigUrl": " https://github.enterprise.com/team"} 2024-10-07T08:07:29Z INFO listener-app getting runner registration token {"registrationTokenURL": " [https://github.enterprise.com/api/v3/orgs/team/actions/runners/registration-token"}](https://github.enterprise.com/api/v3/orgs/team/actions/runners/registration-token%22%7D) 2024-10-07T08:07:30Z INFO listener-app getting Actions tenant URL and JWT {"registrationURL": " https://github.enterprise.com/api/v3/actions/runner-registration"} 2024/10/07 08:07:30 Application returned an error: createSession failed: failed to create session: actions error: StatusCode 404, AcivityId "5be3369d-4ae3-479b-8a85-489c7e9c2986": GitHub.Actions.Runtime.WebApi.RunnerScaleSetNotFoundException, GitHub.Actions.Runtime.WebApi: No runner scale set found with identifier 12.

After reinstallation now we see something like this

2024-10-08T09:22:25Z INFO EphemeralRunner Creating new ephemeral runner registration and updating status with runner config {"version": "0.9.3", "ephemeralrunner": {"name":"def-runner-set-6pqwm-runner-vbm68","namespace":"eks-arc-dev"}} 2024-10-08T09:22:25Z INFO EphemeralRunner Creating ephemeral runner JIT config {"version": "0.9.3", "ephemeralrunner": {"name":"def-runner-set-6pqwm-runner-vbm68","namespace":"eks-arc-dev"}} 2024-10-08T09:22:25Z INFO actions-clients retrieve actions client {"githubConfigURL": "https://github.enterprise.com/team", "namespace": "eks-arc-dev"} 2024-10-08T09:22:25Z INFO actions-clients using cache client {"githubConfigURL": "https://github.enterprise.com/team", "namespace": "eks-arc-dev"} 2024-10-08T09:22:25Z ERROR Reconciler error {"controller": "ephemeralrunner", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "EphemeralRunner": {"name":"def-runner-set-6pqwm-runner-vbm68","namespace":"eks-arc-dev"}, "namespace": "eks-arc-dev", "name": "def-runner-set-6pqwm-runner-vbm68", "reconcileID": "2a8ac671-e305-4c32-9395-0a8305e6b86d", "error": "failed to generate JIT config with Actions service error: actions error: StatusCode 404, AcivityId \"5b380ee9-4ae3-479b-8a85-489c7e9c2986\": GitHub.Actions.Runtime.WebApi.RunnerScaleSetNotFoundException, GitHub.Actions.Runtime.WebApi: No runner scale set found with identifier 12."} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227

Describe the expected behavior

Successful registration of dynamic ARC Runners

Additional Context

#Controller Values
#Controller pod only requires changes to image hosting and image pull secret
image:
  repository: registry.internal.com/team/tools/gha-runner-scale-set-controller
imagePullSecrets: 
  - name: img-secret

--------------------------
#Runners set values
#Setup of github repo/space
githubConfigUrl: "https://github.enterprise.com/team"
githubConfigSecret: gha-token

#Setting minimal and maximal amount of runners.
#Minimal can be 0 but that incease amount of wait time for run as pod have to be spinned up on every pipeline run
#Maximal limits amounts of runners running at same time
minRunners: 2
maxRunners: 5

#Worker pod reconfiguration to use internally hosted image and our image pull secret
template:
  spec:
    containers:
      - name: runner
        image: registry.internal.com/team/tools/actions-runner:1.1
        command: ["/home/runner/run.sh"]
    imagePullSecrets:
    - name: img-secret

Controller Logs

`2024-10-07T08:07:29Z	INFO	listener-app	app initialized
2024-10-07T08:07:29Z	INFO	listener-app	Starting listener
2024-10-07T08:07:29Z	INFO	listener-app	refreshing token	{"githubConfigUrl": "
https://github.enterprise.com/team"}
2024-10-07T08:07:29Z	INFO	listener-app	getting runner registration token	{"registrationTokenURL": "
[https://github.enterprise.com/api/v3/orgs/team/actions/runners/registration-token"}](https://github.enterprise.com/api/v3/orgs/team/actions/runners/registration-token%22%7D)
2024-10-07T08:07:30Z	INFO	listener-app	getting Actions tenant URL and JWT	{"registrationURL": "
https://github.enterprise.com/api/v3/actions/runner-registration"}
2024/10/07 08:07:30 Application returned an error: createSession failed: failed to create session: actions error: StatusCode 404, AcivityId "5be3369d-4ae3-479b-8a85-489c7e9c2986": GitHub.Actions.Runtime.WebApi.RunnerScaleSetNotFoundException, GitHub.Actions.Runtime.WebApi: No runner scale set found with identifier 12.`

-------------------------------------------

`2024-10-08T09:22:25Z	INFO	EphemeralRunner	Creating new ephemeral runner registration and updating status with runner config	{"version": "0.9.3", "ephemeralrunner": {"name":"def-runner-set-6pqwm-runner-vbm68","namespace":"eks-arc-dev"}}
2024-10-08T09:22:25Z	INFO	EphemeralRunner	Creating ephemeral runner JIT config	{"version": "0.9.3", "ephemeralrunner": {"name":"def-runner-set-6pqwm-runner-vbm68","namespace":"eks-arc-dev"}}
2024-10-08T09:22:25Z	INFO	actions-clients	retrieve actions client	{"githubConfigURL": "https://github.enterprise.com/team", "namespace": "eks-arc-dev"}
2024-10-08T09:22:25Z	INFO	actions-clients	using cache client	{"githubConfigURL": "https://github.enterprise.com/team", "namespace": "eks-arc-dev"}
2024-10-08T09:22:25Z	ERROR	Reconciler error	{"controller": "ephemeralrunner", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "EphemeralRunner": {"name":"def-runner-set-6pqwm-runner-vbm68","namespace":"eks-arc-dev"}, "namespace": "eks-arc-dev", "name": "def-runner-set-6pqwm-runner-vbm68", "reconcileID": "2a8ac671-e305-4c32-9395-0a8305e6b86d", "error": "failed to generate JIT config with Actions service error: actions error: StatusCode 404, AcivityId \"5b380ee9-4ae3-479b-8a85-489c7e9c2986\": GitHub.Actions.Runtime.WebApi.RunnerScaleSetNotFoundException, GitHub.Actions.Runtime.WebApi: No runner scale set found with identifier 12."}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227`

Runner Pod Logs

N/a, is not spawning

[kb-med]

Update, issue is caused by previous registration details hanging and not being removed.
Helm uninstall also does not clean-up those hanging objects.

[meiswjn]

Yup, this also happens when updating the Kubernetes Secret that should be used for authenticating. It appears to be a general problem with Custom Resources not being cleaned up.

[gleruzh]

Is there a good way to fix?
something less than "completely remove charts, secrets, custom resources and crd" and reinstall, which works of course.

[meiswjn]

Removing the finalizers actually worked for us.

Its a bit tedious because you have to do it for every CRD (and also service accounts, apparently - everything that has stuck finalizers). This is how you do it with a command:
kubectl get AutoscalingListeners -n <namespace> -o name | xargs -I {} kubectl patch {} -n <namespace> -p '{"metadata":{"finalizers":null}}' --type=merge
This command sets the finalizers to an empty array. I do not know if this brings any technical issues. What it doesnt do is checking if the resource is still needed... to check this, some resources support "--field-selector=status.phase==Succeeded", which is the state when it should actually be removed.

A simple way to check if you caught all of them: It seems like you are not be able to (force) delete the namespace, unless all finalizers are removed.

[gleruzh]

yes, I went the same way - patch custom resources with empty finalizers.