Make EphemeralRunnerReconciler create runner pods earlier #3831
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tweaks the reconciler so that there are fewer requeues in the happy path, improving
runner startup time when dozens or hundreds of jobs are queued.
This is a pull request for the first commit mentioned in my comment at #3276 (comment).
It should fix, or at least alleviate #3276
How the improvement works
Let's simplify the ephemeral-runner reconciliation logic into generally 2 phases: (1) get token and (2) create runner pod.
This change improves the runner pod creation latency by making (2) happen earlier.
How it should work ideally
With three concurrent job startups, it should ideally be like this:
How it works today
With the unnecessary requeues in the happy path, it had to be like this:
You can see that the first two jobs take longer to start than the ideal scenario.
The more requeues in the happy path and concurrent N jobs you have, the longer the first N-1 jobs take to start.