Ephemeral Pool Config and Idle Runner Scale Down Conflicting

[seanuleh]

The documentation seems to indicate that if using ephemeral runners then the pool config should be set and not to adjust the idle config, but I'm not so sure this resulting behaviour is desirable.

Using the following arguments:

      pool_config = [{
          size                = 1
          schedule_expression = "cron(0/5 * * * ? *)"
       }]
      idle_config = []
      scale_down_schedule_expression = "cron(* * * * ? *)"

It seems that the pool config only scales up instances and the idle config always scales down instances. When using ephemeral runners, these two configurations are always competing and results in idle ephemeral runners continuously being terminated. Is this the intended behaviour? Wouldn't it be best to keep the idle runners running instead of continuously cycling them?

If my intention is to have the idle runners remain, should I also configure the idle_config then to not teardown my idle ephemeral instances?

[LXXero]

i've been noticing this same thing - i've just set a bogus date on the scale down expression to disable it entirely, like simply setting the year to something in the past. I didn't think about using idle_config in addition to pool_config, but, I was also chasing some issues where jobs were getting terminated prematurely, so I had been trying to rule it out as a factor entirely.

when using a pool, i'd expect it to basically consume the pool and then append to the requested capacity, so if your pool is 10, you request 15 nodes, it consumes 10, requests 5 extra, the pool then stands up another 10. ideally, i shouldn't end up with any extra nodes - and i should have nothing to scale down, the runners all terminate themselves once the actions are finished.

this also means if you're using a pool I'd expect to need delay_webhook_event set to something greater than 0 - otherwise it may start adding additional runners to the queue before it ever attempts to consumes a pool worker.

edit: i ended up having to turn scale down back on and using idle_config afterall, as i was getting orphaned instances sticking around on occasion, which i'm still not sure why, but, I re-enabled it every 10 minutes, I set it slightly higher than my pool count, and I set my minimum running time to 60 minutes so it's not just tearing down stuff constantly. I'm still testing, but it seems to be working much better now.

[npalm]

The idle config is meant for non empheral runners, to keep some instances idle. The pool is meant for ephemeral runners to keep a pool start builds fast, and ensure if a message is lost the pool runners can take care. We run with a very small pool. And with around 150 ephemeral runners.

[LXXero]

the problem I found was - without the idle_config also setup to maintain some idle workers, the scale down job will be actively fighting against the pool runners, shutting them down simply because they hit the minimum_running_time_in_minutes, but this creates a lot of runner churn and ends up creating periods where there's no available runners, because the current set of idle runners hit minimum_running_time_in_minutes, scale down kills them, and then you have to wait for the pool interval to setup the pool again, and whatever time your AMI takes to start.

i would rather my ephemeral pool runners just stay idle until consumed, so having both of these options together seems to actually make sense. It would be more ideal if the scale-down job could simply ignore the pool runners automatically, without needing to have two separate sets of cron expressions.

in our setup, the pool is currently 25 and I have been seeing up to ~170 ephemeral runners. I have a few workflows that start up 10-20 jobs, so i wanted my pool to be larger than this.

[npalm]

Clear, we let them just die, but understand your case. First thought would that we could add a label, and exempt instances with a certain label form being terminated.

An idea could be to define a label for instances created as part of the pool. And optional pass the label to the scale down lambda to avoid termination. But we cannot control which instance will take up a job. So by having a pool, it could be that instances in the pool run the job, or instances created by an event. The instances created by an ievent will be stil terminated.

We are open to any more advanced design of the pool. But the current design is the one we have created at the time we implemented ephemeral runners. Would you have time to propose a PR?

[LXXero]

I think it's probably reasonable that if a non-pool runner was stood up from an event and not by the pool lambda, that it could get terminated if not consumed before all the pool runners were. Although, to avoid that, I was purposely setting the idle config a little higher than the pool, and over time they'd generally all get consumed before that happens.

Short of ensuring you never stand up more worker nodes than you need, I think my ideal setup would be for the cleanup job to basically work similarly to the idle config for the pool runners, but to automatically just go by the number of pool workers instead of needing a separate configuration, and ideally, there'd be an adjustable burst limit above that, perhaps as an extra value specified along the pool size, so that you could purposely leave X amount of extra runners that were stood up by events, in addition to the pool size.

I think that might be more useful in practice than trying to only terminate the ones stood up by the pool lambda This would give it a much higher chance for that worker to get consumed during the next big workflow run. Likewise, if this burst limit can follow the pool size, then it could scale up/down along with it. I'm truthfully not familiar with the lambda side of the codebase, so I'm not sure I could commit to anything without some further investigation, but the current work-around seems to be "good enough", which always hurts priority on these things...