Limit the number of CPU usage for MeiliSearch

[curquiza]

Based on meilisearch/meilisearch#1706

This would allow users to set the number of CPUs MeiliSearch can use.

Current workaround

Using cpulimit
https://unix.stackexchange.com/questions/84945/how-can-i-start-a-process-using-cpulimit

[igaul]

Hey thanks for opening this!

A few notes from looking:
Meilisearch has an IndexerOpts option for limiting cpus used for indexing used in update_handler which should then also respect a general setting.

I think it would be as simple as adding an env option to Opt, checking it and defaulting to num_cpus when creating actix server
Actix exposes this setting in HttpServer::workers, HttpServer::new(...).workers( number of workers ).disable_signals()

The only downside I can think of is more options.

[gmourier]

Hello @igaul and thanks for the feedback 👋

If I understand correctly you want to be able to specify to Actix to spawn less workers than the number of cores of the machine. Is that it?

May I ask why you want to do this? I am very interested to know why this need appears and if you have encountered any problem with the current behavior.

From your message on the initial issue, I understood that testing out several instances on the same machine creates hundreds of processes.

[igaul]

Hey @gmourier thanks for replying,
Yes I want to be able to set the number of workers instead of relying on cpu count - because I want to 😄. Its not an essential feature, but more of a convenience in my case (instead of limiting the resources) to simulate realistic/slower conditions. This might also cover edge cases such as shared infrastructure that does not accurately report cpus an application is allowed to use.
Thanks again.

[gmourier]

Thanks for your answer @igaul. At some point, we will define a config file to facilitate launching options for a MeiliSearch instance. This will be the occasion for us to make a pass on all the requests that could also become configuration variables and assess their relevance.

[gmourier]

Hi @igaul 👋

The v0.27 will introduce a --max-indexing-threads CLI option that would permit you to limit the number of threads the indexing process is using.

This discussion remains open since the question of the number of cpu that actix will be able to use has not been addressed.