Skip to content

Commit

Permalink
[WFRunner] Estimate resources dynamically
Browse files Browse the repository at this point in the history
* introduce functionality to update resource estimates based on values
  which are collected during monitoring
  (the same that go into pipeline_metric_<pid>.log)

* disentangle resource management from WorkflowExecutor

* manage resources via ResourceManager object
  * memory and cpu
  * semaphores
  * number of parallel tasks to be launched
  * niceness

* main interfaces of ResourceManager
  * ok_to_submit
  * book
  * unbook
  * add_monitored_resources

* group corresponding tasks together to be able to update resource
  estimates for upcoming tasks

* TaskResources as container of a task's resource estimate
  holding
  * resource estimates (initial and new estimates)
  * references to Semaphore object (optional
  * reference to list of related tasks (optional)

* make sure, final niceness assigned to a launched process is the same
  that the TaskResources hold for that process

* enable functionality with --dynamic-resources

* add more comments to classes and functions

* update --cgroup option to require entire path
  --> handle both cgroups v1 and cgroups v2

(cherry picked from commit ef0e84d)
  • Loading branch information
Benedikt Volkel authored and alcaliva committed Aug 23, 2024
1 parent 547ffd0 commit a990107
Showing 1 changed file with 541 additions and 231 deletions.
Loading

0 comments on commit a990107

Please sign in to comment.