Workflow fails when Dask worker memory budget is exceeded

[FyzHsn]

My Machine Learning workflow execution fails during the preprocessing stage of the workflow. Having started the covalent server with covalent start -d, I was able to check the logs covalent logs. The log files show that the reason behind the workflow failure was:

2022-09-14 21:22:51,192 - distributed.worker_memory - WARNING - Worker tcp://127.0.0.1:64888 (pid=45315) exceeded 95% memory budget. Restarting...

It seems like the issue is with Dask worker memory being exceeded. I have two main questions:

1. How can I get more details on the Dask workers?
2. How can I move forward so that the workflow completes successfully?

[venkatBala]

The Dask dashboard will help get more details on the cluster workers. You will however need to install bokeh. The following steps would be helpful

• If covalent is already running, the server needs to be stopped
• pip install bokeh
• Start covalent
• Check the Dask dashboard at localhost:8787 (In the workers tab you can inspect how much memory each Dask worker is running with)
• Secondly this information can also be retrieved via the covalent cluster CLI a.k.a covalent cluster --info

UI screenshots

[FyzHsn]

In order to resolve the Dask worker memory issue, we can reduce the number of workers and increase the amount of memory allocated. Previously, I had 8 workers with 1 GB allocated to each.

To ensure that the Dask workers have enough memory to complete the task, I restarted covalent with:

covalent start -n 4 -m "2GB" -d

This allocated 2 GB per worker which turned out to be enough memory to complete the task.

[venkatBala]

Another alternative here would be to dispatch the compute intensive parts of the workflow to AWS via the Batch executor where memory is not a constraint per say