[Question] Multiple Instances of IsaacLab

[SrinidhiBharadwaj]

Is there a way to use python multiprocessing to spawn multiple instances of isaaclab/isaacsim (GPU memory allowing)?

I have a use case where I need to scale up my data generation process to 1000s of objects. Currently my script creates an environment and runs the test for each object. GPU utilization is roughly 10% (20% on a single GPU, my workstation has 2 GPUs). If I spin up python multiprocessing to test >1 object at a time, the performance of drops considerably. [Single object without multiprocessing takes 80s and if multiprocessing is used for multiple objects, each objects takes ~500s].

I was wondering if there is a way to set this up without taking a performance hit so I don't have to wait 200 days for my data? And if there is a way to run this on multiple GPUs, that would be helpful as well. I tried torch.nn.DataParallel which did not help.

Thanks in advance.

[kellyguo11]

Hi, you can try limiting the CPU threads for each process to make sure that the processes are not fighting each other for resource. An example of doing is can be found here.

[glvov-bdai]

Hi, you can use the Ray integration for running many parallel experiments on different GPUs. However, these experiments have to be truly parallel and cannot depend on each other.

#1301

[SrinidhiBharadwaj]

Thank you for the response. This is great as my ultimate goal is to use Ray. I can use it on AWS with these changes right? Should I stick to Isaac Automator? We have a custom workflow to set up AWS instances and setting up Automator requires privileged access from system admins.

[glvov-bdai]

I recommend using KubeRay on EKS with my custom dockerfile as there is some path trickery to get Ray to work. I am unfamiliar with Automator. Out of the box, only GCP works, as my company uses GCP. However, if you make a kuberay.yaml.jinja file like https://github.com/isaac-sim/IsaacLab/blob/5b99aec97974167b7c4627bb8ea15887f78cd5c1/source/standalone/workflows/ray/cluster_configs/google_cloud/kuberay.yaml.jinja for AWS EKS, the rest should work.

For more instructions see

https://github.com/isaac-sim/IsaacLab/blob/5b99aec97974167b7c4627bb8ea15887f78cd5c1/docs/source/features/ray.rst

I'd also be happy to help get AWS EKS isaacray up and running. If someone gives me access to an AWS EKS server with an RTX GPU nodepool I'd be happy to write the additionally kuberay.yaml.jinja for AWS

[SrinidhiBharadwaj]

Cool, thanks, I will keep you posted on it.