Extract distributed backend and make it pluggable

[Jeffwan]

Hi community, I am new to NVFlare. I am curious whether NVFlare could build a generic resource backend and user can plug different resource managers?

We use Ray(https://www.ray.io/) as our resource layer to run big data and machine learning jobs. Trying to see if we can run NVFlare on Ray as well.

Hope to get more clues from the community! Thanks!

[YuanTingHsieh]

Hi @Jeffwan ,

Thanks for the question!

There are two aspects here.

NVFlare now DOES support different resource managers.
But we leave it to the users to write their own custom resource managers/consumers and specified required keys in "meta.json" in their jobs.
This is doable as of now.

You do bring up an interesting point, "Generic resource backend".
If I understand correctly, this means that NVFlare needs to have a "template" or "protocol".
Saying hey each of the ResourceManager/Consumers must support the common resource backend like: "CPU", "CPU memory", "GPU", "GPU memory".
And then these keys can be fixed in meta.json (and for the init function for those resource managers/consumers)
I think this common backend/template/protocol is also bring up sometimes ago by @nvidianz and @chesterxgchen

[chesterxgchen]

Both Ray and NVFLARE has its own communication protocols. Not sure what’s your thoughts on using ray and Nvflare together. Resource manager may not be the only integration points.

[Jeffwan]

Great. I think that's exact what I want. Could you share some guidance on how to bring user's own executor for task training? I have no idea how to plug to different envs for training yet.

[chesterxgchen]

Sorry for the delayed response, I forgot about this thread. You can take a look at the code, where several exectors are defined
https://github.com/NVIDIA/NVFlare/tree/dev/nvflare/app_common/executors
one is for used for Machine learning : https://github.com/NVIDIA/NVFlare/blob/dev/nvflare/app_common/executors/learner_executor.py
Some is used for Statistics calculation: https://github.com/NVIDIA/NVFlare/blob/dev/nvflare/app_common/executors/statistics/statistics_executor.py

As long you implement the Executor interface, you can do whatever your want.

[chesterxgchen]

the customized executor, for example in my_executor.py can be placed in "custom" directory.

[Jeffwan]

I will give it a try and report back my testing result.

[YuanTingHsieh]

another custom executor example: https://github.com/NVIDIA/NVFlare/blob/dev/examples/xgboost/tree-based/job_configs/bagging_base/app/custom/higgs_executor.py