Support submitting jobs with the file structure seen in examples

[parkeraddison]

Hi NVFlare team!

Recently we refactored our own codebase to be a lot more modular and maintainable when working with multiple different jobs and jobs with multiple apps (e.g. some jobs use different workflows, some apps use different data, networks, or other FL components). The way we accomplished this was by using a file structure similar to that in your examples/:

.
├── config
│   ├── simple-job
│   │   ├── meta.json
│   │   └── simple-app
│   │       └── config
│   │           ├── config_fed_client.json
│   │           └── config_fed_server.json
│   └── complex-job
│       ├── meta.json
│       ├── A-app
│       │   └── config
│       │       └── config_fed_client.json
│       ├── B-app
│       │   └── config
│       │       └── config_fed_client.json
│       └── server-app
│           └── config
│               └── config_fed_server.json
└── custom
    └── src
        ├── learners
        │   └── common_learner.py
        ├── networks
        │   ├── common_networks.py
        ├── utils
        │   ├── common_utils.py
        └── workflows
            └── complex_workflow.py

This way, the config of the various jobs can point to shared modules making it easy to maintain a single file rather than duplicating all custom/ files for a new job. In the event a job needs a different module, we write it as part of our src so that we can rely on perhaps shared utilities or re-use the workflow in a future job. If we improve a module such as the algorithm used in our learner, all jobs can benefit from this enhancement. And we stay DRY ;)

Issue

This structure is fun and works in the examples, but it doesn't work when submitting a job in a real deployment. If you copy config/ and custom/ to the admin's transfer/ directory then submit e.g. submit_job config/simple-job, only simple-job/simple-app/config/ gets transferred to the participants, so no custom modules are able to be loaded.

admin/transfer/
├── config
│   ├── simple-job
│   │   └── simple-app
│   │       └── config
│   │           └── ...
│   └── complex-job
│       └── ...
└── custom
    └── src
        └── ...

site1/jobId/app_site1/
└── config
    └── ...

The reason this structure works in the examples is because (I think) all of the processes are launched from a terminal with the src/ directory above accessible from the pwd, rather than relying on the admin transferring that custom code.

Workaround

The workaround we're currently using is to additionally copy custom/ into admin/transfer/simple-job/simple-app/ before submitting -- but this is a bit tedious for something like complex-job in the example above where we'd need to copy custom/ into all of the three apps (A-, B-, and server-) before submitting. Plus it leaves some more room for human error, stale files, etc.

cp -r config/ admin/transfer/
cp -r custom/ admin/transfer/config/simple-job/simple-app/
> submit_job config/simple-job

Solution

Is there a solution to allow a maintainable file structure like the above example to be used in real deployment? I.e. by copying both config/ and custom/ to the admin's transfer/ directory then having all necessary custom modules sent to the participants when submitting a job?

Some ideas I had were perhaps:

1. Specify custom/ source code paths in the meta.json
   • Similar to deploy_map, we could specify different filepaths (relative to admin/transfer/) to be sent to specific clients
2. Admin always sends any custom/ directory at the root of admin/transfer/
   • Quick and easy, but less customizable and possibly you might transfer something unintended that could reveal information about one participant to another
3. Package up all source code into a Python package and require all participants to install it
   • It works, but this wouldn't be ideal IMO since it would have the same concerns as always sending the entire custom/ directory, and require additional dev/IT work to set up all participants

Gotta admit that developing with this modular file structure is honestly a joy and lets us harness the full power of your config parsing, components, nested components, etc -- so I'm hoping this will be supported moving forward :D Lmk what you think and if I can help at all!

[chesterxgchen]

Can you simply set symbolic link of customer code directory to each job’s customer folder

this way, you need one time setup and the code will reflect the latest changes. No need to anything else

[parkeraddison]

I really like this idea! I gave it a try, only the symlink file was copied instead of recursing through the link (e.g. there was a plain file named "custom" which was copied to all participants).

Probably just has to do with not following symlinks somewhere, I may poke around to find out where.

[parkeraddison]

Here's the fix! In nvflare.fuel.utils.zip_utils.get_all_file_paths() we can just add an argument to os.walk to use followlinks=True.

NVFlare/nvflare/fuel/utils/zip_utils.py

Lines 60 to 78 in 9d8bbd9

	def get_all_file_paths(directory):
	"""Gets all file paths in the directory.
	Args:
	directory: directory to get all paths for
	Returns:
	A list of paths of all the files in the provided directory
	"""
	file_paths = []
	# crawling through directory and subdirectories
	for root, directories, files in os.walk(directory):
	for filename in files:
	file_paths.append(normpath_for_zip(os.path.join(root, filename)))
	for dir_name in directories:
	file_paths.append(normpath_for_zip(os.path.join(root, dir_name)))
	return file_paths

Line #L72 changes to

for root, directories, files in os.walk(directory, followlinks=True):

After this, the file upload from admin/transfer/ to the various participants will include contents of soft-linked directories! Nothing else needed to be modified.

I'm currently using this modification on my end. Off the top of my head there may be need for an additional test case (e.g. adding a symlink in zip_utils_test.py) and safeguarding logic (e.g. to prevent accidental recursion) before turning this into a PR.

[chesterxgchen]

Pre installation of the custom code to clients and set python path is also an option. Some clients prefer to use that way in production as this reduce the potential risk of unknown code uploaded to their site

[parkeraddison]

Agreed pre-installation is the right choice in some contexts, for prototyping and demonstrations it gets a bit tedious though

[chesterxgchen]

right, so people use symlink or PYTHONPATH during development and prototyping and then in production switch to pre-install

[YuanTingHsieh]

@parkeraddison thanks for the discussion.

You raise some good points/observations here.

Just a note here, we are also thinking if there is a better way to organize "examples" such that it is easier for people to deploy/run right away?

We are thinking of a more structural directory than what we already had here.
For example we could say we re-organize each example to have the following structure:

example_name/
    scripts/
        setup.sh (include the logic to set up data on each site)
    codes/
         all these custom codes falls into here
    config/
         the required configurations should be put in here

If we make these example folder structure more formalize and rigorous, then it is very natural for NVFlare to help run setup.sh and setup the link to these codes at deploy time.

You can also provide your opinions on what should be the structure and required files.
We will also take them into consideration when we decide the final structure.