Cifar10-example Provisioning

[LeandroDiL]

Hi!
I'd like to execute the cifar10 example in a real world scenario. I have an host, which IP is A, that is the host from which I do the provision and generate the folders for client, admin... and I want it to be the server, and another 2 hosts (Virtualbox VMs),with IP B and C, and I want them to be the clients.
Now, what do I have to write exactly into the file project.yml in order to compile it exactly?

Here it's my project.yml:

When I try to run the start.sh file in the local host machine with the command: bash nvflare_project/cifar10-sim/workspace/nvflare/prod_00/192.168.56.1/startup/start.sh the server start and I can connect to it running the command bash nvflare_project/cifar10-sim/workspace/nvflare/prod_00/site-a/startup/start.sh e bash nvflare_project/cifar10-sim/workspace/nvflare/prod_00/site-a/startup/start.sh from virtual machines. In the server terminal I obtain something like this:

The question is: since after the connection of the 2 clients nothing happened, how can I run the program now that both clients are connected to the servers? I already set the number of clients equal to 2 in the "meta.json" file and the "config_fed_server.json" file since this could be an issue.
I think the problem could be that I need an admin entity in my project.yml file. Something like:

- name: super@test.org
    type: admin
    org: nvidia
    role: project_admin

But I don't know which email should I use, how can I use the server to be also the admin or if is really necessary to have ad admin role.
Hope u can help me, ty in advance!

[YuanTingHsieh]

Hi @LeandroDiL

Thanks for your interest and the question!

You are correct, you want to add an admin entity in your project.yml.

Let's just walk through what to fill there one by one:

For name, you can just use any name you like, it just needs to be "xxx@yyy".
It does not even need to be a real email address.

For type, you want "admin" so this entity will leads to the generation of admin startup kits.

For org, I notice your other sites are all under "my_org", so we can just use that.

For role, "project_admin" means the super user, so that's use that for now.

So you can try this:

- name: leandrodil@gmail.com
  type: admin
  org: my_org
  role: project_admin

Then after you run provision, you should be able to connect to the server using something like:

bash nvflare_project/cifar10-sim/workspace/nvflare/prod_00/leandrodil@gmail.com/startup/fl_admin.sh

Then you can use a bunch of admin commands to interact with the systems:

https://nvflare.readthedocs.io/en/main/real_world_fl/operation.html

[LeandroDiL]

Hi @YuanTingHsieh ,
First of all, ty for the support. I tried what u said and I managed to connect to the admin console.
The problem now is that locally I run the project doing exactly what is written in the github docs here: https://github.com/NVIDIA/NVFlare/blob/2.3/examples/advanced/cifar10/cifar10-sim/README.md. Basically I do the following commands after the setup of the environment:

• export PYTHONPATH=${PWD}/..
• ./prepare_data.sh
• ./run_simulator.sh cifar10_fedavg 1.0 8 8

These commands are not possible from the admin console. So the question is: how can I run the project if I have these limitations? There's a way to do the same thing with other commands? (the submit_job job_folder command is not helpful since he gives me always the error "is not a valid folder") Ty in advance!

[YuanTingHsieh]

@LeandroDiL

Hello, thanks for following up!

For the first two steps, they should be done in each client site.
The third step is for running locally using simulator, since we are trying in real world scenario we DO NOT need the third step.

Please make sure you follow every step in https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/cifar10-real-world#readme first.

---

After you done that, then we can continue on your custom project.yml.

1. On EVERY CLIENT SITE (site-a and site-b in your case), you need to run the following steps:

   • export PYTHONPATH=${PWD}/..: this is to make sure that the python script can find the codes in https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/pt
   • ./prepare_data.sh: this is to make sure each client site has training/validation data.
   • These steps need to be run in regular console/terminal, NOT inside admin console.

2. Then on any machine with NVFlare installed, you:

   • Get the NVFlare examples by git pulling and checkout to branch 2.3
   • Copy the admin startup kits here
   • Put those examples inside the "transfer" folder
   • Run the Admin console and use "submit_job" command to submit

The NVFlare system looks like this:

                                                 <-----> NVFlare client
Admin Console   <------>    NVFlare server       <-----> NVFlare client
                                                 <-----> NVFlare client

[LeandroDiL]

Ty again for the support.
Unfortunately I coudn't make it work so I tried the real-world example following every step in the official github readme branch 2.3.
The only difference between the example provided here : https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/cifar10-real-world and my example is that I want two clients and I don't have a GPU. So for that reason I changed the avaible clients in the meta.json file and in the config_fed_server.json file, setting also the number of GPU avaible to 0.
When I start the FL system using the command ./start_fl_secure.sh 2 I got :

The question is : where can I see the running log of the example like in the cifar10-sim example? Because doing so its like I launched something that didn' give me error but also it's like doing nothing. The FL system it's not working!
Ty in advance like always.

P.S. Maybe I discovered the problem: after the clients are connected to the server and after u opened the admin console u need to copy the job folder in the admin console "transfer" folder so when u submit the job the admin entity can find the job and launch it giving an unique ID to it. I don't know if it is necessary only for me because I did something wrong but if it's not the case I suggest to update the wiki about that!

[YuanTingHsieh]

@LeandroDiL thanks for the update.

As described in this example: https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/cifar10-real-world
We ask users to use "./submit_job.sh [job] [alpha]" to submit the jobs.

If users instead using the admin console directly, then YES you are correct the jobs need to be first copied into the admin console startup kit's "transfer" folder.

[YuanTingHsieh]

Please let us know if you can run your experiment successfully, thanks!

[IsaacYangSLA]

After submitting the job from admin console, please run list_jobs to see the job id. To see the log.txt for that job in site-1, first run ls site-1 to find the folder name that matches the job id. Assume the job id is 63cd32e1-397f-4796-aa7e-dad5707f6a53 then cat site-1 63cd32e1-397f-4796-aa7e-dad5707f6a53/log.txt will show the log.txt.

[YuanTingHsieh]

close as no activity. @LeandroDiL feel free to open when you got new information/questions!