[Q&A] Client can't connect to server

[LSnyd]

Python version (python3 -V)

3.9

NVFlare version (python3 -m pip list | grep "nvflare")

2.4

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.4

Operating system

Ubuntu

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I want to run the real-world example of cifar on one on-premise machine by starting server, admin and clients from different terminal. The server runs and the admin connects, but there seems to be a problem with the client connection. I ran the same project in the same approach on my laptop and both clients and admins connected to the server. Does anyone have an idea why it doesn't work on the on-premise machine?

start fl because of no pid.fl
new pid 2765918
Waiting for SP....
2024-04-24 09:38:04,485 - CoreCell - INFO - site-1: created backbone external connector to grpc://localhost:8002
2024-04-24 09:38:04,485 - ConnectorManager - INFO - 2765918: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2024-04-24 09:38:04,487 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:64064] is starting
2024-04-24 09:38:04,989 - CoreCell - INFO - site-1: created backbone internal listener for tcp://localhost:64064
2024-04-24 09:38:04,990 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE grpc://localhost:8002] is starting
2024-04-24 09:38:04,991 - FederatedClient - INFO - Wait for engine to be created.
2024-04-24 09:38:05,097 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-04-24 09:38:05,097 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 N/A => localhost:8002] is created: PID: 2765918
2024-04-24 09:38:05,118 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 2765918
2024-04-24 09:38:05,118 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00002 Not Connected]
2024-04-24 09:38:06,122 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-04-24 09:38:06,122 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => localhost:8002] is created: PID: 2765918
2024-04-24 09:38:06,135 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 2765918
2024-04-24 09:38:06,135 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-04-24 09:38:08,139 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-04-24 09:38:08,139 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 N/A => localhost:8002] is created: PID: 2765918
2024-04-24 09:38:08,151 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00004 Not Connected] is closed PID: 2765918
2024-04-24 09:38:08,151 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00004 Not Connected]
2024-04-24 09:38:12,154 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-04-24 09:38:12,155 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 N/A => localhost:8002] is created: PID: 2765918
2024-04-24 09:38:12,168 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00005 Not Connected] is closed PID: 2765918
2024-04-24 09:38:12,168 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00005 Not Connected]
2024-04-24 09:38:12,169 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://localhost:8002] in 8 seconds
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/miniconda3/envs/nvflare/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/miniconda3/envs/nvflare/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/ha/dummy_overseer_agent.py", line 112, in _rnq_worker
    self._do_callback()
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/ha/dummy_overseer_agent.py", line 106, in _do_callback
    self._update_callback(self)
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/private/fed/client/fed_client_base.py", line 147, in overseer_callback
    self.set_primary_sp(sp)
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/private/fed/client/fed_client_base.py", line 362, in set_primary_sp
    return self.set_sp(self._get_project_name(), sp)
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/private/fed/client/fed_client_base.py", line 162, in set_sp
    self._create_cell(location, scheme)
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/private/fed/client/fed_client_base.py", line 220, in _create_cell
    raise RuntimeError(f"Failed to get engine after {time.time()-start} seconds")
RuntimeError: Failed to get engine after 15.003176212310791 seconds
2024-04-24 09:38:20,171 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-04-24 09:38:20,172 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 N/A => localhost:8002] is created: PID: 2765918
2024-04-24 09:38:20,185 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 2765918
2024-04-24 09:38:20,185 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00006 Not Connected]
2024-04-24 09:38:20,186 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://localhost:8002] in 10 seconds
2024-04-24 09:38:30,189 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-04-24 09:38:30,189 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 N/A => localhost:8002] is created: PID: 2765918
2024-04-24 09:38:30,201 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00007 Not Connected] is closed PID: 2765918
2024-04-24 09:38:30,202 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00007 Not Connected]
2024-04-24 09:38:30,202 - nvflare.fuel.f3.sfm.conn_manager - INFO - Retrying [CH00001 ACTIVE grpc://localhost:8002] in 10 seconds
The client could not register to server. 
2024-04-24 09:38:35,443 - MPM - ERROR - Execute exception: RuntimeError: Login failed.
2024-04-24 09:38:35,444 - MPM - ERROR - Traceback (most recent call last):
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
  File "/miniconda3/envs/nvflare/lib/python3.9/site-packages/nvflare/private/fed/app/client/client_train.py", line 120, in main
    raise RuntimeError("Login failed.")
RuntimeError: Login failed.

The this is the project.yml:

api_version: 3
name: example_project
description: NVIDIA FLARE sample project yaml file

participants:
  # change example.com to the FQDN of the server
  - name: localhost
    type: server
    org: nvidia
    fed_learn_port: 8002
    admin_port: 8003
  - name: site-1
    type: client
    org: nvidia
    # Specifying listening_host will enable the creation of one pair of
    # certificate/private key for this client, allowing the client to function
    # as a server for 3rd-party integration.
    # The value must be a hostname that the external trainer can reach via the network.
    # listening_host: site-1-lh
  - name: site-2
    type: client
    org: nvidia
  - name: admin@nvidia.com
    type: admin
    org: nvidia
    role: project_admin

# The same methods in all builders are called in their order defined in builders section
builders:
  - path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    args:
      template_file: master_template.yml
  - path: nvflare.lighter.impl.template.TemplateBuilder
  - path: nvflare.lighter.impl.static_file.StaticFileBuilder
    args:
      # config_folder can be set to inform NVIDIA FLARE where to get configuration
      config_folder: config

      # scheme for communication driver (currently supporting the default, grpc, only).
      # scheme: grpc

      # app_validator is used to verify if uploaded app has proper structures
      # if not set, no app_validator is included in fed_server.json
      # app_validator: PATH_TO_YOUR_OWN_APP_VALIDATOR

      # when docker_image is set to a docker image name, docker.sh will be generated on server/client/admin
      # docker_image:

      # download_job_url is set to http://download.server.com/ as default in fed_server.json.  You can override this
      # to different url.
      # download_job_url: http://download.server.com/

      overseer_agent:
        path: nvflare.ha.dummy_overseer_agent.DummyOverseerAgent
        # if overseer_exists is true, args here are ignored.  Provisioning
        #   tool will fill role, name and other local parameters automatically.
        # if overseer_exists is false, args in this section will be used and the sp_end_point
        # must match the server defined above in the format of SERVER_NAME:FL_PORT:ADMIN_PORT
        # 
        overseer_exists: false
        args:
          sp_end_point: localhost:8002:8003

  - path: nvflare.lighter.impl.cert.CertBuilder
  - path: nvflare.lighter.impl.signature.SignatureBuilder

[YuanTingHsieh]

Hi @LSnyd

Thanks for the interest and question!

This is a infrastructure setup problem that is not specific to NVFlare.
My suggestions below:

1. You need to figure out the FQDN of this on-premise machine, whether you are using AWS, GCP, Azure, etc there should be a way to find it, for example I open an instance in Oracle Cloud, and my name is "vcn123456.oraclevcn.com"

2. Then you need to modify the server name in your "project.yml" from "localhost" to this name, for example:

participants:
  # change example.com to the FQDN of the server
  - name: vcn123456.oraclevcn.com
    type: server
    org: nvidia
    fed_learn_port: 8002
    admin_port: 8003

3. You need to make sure your on-premise machine open ports of 8002 and 8003 and can be accessed from outside, check networking documentation of those cloud providers or check with your IT.

4. You need to make sure your client can find the server using this name, one trick to do this is to edit the "/etc/hosts" on the client side, for example, assume my machine "vcn123456.oraclevcn.com" is of public IP address "123.222.222.222"
   I will add an entry (sudo vim /etc/hosts):

123.222.222.222 vcn04231914.oraclevcn.com
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost

@IsaacYangSLA please help comment if you have other good practices, thanks!

[LSnyd]

Hi @YuanTingHsieh,

thanks a lot for your help. However, I am not yet trying to access the on-premise machine from the outside. I opened 4 different terminals for server, admin, client 1 and client 2 on the same machine. Hence, client 1 and client 2 should be able to connect to the server via localhost and the opened ports, right?

[chesterxgchen]

since you are running everything locally on the same host. why not trying to use POC command, it also generate project.yml file or take your project.yml and setup the clients and server for you locally.

nvflare poc prepare -n 2

[YuanTingHsieh]

@LSnyd thanks for the reply.

As @chesterxgchen suggested, maybe you can try to use our POC command first to see if it works?

You can do:

nvflare poc prepare -n 2

Then the following command to start the whole system (and will login to admin CLI automatically)

nvflare poc start

Then inside you should be able to "list_jobs" "check_status server" and all that.

Can you try if this work for you?
If this does not work, can you share more information of your running environment?
Like if it is AWS Azure GCP OCI or anything.
You using any docker container or just run directly in console

[YuanTingHsieh]

close due to inactivity, feel free to re-open