Running multiple external controllers for the same robot

[tonynajjar]

Hello, I'm new to Webots and participating in the wrestling competition. I'm using ROS so I'm creating a distributed system of Nodes. In many of those nodes I need to access controllers.Robot (e.g to retrieve sensor data or to actuate motors). However when I try to do this I get:

INFO: 'participant' extern controller: refusing connection attempt from another extern controller.

Is there a way to run multiple external controllers for the same robot?

I found this link but it doesn't answer my question:

https://cyberbotics.com/doc/guide/running-extern-robot-controllers#single-simulation-and-multiple-local-extern-robot-controllers - This talks about multiple controllers for multiple robots

[omichel]

Actually, if you want to use ROS, you should not use the Webots controller API at all.
Instead, you should use the webots_ros2 interface.
In webots_ros2, a launcher usually start both Webots and your ROS nodes.
You should therefore simplify this launcher: remove the launch of Webots and keep only the launch of your ROS controller node (and all the other ROS nodes that you need). Before starting this launcher, you should set the WEBOTS_CONTROLLER_URL environment variable (as it is currently set in the default Dockerfile) which is defined by the CI, so that your ROS controller will be able to connect with Webots.
Here is an example of launcher: https://github.com/cyberbotics/webots_ros2/blob/master/webots_ros2_epuck/launch/robot_launch.py
This one is for an e-puck robot and launches Webots as well. So, you can modify it to adapt it to your need and in particular remove the launch of Webots from it. This is about here in the launcher. You should be able to test all this locally on your Linux machine before pushing it to the CI.
Let me know if you have any further question while doing this.

[tonynajjar]

Here is what I did:

• change from the vulcanexus image to osrf/ros:humble-desktop
• install webots and webots_ros2 via apt
• I have these environment variables in the dockerfile:

ENV WEBOTS_HOME /usr/local/webots
ENV PATH /usr/local/webots:${PATH}
ENV PYTHONPATH=${WEBOTS_HOME}/lib/controller/python
ENV PYTHONIOENCODING=UTF-8
ENV LD_LIBRARY_PATH=${WEBOTS_HOME}/lib/controller:${LD_LIBRARY_PATH}
ARG WEBOTS_CONTROLLER_URL
ENV WEBOTS_CONTROLLER_URL=${WEBOTS_CONTROLLER_URL}

and I still get the same error... it's mind-boggling

root@logi-XMG-CORE-REN-M20:~/ros2/src# ros2 launch webots_wrestler simulation.py
[INFO] [launch]: All log files can be found below /root/.ros/log/2023-02-03-21-09-19-674367-logi-XMG-CORE-REN-M20-82065
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [webots-1]: process started with pid [82066]
[INFO] [driver-2]: process started with pid [82068]
[INFO] [ros2_supervisor.py-3]: process started with pid [82070]
[driver-2] Cannot connect to Webots instance, retrying in 0 second...
[ros2_supervisor.py-3] Cannot connect to Webots instance on socket "/tmp/webots-1234/ipc/Ros2Supervisor/extern", retrying for another 50 seconds...
[driver-2] Cannot connect to Webots instance on socket "/tmp/webots-1234/ipc/participant/extern", retrying in 1 second...
[driver-2] Cannot connect to Webots instance on socket "/tmp/webots-1234/ipc/participant/extern", retrying in 2 seconds...
[ERROR] [driver-2]: process has died [pid 82068, exit code 1, cmd '/root/ros2/install/webots_ros2_driver/lib/webots_ros2_driver/driver --ros-args --params-file /tmp/launch_params_ot45fq2i'].

This error is from own driver but also reproduces with the test from webots_ros2_tests

Here is a few information about the new container:

root@logi-XMG-CORE-REN-M20:~/ros2/src# echo $LD_LIBRARY_PATH 
/opt/ros/humble/opt/rviz_ogre_vendor/lib:/opt/ros/humble/lib/x86_64-linux-gnu:/opt/ros/humble/lib:/usr/local/webots/lib/controller:

root@logi-XMG-CORE-REN-M20:~/ros2/src# webots --version
Webots version: R2023a

webots_ros2 is not installed from source anymore.

[tonynajjar]

Maybe this helps you:

root@logi-XMG-CORE-REN-M20:~/ros2/src# ros2 run --prefix 'gdb -ex run --args' webots_ros2_driver driver --ros-args --params-file /tmp/launch_params_kp8h8c7_
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/ros/humble/lib/webots_ros2_driver/driver...
(No debugging symbols found in /opt/ros/humble/lib/webots_ros2_driver/driver)
Starting program: /opt/ros/humble/lib/webots_ros2_driver/driver --ros-args --params-file /tmp/launch_params_kp8h8c7_
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff60d4640 (LWP 17083)]
[New Thread 0x7ffff58d3640 (LWP 17084)]
[New Thread 0x7ffff504b640 (LWP 17085)]
[New Thread 0x7ffff484a640 (LWP 17086)]
[New Thread 0x7ffff4049640 (LWP 17087)]
[New Thread 0x7ffff3848640 (LWP 17088)]
[New Thread 0x7ffff3047640 (LWP 17089)]
[New Thread 0x7ffff2839640 (LWP 17090)]
[New Thread 0x7ffff1f58640 (LWP 17091)]
[New Thread 0x7ffff16bb640 (LWP 17092)]
[New Thread 0x7ffff0e45640 (LWP 17093)]
[Thread 0x7ffff484a640 (LWP 17086) exited]
[Thread 0x7ffff4049640 (LWP 17087) exited]
[Thread 0x7ffff3848640 (LWP 17088) exited]
[Thread 0x7ffff2839640 (LWP 17090) exited]
[Thread 0x7ffff3047640 (LWP 17089) exited]
[Thread 0x7ffff504b640 (LWP 17085) exited]
[Thread 0x7ffff1f58640 (LWP 17091) exited]
[Thread 0x7ffff58d3640 (LWP 17084) exited]
[Thread 0x7ffff60d4640 (LWP 17083) exited]
[Thread 0x7ffff0e45640 (LWP 17093) exited]
[Thread 0x7ffff6e96f40 (LWP 17074) exited]
[Thread 0x7ffff16bb640 (LWP 17092) exited]
[New process 17074]
[Inferior 1 (process 17074) exited with code 01]

[tonynajjar]

I localized where it crashes 🕵️ , it's at this line

[ygoumaz]

From what I see, it is still the wrong libController that is used by your ROS package. Is the bug still occurring if you remove the following environment variable from your Dockerfile ?

ENV LD_LIBRARY_PATH=${WEBOTS_HOME}/lib/controller:${LD_LIBRARY_PATH}

LD_LIBRARY_PATH is used to get the path to the Webots libController only if you use C/C++ or (sometimes) Python extern controllers. Same goes with PYTHONPATH and PYTHONIOENCODING, which are only useful if you use Python extern controllers. These environment variables may infer with the ROS package and should be unset if you are using the ROS package.

[Jean-Eudes-le-retour]

I've made a simple working ROS 2 example, you can check out the code here: https://github.com/cyberbotics/wrestling-alice-ros-2

[omichel]

The Vulcanexus ROS distribution provides a fairly lightweight humble-simulation docker image that includes the webots_ros2 package. That should be very convenient to use in the wrestling competition.

The Alice (ROS) example provide a complete template based on Vulcanexus for participants willing to use ROS for their robot controller.