euROBIN Human Tracking

Human tracking pipeline used in euROBIN system as described in XXX.

Description

The pipeline consists of the following main components :

• Human detection
• Human tracking
• Pose estimation (2D)
• Gaze estimation

Installation

How to install with docker and launch interactively

• Get the appropriate cuda keyring file (cuda-keyring_1.0-1_all.deb) that may be required to build the docker from nvidia, optionally you can try without it by commenting the corresponding lines in Dockerfile

• Download models and place them in ./models/

wget https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_320_273e_coco/yolov3_d53_320_273e_coco-421362b6.pth
mv yolov3_d53_320_273e_coco-421362b6.pth models/

Go to : Onedrive and download the model then put in under models/vitpose_small.pth

The 6DRepNet models (face detection and 6D pose estimation) are already in /models (downloaded from SixDRepNet/weights)

• Build docker

docker build -t inria_docker:rgbd_detect .

• Launch docker

sh launch.sh

• Launch the pipeline

python rgbd_detect_3d_dir.py -sixdrep

ROS Interface

Input / output topics and format

Input topics :

• [namespace]/rgb : Ìmage message (bgr8 format)
• [namespace]/depth : Image message (mono16 format)
• [namespace]/pcl : PointCloud2 message (.data buffer is expected to contain 3 or 6 fields of float32 such that it can be reshaped to a numpy array of size [image_height, image_width, n_fields] that offers pixel correspondance with the rgb and depth image)

Output topics :

• [namespace]/human : TransformStamped message with detected human position

Output tf :

• TransformStamped with child frame id [namespace]/human

Options

Flags to run with options

See python rgbd_detect_3d_dir.py -h

Use export VERBOSE=0 to set minimal verbosity (warnings, debug, errors and success). Level 1 adds info and 2 adds timing info.

Acknowledgments

The implementation is based on the following implementations of human detection and tracking, human pose estimation and 6D Head Pose estimation

• ViTPose
• MMDetection
• MMPose
• 6DRepNet and SixDRepNet

TODO

• Add a description of all options
• Add instructions and requirements.txt for local installation