Seeking for orientation for developing Bimanual manipulation using imitation learning task and custom controller.

[mikelasa]

Hello,

Sorry If I posted in the wrong place, I wasn't sure where to post my doubts

My goal is to create an environment for imitation learning using two robots, where a box needs to be manipulated to a certain place in a bimanual fashion. To do so, my idea is to teleoperate the robot with the spacemouse during the teaching phase to collect data, similar as in collect_demonstrations.py . I'm seeking for orientation on how can I reach my goal as I have doubts with the teleoperation based intask Isaac-Lift-Cube-Franka-IK-Rel-v0.

So far I have an Interactive scene with a classic robot simulation where I apply the torques to the robots using set_joint_effort_target after computing the torques for each joint. In order to create the task I need to further develop my code into a `ManagerBasedEnv and create a new task (correct me if I'm wrong) where I feel a bit lost.

My doubt is:

1. The tasks task Isaac-Lift-Cube-Franka-IK-Rel-v0 uses the MDP sub-module to create the action terms to use in the environment. Here I see that the controller behind the teleoperation system is based in differential IK controller, and the action passed to the robot is applied by the mdp.JointPositionActionCfgwhich is set by the agent environment configuration in ik_rel_env_cfg.py using the following code:

# Set actions for the specific robot type (franka)
        self.actions.body_joint_pos = DifferentialInverseKinematicsActionCfg(
            asset_name="robot",
            joint_names=["panda_joint.*"],
            body_name="panda_hand",
            controller=DifferentialIKControllerCfg(command_type="pose", use_relative_mode=True, ik_method="dls"),
            scale=0.5,
            body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(pos=[0.0, 0.0, 0.107]),
        )

I see that to use the IK controller, the MDP module provides a DifferentialInverseKinematicsActionCfg which is a configuration class to configure the controller into the action manager via DifferentialInverseKinematicsAction. I see that the class DifferentialInverseKinematicsAction is the one that actually computes the joint position to apply as action to the robot right?

This makes me wonder if I should create into the MDP module a new class to actually introduce my controller and follow the same structure in order to apply actions to the robot.
So far I dont know if this should be the approach to reach my goal or should I think in something else so, I would appreciate any input on this and also discuss about it.

Thanks in advance,
Mikel