Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, Yuke Zhu
We tackle the problem of developing humanoid loco-manipulation skills with deep imitation learning. The challenge of collecting human demonstrations for humanoids, in conjunction with the difficulty of policy training under a high degree of freedom, presents substantial challenges. We introduce TRILL, a data-efficient framework for learning humanoid loco-manipulation policies from human demonstrations. In this framework, we collect human demonstration data through an intuitive Virtual Reality (VR) interface. We employ the whole-body control formulation to transform task-space commands from human operators into the robot's joint-torque actuation while stabilizing its dynamics. By employing high-level action abstractions tailored for humanoid robots, our method can efficiently learn complex loco-manipulation skills. We demonstrate the effectiveness of TRILL in simulation and on a real-world robot for performing various types of tasks.
If you find our work useful in your research, please consider citing.
- Python 3.8.5 (recommended)
- Robosuite 1.4.0
- Robomimic 0.2.0
- PyTorch
Please see Getting Started.
We provide our demonstration dataset in the door simulation environment (link) and trained models of the Visuomotor Policies (link). We also plan to open our demonstration dataset and trained models in the workbench simulation environment in the near future.
Please see this page for detailed information on our implementation, including the whole-body controller, model architecture, and teleoperation system.
The implementation of the whole-body control is based on PyPnC.
@inproceedings{seo2023trill,
title={Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation},
author={Seo, Mingyo and Han, Steve and Sim, Kyutae and
Bang, Seung Hyeon and Gonzalez, Carlos and
Sentis, Luis and Zhu, Yuke},
booktitle={IEEE-RAS International Conference on Humanoid Robots (Humanoids)},
year={2023}
}