VR teleoperation interface for learning loco-manipulation of humanoid robots

Access full-text files



Journal Title

Journal ISSN

Volume Title



Our world is designed by humans, for humans. This makes humanoid robots the most suitable general-purpose platform to automate repetitive or dangerous tasks done by people. However, due to the complex dynamics and high degrees-of-freedom of humanoid robots as well as the shortage of demonstration data, research in robot learning for humanoids is scarce. To address these challenges, I present a VR interface named TRILL (TeleopeRation Interface for Learning Loco-manipulation) to collect human demonstrations for humanoid robots in both simulation and reality. The demonstrations are then used to train a baseline Imitation Learning algorithm that uses an underlying controller to abstract away the complexity of whole-body control. I further propose that by embedding this data collection mechanism in VR video games, we can amass a large-scale dataset of high quality human demonstrations that can drive the development of future autonomous humanoids. To illustrate the feasibility of this idea, we collect a small dataset on toy tasks in simulation and real robot using the VR interface. We then show that the trained policy can be deployed in simulation with a reasonable success rate. A video demo of the VR teleoperation can be found here: https://youtu.be/PNZTwtcRhVU.


LCSH Subject Headings