Combining simulated predictions and real-world data for efficient robot model adaptation

Researchers, governments, and companies have recently begun deploying intelligent robots into a variety of increasingly unstructured environments where they may face a broad range of tasks, scenarios, and parameter variations. Thanks to machine learning and control techniques, these robots can be taught models of their environment and can adapt their models by observing their surroundings, exploring, and accepting input provided by nearby humans. Unfortunately, most existing methods for performing this model adaptation use numerical optimization techniques that require a large amount of data from the robot's current environment. Collecting this data can be time-consuming, costly, and potentially unsafe for the robot or its surroundings. Alternatively, robots are able to learn models quickly and cheaply by using simulated data, but the models may be infeasible when deployed on a physical system due to differences in physical behavior between the real world and the simulation used for training—a phenomenon known as the reality gap. Data scarcity and the reality gap are both unsolved problems in robotics that hinder a robot's capability to operate in unstructured environments. This dissertation shows that robots can use simulation in conjunction with a small amount of real-world data and human input to perform model adaptation and cross the reality gap more efficiently and effectively than existing techniques. If algorithms and machine learning techniques are designed carefully, small amounts of high-quality real-world data and quick-to-collect, low-quality simulated data can complement each other during adaptation, leading to increased data efficiency without sacrificing accuracy. Whereas prior work generally considers simulation as a simple tool for either pretraining or planning, in this work it plays an more important role in model adaptation by safely and quickly converting between parameter space, which is where optimization occurs, and state space, which is where most robot and human goals are defined and evaluated. A robot can then utilize simulation's predictive power to guide its parameter exploration and data collection processes, enabling more data-efficient active learning. In particular, this effort shows that simulation is useful during every step of the adaptation process (initialization/pretraining, robot exploration, collecting human input, and updating model parameters) in a variety of simulated and real robotics tasks. The dissertation makes the following contributions: first, the development of a new simulation-assisted robot model adaptation framework, which alternates between simulation, real-world data collection, and model learning; second, iterative residual tuning (IRT), a new model adaptation algorithm that uses a neural network pretrained on simulated data in conjunction with minimal observations from physical robot exploration; third, a pair of experiments showing IRT's applicability to challenging real-world robotics tasks; and fourth, Preference-based Uncertainty-aware Model Adaptation (PUMA), a simulation-assisted model adaptation algorithm that allows a robot to learn an improved controller for an unknown environment by simultaneously performing system identification from robot exploration and learning to estimate human preferences from simple state-based input. Together, these contributions develop simulation as a highly effective tool for a robot to learn models of its environment safely and efficiently