Sensing-based capability-aware robot programming guidance system for non-expert users

Skill-based robot programming is commonly used in manufacturing automation to provide non-expert users with flexible ways to specify various robot tasks in the robot workspace. However, non-experts typically are unaware of the capability of the robot system, such as the applicability and the quality of the skill and parameters they specify. Users may have to wait until runtime to find inappropriate parameter settings and other errors in platform-specific specifications, making robot programming difficult. How to develop a robot programming system that can allow non-expert users to specify robot tasks with only basic knowledge of the robot's capability during task programming is a significant problem. Although existing works tackle this problem by pre-embedding experts' knowledge of the robot skills and system and only expose what users can specify in the programming, incorporating the sensing-based capability of the robot system is not well studied. Without sufficient ability for detection or monitoring, a robot system may be late in capturing execution faults that could drastically reduce performance or cause a catastrophic system crash. In this dissertation, we shall address the above issue by the following. (1) We introduce a programming system framework that incorporates the sensing-based capability of the robot system to generate guidance suggestions in assisting non-expert users in specifying effective robot programs. (2) We develop several components for the system. LASSO is a programming tool designed to construct spatial information of target objects for robot skills. The SQRP system extracts the spatial and temporal requirements from a robot skill to compute sensing-based capability. We further develop the SQGS system for users to define fine-grained requirements for sensing-based capability and provide guidance suggestions for parameter selection and environment reconfiguration. Ultimately, we develop the SensingRG system to extend our framework to include more complicated robot skills and tasks. (3) We evaluate the efficacy of our system framework by conducting a user study on a real-world robot task and demonstrate our framework on a real-world robot toy assembly task to show generalizability.