Human detection, gesture recognition, and policy generation for human-aware robots




Jorgensen, Steven Jens M.

Journal Title

Journal ISSN

Volume Title



For robots to be deployable in human occupied environments, the robots must have human-awareness and generate human-aware behaviors and policies. This thesis posits that a human-aware robot must be capable of (1) human detection and tracking, (2) human action or intent recognition and (3) intelligent, human-aware action generation. This work presents and evaluates a methodology for each stated capability. In Chapter 2, a method for practical side-by-side human detection for the Valkyrie robot using the Multisense SL sensor is presented. An explanation of why current off-the-shelf techniques are not suitable and a depth-based algorithm using point cloud descriptors and a Random Forest classifier for detecting humans under occlusion, in close proximity, in varying sparsity, and in random poses on the Multisense SL sensor are presented. In Chapter 3, action recognition of arm motion gestures is framed as a supervised learning problem. A popular technique for gesture representation with dynamic movement primitives (DMPs) and its classification using Gaussian Mixture Models (GMMs) is explored. The approach is tested under various hypotheses to understand the intricacies of using DMPs for movement representation. The following findings are reported: (a) recognition rate is sensitive to the number of basis weights, (b) DMPs can be used to recognize two linear motions, (c) rhythmic gestures can be differentiated with the discrete formulation of DMPs, and (d) DMPs can represent static-type gestures. In Chapter 4, a novel technique for (a) representing Human-Robot- Interaction as a dynamical system, and (b) using model predictive control to generate control policies is presented. The approach is motivated by using a scenario in which an Assistive Robot must be productive by bringing work to the human but must also be mindful of the human's workload. By modeling the interaction as a dynamical system, advances in control theory can be leveraged to generate useful control policies.


LCSH Subject Headings