Towards lifelong and long-term sensor-based human activity recognition

The field of Human Activity Recognition (HAR) has witnessed remarkable advancements in the past decade, fueled by advancements in mobile devices, sensors, and computational methods. HAR has found widespread applications in areas such as mobile health monitoring, disorder diagnosis, personal assistance, and smart environments. However, despite significant progress, HAR faces critical learning challenges that hinder its deployment in real-world scenarios. The prevailing approach in HAR relies on offline data collected under controlled laboratory settings, which fails to capture the dynamic nature of real-world environments and individuals’ behaviors. Moreover, the heterogeneity of sensor devices, specifications, and placements introduces further challenges for long-term deployment. To address these shortcomings, lifelong adaptive learning or continual learning, in HAR, becomes crucial to enable large-scale, long-term, and sustainable activity recognition systems. This dissertation aims to tackle the challenges of lifelong adaptive learning in HAR, focusing on three key components: ground truth annotation with minimal user burden, distribution shifts in HAR, and continual model adaptation. The first component tackles the time-consuming task of acquiring ground truth information by exploring active learning in an effort to minimize data annotation effort and in turn user burden. The second component delves into the problem of distribution shifts introduced by device heterogeneity, sensor placements, and contextual environments, and their impact on model generalizability. The third component explores incremental model adaptation to accommodate changes in behavior and environment as well as introducing new activity classes while mitigating catastrophic forgetting of previously learnt information. A proposed continual learning framework, LAPNet, addresses such challenges. To drive new research in this area, we also introduce a large-scale, stronglylabelled, multi-device, and multi-location human activity dataset collected from inertial sensors, MotionPrint. This dataset was collected from 50 people, time-aligned across six locations, with sampling rates up to 800 Hz, resulting in 200 cumulative hours and 408 million samples of data. Our comprehensive analysis reveals a significant challenge in constructing lifelong deployable motion models that can effectively generalize to diverse sensor locations. This observation presents a valuable opportunity for the machine learning community to engage in new research endeavors focused on developing adaptable and generalizable learning algorithms. Addressing the issue of sensor heterogeneities is a critical aspect in realizing our vision of lifelong learning for HAR. Furthermore, we introduce a pioneering line of research, cross-location data synthesis. This research explores the problem of generating synthetic motion data in one location using available data from another location. The work presented in this dissertation contributes to pushing the field of HAR towards long-term sustainable deployment in real-world settings. By addressing the challenges of achieving practical deployable recognition systems, the research provides insights and methodologies for developing personalized and adaptable activity recognition systems that can continuously learn and adapt to evolving circumstances, enhancing their usability and effectiveness in practical applications. In addition, the release of MotionPrint lowers the entry barrier and facilitates progress in the field of lifelong adaptive learning for HAR, fostering the development of innovative techniques and impactful applications in ubiquitous computing and interactive systems.