HuMAn: Complex Activity Recognition with Multi-Modal Multi-Positional Body Sensing


Current state-of-the-art systems in literature using wearables are not capable of distinguishing large number of fine-grained and/or complex human activities, which may appear similar but with vital differences in context, such as lying on floor versus lying on bed versus lying on sofa. This paper fills the gap by proposing a novel system, called HuMAn, that recognizes and classifies complex at-home activities of humans with wearable sensing. Specifically, HuMAn makes such classification feasible by leveraging selective multi-modal sensor suites from wearable devices, and enhances the richness of sensed information for activity classification by further leveraging placement of wearables across multiple positions on human body. The HuMAn system consists of the following components: (a) a practical feature set extraction method from specific multi-modal sensor suites; (b) a novel two-level structured classification algorithm that improves accuracy by leveraging sensors in multiple body positions; and (c) improved refinement in classification of complex activities with minimal external infrastructure support (e.g., only a few Bluetooth beacons used for location context). The proposed system is evaluated with 10 users in real home environments. Experimental results demonstrate that the HuMAn system can detect 21 complex at-home activities with high degree of accuracy. For same-user evaluation strategy, the average activity classification accuracy is as high as 95 percent over all of the 21 activities. For the case of 10-fold cross-validation evaluation strategy, the average classification accuracy is 92 percent, and for the case of leave-one-out cross-validation strategy, the average classification accuracy is 75 percent.


Computer Science

Research Center/Lab(s)

Intelligent Systems Center

Second Research Center/Lab

Center for High Performance Computing Research


This work was supported in part by the US National Science Foundation (NSF) under Award Nos. IIS-1404673, IIP1648907, IIP-1540119, IIS-1254117 and II-New-1205695.

Keywords and Phrases

Pattern recognition; Statistical methods; Structured programming; Wearable technology; 10-fold cross-validation; Activity classifications; Classification accuracy; Classification algorithm; Complex activity; Conditional random fields; Leave-one-out cross validations; Multimodal sensor; Classification (of information); Activity recognition; Biomedical monitoring; Complex activity recognition; Feature extraction; Legged locomotion; Smart health; Smartphone multi-modal sensors; Wrist

International Standard Serial Number (ISSN)

1536-1233; 1558-0660

Document Type

Article - Journal

Document Version


File Type





© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

01 Apr 2019