An Individualized System of Skeletal Data-Based CNN Classifiers for Action Recognition in Manufacturing Assembly


Real-time Action Recognition (ActRgn) of assembly workers can timely assist manufacturers in correcting human mistakes and improving task performance. Yet, recognizing worker actions in assembly reliably is challenging because such actions are complex and fine-grained, and workers are heterogeneous. This paper proposes to create an individualized system of Convolutional Neural Networks (CNNs) for action recognition using human skeletal data. The system comprises six 1-channel CNN classifiers that each is built with one unique posture-related feature vector extracted from the time series skeletal data. Then, the six classifiers are adapted to any new worker through transfer learning and iterative boosting. After that, an individualized fusion method named Weighted Average of Selected Classifiers (WASC) integrates the adapted classifiers as an ActRgn system that outperforms its constituent classifiers. An algorithm of stream data analysis further differentiates the actions for assembly from the background and corrects misclassifications based on the temporal relationship of the actions in assembly. Compared to the CNN classifier directly built with the skeletal data, the proposed system improves the accuracy of action recognition by 28%, reaching 94% accuracy on the tested group of new workers. The study also builds a foundation for immediate extensions for adapting the ActRgn system to current workers performing new tasks and, then, to new workers performing new tasks.


Mechanical and Aerospace Engineering


All the authors of this paper received financial support from the National Science Foundation through the Award CMMI-1646162.

Keywords and Phrases

Action Recognition; Classifier Fusion; Convolutional Neural Network; Deep Learning; Iterative Boosting; Smart Manufacturing; Transfer Learning

International Standard Serial Number (ISSN)

0956-5515; 1572-8145

Document Type

Article - Journal

Document Version


File Type





© 2021 The Authors, All rights reserved.

Publication Date

26 Jul 2021