An Individualized System of Skeletal Data-Based CNN Classifiers for Action Recognition in Manufacturing Assembly
Abstract
Real-time Action Recognition (ActRgn) of assembly workers can timely assist manufacturers in correcting human mistakes and improving task performance. Yet, recognizing worker actions in assembly reliably is challenging because such actions are complex and fine-grained, and workers are heterogeneous. This paper proposes to create an individualized system of Convolutional Neural Networks (CNNs) for action recognition using human skeletal data. The system comprises six 1-channel CNN classifiers that each is built with one unique posture-related feature vector extracted from the time series skeletal data. Then, the six classifiers are adapted to any new worker through transfer learning and iterative boosting. After that, an individualized fusion method named Weighted Average of Selected Classifiers (WASC) integrates the adapted classifiers as an ActRgn system that outperforms its constituent classifiers. An algorithm of stream data analysis further differentiates the actions for assembly from the background and corrects misclassifications based on the temporal relationship of the actions in assembly. Compared to the CNN classifier directly built with the skeletal data, the proposed system improves the accuracy of action recognition by 28%, reaching 94% accuracy on the tested group of new workers. The study also builds a foundation for immediate extensions for adapting the ActRgn system to current workers performing new tasks and, then, to new workers performing new tasks.
Recommended Citation
M. Al-Amin et al., "An Individualized System of Skeletal Data-Based CNN Classifiers for Action Recognition in Manufacturing Assembly," Journal of Intelligent Manufacturing, Springer, Jul 2021.
The definitive version is available at https://doi.org/10.1007/s10845-021-01815-x
Department(s)
Mechanical and Aerospace Engineering
Keywords and Phrases
Action Recognition; Classifier Fusion; Convolutional Neural Network; Deep Learning; Iterative Boosting; Smart Manufacturing; Transfer Learning
International Standard Serial Number (ISSN)
0956-5515; 1572-8145
Document Type
Article - Journal
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2021 The Authors, All rights reserved.
Publication Date
26 Jul 2021
Comments
All the authors of this paper received financial support from the National Science Foundation through the Award CMMI-1646162.