Abstract
Production innovations are occurring faster than ever. Manufacturing workers thus need to frequently learn new methods and skills. In fast changing, largely uncertain production systems, manufacturers with the ability to comprehend workers' behavior and assess their operation performance in near real-time will achieve better performance than peers. Action recognition can serve this purpose. Despite that human action recognition has been an active field of study in machine learning, limited work has been done for recognizing worker actions in performing manufacturing tasks that involve complex, intricate operations. Using data captured by one sensor or a single type of sensor to recognize those actions lacks reliability. The limitation can be surpassed by sensor fusion at data, feature, and decision levels. This paper presents a study that developed a multimodal sensor system and used sensor fusion methods to enhance the reliability of action recognition. One step in assembling a Bukito 3D printer, which composed of a sequence of 7 actions, was used to illustrate and assess the proposed method. Two wearable sensors namely Myo-armband captured both Inertial Measurement Unit (IMU) and electromyography (EMG) signals of assembly workers. Microsoft Kinect, a vision based sensor, simultaneously tracked predefined skeleton joints of them. The collected IMU, EMG, and skeleton data were respectively used to train five individual Convolutional Neural Network (CNN) models. Then, various fusion methods were implemented to integrate the prediction results of independent models to yield the final prediction. Reasons for achieving better performance using sensor fusion were identified from this study.
Recommended Citation
M. Al-Amin et al., "Action Recognition in Manufacturing Assembly using Multimodal Sensor Fusion," Procedia Manufacturing, vol. 39, pp. 158 - 167, Elsevier B.V., Aug 2019.
The definitive version is available at https://doi.org/10.1016/j.promfg.2020.01.288
Meeting Name
25th International Conference on Production Research Manufacturing Innovation: Cyber Physical Manufacturing, ICPR 2019 (2019: Aug. 9-14, Chicago, IL)
Department(s)
Computer Science
Second Department
Mechanical and Aerospace Engineering
Third Department
Engineering Management and Systems Engineering
Research Center/Lab(s)
Center for Research in Energy and Environment (CREE)
Keywords and Phrases
Action Recognition; Deep Learning; Manufacturing Assembly; Multimodal Sensor System; Sensor Fusion
International Standard Serial Number (ISSN)
2351-9789
Document Type
Article - Conference proceedings
Document Version
Final Version
File Type
text
Language(s)
English
Rights
© 2019 The Authors, All rights reserved.
Creative Commons Licensing
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Publication Date
01 Aug 2019
Included in
Computer Sciences Commons, Mechanical Engineering Commons, Operations Research, Systems Engineering and Industrial Engineering Commons
Comments
This work was supported by NSF grant CMMI-1646162 on cyber-physical systems.