A Self-Aware and Active-Guiding Training & Assistant System for Worker-Centered Intelligent Manufacturing


Training and on-site assistance is critical to help workers master required skills, improve worker productivity, and guarantee the product quality. Traditional training methods lack worker-centered considerations that are particularly in need when workers are facing ever-changing demands. In this study, we propose a worker-centered training & assistant system for intelligent manufacturing, which is featured with self-awareness and active-guidance. Multi-modal sensing techniques are applied to perceive each individual worker and a deep learning approach is developed to understand the worker's behavior and intention. Moreover, an object detection algorithm is implemented to identify the parts/tools the worker is interacting with. Then the worker's current state is inferred and used for quantifying and assessing the worker performance, from which the worker's potential guidance demands are analyzed. Furthermore, onsite guidance with multi-modal augmented reality is provided actively and continuously during the operational process. Two case studies are used to demonstrate the feasibility and great potential of our proposed approach and system for applying to the manufacturing industry for frontline workers.


Mechanical and Aerospace Engineering

Second Department

Computer Science

Third Department

Engineering Management and Systems Engineering

Research Center/Lab(s)

Center for Research in Energy and Environment (CREE)


This research work is supported by the National Science Foundation grant CMMI-1646162 and NRI-1830479, and also by the Intelligent Systems Center at Missouri University of Science and Technology.

Keywords and Phrases

Augmented reality; Cyber-physical system; Deep learning; Intelligent manufacturing; Smart manufacturing

International Standard Serial Number (ISSN)


Document Type

Article - Journal

Document Version


File Type





© 2019 Society of Manufacturing Engineers (SME), All rights reserved.

Publication Date

01 Aug 2019