A Gaze-driven Manufacturing Assembly Assistant System with Integrated Step Recognition, Repetition Analysis, and Real-time Feedback

Author

Haodong Chen
Niloofar Zendehdel
Ming C. Leu, Missouri University of Science and TechnologyFollow
Zhaozheng Yin

Abstract

Modern manufacturing faces significant challenges, including efficiency bottlenecks and high error rates in manual assembly operations. To address these challenges, we implement artificial intelligence (AI) and propose a gaze-driven assembly assistant system that leverages artificial intelligence for human-centered smart manufacturing. Our system processes video inputs of assembly activities using a Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) network for assembly step recognition, a Transformer network for repetitive action counting, and a gaze tracker for eye gaze estimation. The application of AI integrates the outputs of these tasks to deliver real-time visual assistance through a software interface that displays relevant tools, parts, and procedural instructions based on recognized steps and gaze data. Experimental results demonstrate the system's high performance, achieving 98.36% accuracy in assembly step recognition, a mean absolute error (MAE) of 4.37%, and an off-by-one accuracy (OBOA) of 95.88% in action counting. Compared to existing solutions, our gaze-driven assistant offers superior precision and efficiency, providing a scalable and adaptable framework suitable for complex and large-scale manufacturing environments.

Recommended Citation

H. Chen et al., "A Gaze-driven Manufacturing Assembly Assistant System with Integrated Step Recognition, Repetition Analysis, and Real-time Feedback," Engineering Applications of Artificial Intelligence, vol. 144, article no. 110076, Elsevier; International Federation of Automatic Control (IFAC), Mar 2025.

The definitive version is available at https://doi.org/10.1016/j.engappai.2025.110076

Department(s)

Mechanical and Aerospace Engineering

Publication Status

Open Access

Comments

National Science Foundation, Grant CMMI- 1954548

Keywords and Phrases

Application of artificial intelligence; Assembly assistance; Eye gaze estimation; Implemented artificial intelligence; Repetitive action counting; Transformer

International Standard Serial Number (ISSN)

0952-1976

Document Type

Article - Journal

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2025 Elsevier; International Federation of Automatic Control (IFAC), All rights reserved.

Publication Date

15 Mar 2025