Abstract

Modern manufacturing faces significant challenges, including efficiency bottlenecks and high error rates in manual assembly operations. To address these challenges, we implement artificial intelligence (AI) and propose a gaze-driven assembly assistant system that leverages artificial intelligence for human-centered smart manufacturing. Our system processes video inputs of assembly activities using a Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) network for assembly step recognition, a Transformer network for repetitive action counting, and a gaze tracker for eye gaze estimation. The application of AI integrates the outputs of these tasks to deliver real-time visual assistance through a software interface that displays relevant tools, parts, and procedural instructions based on recognized steps and gaze data. Experimental results demonstrate the system's high performance, achieving 98.36% accuracy in assembly step recognition, a mean absolute error (MAE) of 4.37%, and an off-by-one accuracy (OBOA) of 95.88% in action counting. Compared to existing solutions, our gaze-driven assistant offers superior precision and efficiency, providing a scalable and adaptable framework suitable for complex and large-scale manufacturing environments.

Department(s)

Mechanical and Aerospace Engineering

Publication Status

Open Access

Comments

National Science Foundation, Grant CMMI- 1954548

Keywords and Phrases

Application of artificial intelligence; Assembly assistance; Eye gaze estimation; Implemented artificial intelligence; Repetitive action counting; Transformer

International Standard Serial Number (ISSN)

0952-1976

Document Type

Article - Journal

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2025 Elsevier; International Federation of Automatic Control (IFAC), All rights reserved.

Publication Date

15 Mar 2025

Download

Full Text Link

Included in

Manufacturing Commons

Share

 
COinS