American Sign Language Alphabet Recognition using Leap Motion Controller

Abstract

Recognition of American Sign Language (ASL) alphabet not only could bring benefits to the ASL users, but also could provide solutions for natural human-computer/robot interactions in many applications. In this paper, we propose a method for ASL alphabet recognition with use of a Leap Motion Controller (LMC). The skeleton data from the native LMC API is transformed by a skeleton module into a vector of the angle features. Meanwhile, two raw infrared-radiation (IR) images are captured and each of them is fed into a vision module using a Convolutional Neural Network (CNN) for visual feature extraction, which results in two feature vectors. Those three feature vectors are then fed into a fusion neural network to output the predicted label. An ASL alphabet dataset is established, on which the proposed model is evaluated. The results show that our proposed method achieves the prediction accuracies of 80.1% and 99.7% in the leave-one-out and the half-half experiments, respectively.

Meeting Name

2018 Institute of Industrial and Systems Engineers Annual Conference and Expo, IISE 2018 (2018: May 19-22, Orlando, FL)

Department(s)

Mechanical and Aerospace Engineering

Second Department

Computer Science

Research Center/Lab(s)

Intelligent Systems Center

Comments

This research work was supported by the National Science Foundation grant CMMI-1646162 on cyber-physical systems and also by the Intelligent Systems Center at Missouri University of Science and Technology.

Keywords and Phrases

American Sign Language; Convolutional Neural Network; Deep learning; Leap Motion Controller; Machine learning

Document Type

Article - Conference proceedings

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2018 Institute of Industrial and Systems Engineers (IISE), All rights reserved.

Publication Date

01 May 2018

This document is currently not available here.

Share

 
COinS