A Computer Vision and HCI System for Robotic Arm Control


We developed a computer vision based object classification method and a human computer interaction (HCI) system which uses a Cyber Glove and a Flock of Birds® motion tracker to translate hand gestures into commands for a robotic arm. Our system consists of a conveyor, a simulated robotic manipulator, an HCI setup, and a vision system. First, the speed and the position of the objects are dynamically computed on a moving conveyor using standard motion estimation techniques. Then, objects are classified by an artificial neural network (ANN) using various image processing techniques and a moment invariant method. After the classification step, object specific commands are issued using hand gestures to further control the robotic arm which are translated into a command sequence via the HCI system. The system is successfully tested for different objects moving on a conveyor. The hardware implementation and the overview of the algorithms used along with the results obtained are presented.


Mechanical and Aerospace Engineering

Second Department

Computer Science


Ford Foundation
National Science Foundation (U.S.)
University of Missouri--Rolla. Intelligent Systems Center

Keywords and Phrases

Artificial Neural Network; Moment Invariants; Object Classification; Human-computer interaction; Image processing

Document Type

Article - Conference proceedings

Document Version


File Type