Low-cost Design of Vision-based Natural User Interface via Dynamic Hand Gestures

Abstract

Advancement in computer vision and pattern recognition fields has opened many new dimensions for object recognition and visual tracking in videos. But vision-based interaction of a human with a machine using moving hand gestures is still in a primitive stage because seamless localization and conversion of irregular trajectory to command in RGB images are challenging tasks. The hand is a non-rigid object with diverse posture shapes, some posture occupies a larger area and some very small. The surface of a hand is uneven thus edges are not very clear during movement. Therefore, algorithms example background subtraction, segmentation using contour information, and skin color detection are not practically applicable in unconstrained background conditions. Hence researchers prefer advanced cameras that can detect the skeleton structure and provide more information apart from RGB values for each pixel of the image frame to facilitate the hand detection stage in dynamic hand gesture recognition (DHGR). The approach of using advanced cameras in DHGR increases the overall cost of interfaces, subjects require technical knowledge to operate them, and hence applications of DHGR are limited to those areas where complexity, cost, and expertise are not very critical factors. The goal of this paper is to propose the low-cost, simple design of vision-based human-machine interaction via dynamic hand gestures that directly work on RGB images, can be easily integrated with any day-to-day machine and the technique is invariant to the user’s age and skills. The architecture of the proposed method utilizes Faster Region-based Convolutional Neural Network (Faster R-CNN) for hand spotting and recognition, tracking is accomplished using SIFT (scale-invariant feature transform), modified Backpropagation Artificial Neural Network is final added to efficiently translate hand movement into machine command with optimal computations. The proposed technique is simple yet robust in hand detection and capable to track and interpret a non-rigid object directly in the colored videos, captured in a realtime environment.