Deep Learning-Based Hand Gesture Recognition for Simulated Robot Control in CoppeliaSim

Abstract

This paper presents a deep learning-based hand gesture recognition system for controlling a simulated robot in CoppeliaSim. A monocular camera is used to capture hand images, while MediaPipe extracts 21 hand landmarks in real time. These landmarks are processed by a deep learning model to recognize predefined gestures, which are mapped to robot motion commands, including moving forward, moving backward, turning left, turning right, and stopping. The recognized commands are transmitted to CoppeliaSim through a Python interface for real-time robot control. The proposed system is evaluated using standard performance metrics, including Accuracy, Precision, Recall, F1-score, and Confusion Matrix. Experimental results obtained on the held-out test set demonstrate high gesture-recognition accuracy. The proposed approach provides a simple and computationally efficient solution for human–robot interaction, with potential applications in intelligent robotics, industrial automation, and smart control systems.