Vision-Based Hand Gesture Keyboard for Game Control Using OpenCV

Abstract

A vision-based system is designed to enable real-time hand gesture control in gaming, without any need for physical accessories. The system uses a standard webcam to record and track hand gestures that mimic steering-wheel movements. A virtual steering indicator is also displayed on the screen, which rotates based on hand tilt to reflect direction and intensity of steering. These gestures are mapped to keyboard keys such as W (acceleration), A (left), D (right), and in certain configurations, S (reverse). The system is developed entirely in Python, using OpenCV for real-time image processing and MediaPipe for precise 21-point hand landmark detection. By programmatically simulating keyboard events, PyAutoGUI enables interaction with almost any PC game that supports conventional keyboard input. The system is able to recognize gestures with low latency and does not require any external equipment or integration with the game. To establish an initial steering reference, users align their hands in a straight position during a short calibration step, simulating a virtual steering wheel. After calibration, the system calculates tilt angles in real time and converts them into keyboard inputs. This touchless interface provides a new way of interaction, especially for users interested in immersive technologies, and also offers an alternative to traditional controllers. The system was tested on the kart racing game (Smash Karts) and demonstrated smooth performance. It also shows potential as a Human-Computer Interaction (HCI) technique in gaming, with scope for further improvements and application in more complex game environments.