Design and Development of a Real-Time Indian Sign Language Conversion System to Text and Speech Using ISLGesture50 Dataset

Abstract

Indian Sign Language (ISL) is the first language for millions of deaf and mute people in India. However, the development of efficient, real, time automatic ISL interpretation systems has been hindered by numerous challenges such as signer variability, scarcity of large annotated datasets, complicated hand articulations, and time constraints for processing. In this work, we demonstrate a real, time vision, based system for transforming Indian Sign Language gestures into Hindi text and speech, which involves the creation of a new dataset called ISLGesture50.Our system is a combination of hand and gesture detection using YOLOv8, hand landmark extraction with MediaPipe, and a temporal consistency, based decision algorithm which recognizes the gestures that are stable across multiple frames and then appends them into the sentence form. This approach drastically lowers the number of false positives in live video situations. The identified gesture string is converted into Hindi linguistic tokens, followed by automatic punctuation and text, to, speech conversion.The main idea in this paper is a low, latency, isolated, gesture recognition method, which contrasts with typical sequence, based deep learning methods that need large continuous datasets. Therefore, the proposed method can be used in real, time on devices with limited computing power. Experimental setups and comparisons with various sign language recognition methods reveal that the integration of object detection with landmark, based features and temporal stability logic enhances the performance and user, friendliness of the system. This research provides a new real, time ISL translation system, a task, specific dataset, and an algorithmic structure that can be used in the next continuous sign language research.