A Real-Time Web-Based Speech-to-American Sign Language Translation System Using a Pre-Trained CResD-Net

Abstract

Communication between hearing disabled and society remain challenging as both rely on different modes of communication. Most of existing work in this field converts speech into text in offline mode. It makes them unsuitable for real-time conversations. This paper presents a real-time speech-to-American Sign Language (ASL) translation. Django framework is used to create front-end of the system. First, this system captures speech from a web application using browser microphone. Second, it processes the speech to convert in waveform and remove the silence. Third, it recognized the speech using a pretrained speech command recognition model. Then, it maps the recognized command to its corresponding American Sign Language video from the WLASL dataset. Finally, it displayed the predicted command and sign video on the browser. The system was evaluated using real-time speech input from five participants. Translation accuracy and average response time were evaluated using real-time speech. The experimental results demonstrate that the system can accurately recognize predefined speech commands and display the corresponding ASL videos interactively to users.