A DEEP LEARNING FRAMEWORK FOR VISUAL SPEECH RECOGNITION USING 3D CONVOLUTIONAL NETWORKS AND ATTENTION MECHANISMS

Abstract

Hardware problems in audio devices along with significant levels of background noise often cdistort speech content captured in videos. A new method is needed for restoring such speech data, a computerized lip-reading system generates text out of speechless mouth movements. Frame-based techniques cannot account for the dynamics involved in lip movements, and the phonological confusion among similar-sounding phonemes. The current research aims to design an end-to-end neural network capable of treating video as one whole model. In particular, the architecture uses a 3D ResNet-18 as the feature extractor due to its optimal performance in terms of both computation and feature extraction. The attention module helps the network ignore static background pixels and pay attention to the motion of the lips. Then, a bidirectional recurrent decoder maps the attended features into sequential text outputs. The network was trained for 150 epochs end-to-end using Adam optimization (η = 1 × 10⁻⁴) with horizontal flips, temporal jittering, He initialization, and dropout (0.3). On the GRID Corpus, the word accuracy reached 97.0 ± 0.2%, the CER was 1.4 ± 0.1%, and the sentence-level accuracy was 84.6 ± 0.5% on average for five runs. Furthermore, using a trigram-based language model during beam search decreased the word error rate from 3.0% to 1.8%, which shows that most of the remaining errors are due to decoding errors, not to the inherent ambiguity in the visuals. Thus, the system can transcribe inaudible video recordings accurately into language and is therefore a valuable transcription tool for security analysis and healthcare applications.