A Multimodal Real-Time Assistive Communication Framework Integrating Sign Language Recognition, Optical Character Recognition, and Facial Expression Analysis

Abstract

Effective communication for individuals with hearing and speech impairments remains a critical yet under-addressed challenge in human–computer interaction (HCI). Although prior systems have achieved notable progress in sign language recognition (SLR), optical character recognition (OCR), and facial expression recognition (FER) individually, no existing platform unifies these three modalities into a single real-time, consumer-deployable application. We propose a unified multimodal assistive communication framework that concurrently processes sign-language gestures, printed or on-screen text, and facial affect from a standard webcam feed with an end-to-end latency below 59 ms per frame on commodity hardware. The SLR module employs MediaPipe hand-landmark tracking coupled with a custom fully connected neural network (FCNN) achieving 94.18 % accuracy on a 26-class ASL test set. The OCR module achieves 98.0 % character recognition accuracy on printed documents and 92.1 % on camera-captured text. The FER module attains 95.0 % macro-accuracy on the FER2013 benchmark under controlled conditions. A five-stage multimodal fusion pipeline routes all recognition outputs to a shared text buffer supporting real-time translation across 50+ languages and bidirectional speech–text conversion. An ablation study across six system configurations confirms that every module contributes positively to overall utility, with the full configuration rated 4.7/5.0 by independent evaluators at 17 FPS. Comparative analysis against seven recent systems demonstrates that   is the only approach providing simultaneous SLR, OCR, and FER on consumer hardware without specialised sensors.