Vision-Language Hybrid Transformer OCR (VLHT-OCR): A Novel Transformer-Based Optical Character Recognition Model

Abstract

Optical Character Recognition (OCR) has progressed quickly with the rise of deep learning. However, current methods based on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) struggle with understanding context, handling long-range dependencies, and being sturdy against noise, distortions, and variations in handwriting. In this study, it presents VLHT-OCR, a Vision-Language Hybrid Transformer model. It uses a Vision Transformer (ViT) encoder  for patch-based feature extraction, a multi-scale fusion module inspired by Perceiver IO for hierarchical representation, and an autoregressive Transformer decoder with cross-attention for text generation. To enhance linguistic coherence and context accuracy, we integrate a pretrained language model (BERT/GPT-2) for post-processing. Our contributions are threefold: (i) a new hybrid architecture that combines visual and linguistic modes, (ii) clear multi-scale feature fusion that captures detailed and overall text patterns, and (iii) the addition of language-aware refinement to reduce semantic inconsistencies. We test our model on three benchmark datasets: ICDAR-2019 Handwritten Text, IAM Handwritten Database, and CORD Receipt Dataset. The results show state-of-the-art performance, with a Character Error Rate (CER) of 5.2% and a Word Error Rate (WER) of 7.9%. This outperforms recent transformer-based OCR frameworks like TrOCRand Donut by 3–5%. Ablation studies confirm that multi-scale fusion and language model integration are effective. Thus, VLHT-OCR offers a strong, multilingual, and handwriting-capable OCR solution with great potential for digitizing documents in the real world.