Unified Multimodal Misinformation Detection Model Based on EfficientNet-B0, BERT and Stack Ensemble

Abstract

Multimodal misinformation has been identified as a key challenge in social media platforms, where the misleading information is presented using both textual and visual modalities. The current state-of-the-art methods for detecting misinformation are primarily unimodal or based on monolithic deep learning models that lack the ability to generalize across different patterns of misinformation. This paper presents a Hybrid Vision-Language Stacked Ensemble Model that combines deep semantic features from EfficientNet-B0 and BERT with traditional ensemble learning techniques to handle multimodal misinformation detection. In particular, high-level image features were extracted using EfficientNet-B0 and text embeddings using a pre-trained BERT model. The learned multimodal feature representations were then used by a stacked ensemble classifier that leverages the complementary capabilities of Random Forest, Gradient Boosting, and a Logistic Regression meta-learner. Experiments conducted on the MMFakeBench dataset for binary and multiclass misinformation classification tasks have shown that the proposed framework achieves better performance compared to individual base learners and conventional fusion approaches. Further detailed evaluation had also validated the efficacy of modality-wise feature fusion and the robustness of the ensemble learning approach against class imbalance and noisy data. The proposed framework provides a scalable, interpretable, and robust solution for real-world multimodal misinformation detection on social media platforms.