A BERT-Based Deep Learning Framework for Multimodal Hate Speech Detection

Abstract

In the recent decade, with the development of social media applications, there has been an increasing dispersion of harmful and abusive content across various languages and media formats. And these contents often appear in multimodal forms where textual messages are in the combination with images, memes, or other visual elements. Prior studies reveal that traditional text-based detection methods often fail to capture the complex relationships among these modalities, making it difficult to identify implicit or context-dependent harmful expressions. 
Through this study, dealing with the above challenges in mind, we have made a humble attempt to implement a transformer-based multimodal deep learning framework for detecting toxic/hate speech online content. 
Two proposed approaches, Bidirectional Encoder Representations from Transformers (BERT) & multilingual BERT (mBERT), have been made for the extraction of textual features, while deep vision-based models have been made for visual features extraction. The outcome of both textual and visual representations are then incorporated by using multimodal fusion techniques for better understanding of complex patterns in online content.
The proposed model is evaluated on datasets specifically designed for the research domain of hate speech and harmful content detection, such as MMHS150K and the Facebook Hateful Memes datasets. The outcomes of this study show that the suggested multimodal approach is much more efficient compared to conventional approaches which employ only one channel. Using both textual and visual channels allows capturing contextual relationships and hints, like understanding sarcasm. Also, this approach supports safer, more reliable moderated systems that improve the identification of harmful and culturally nuanced content on social media.