A new strategy for incorporating BERT embeddings to enhance static word embeddings: The case of COVID-19 SA

Abstract

COVID-19 has claimed many lives to date, not only due to the virus’ physical infection but also due to mental illness, which is related to people’s emotions and psychology. People have been panicked, nervous, and sad as the number of positive cases has increased quickly worldwide. This deadly epidemic has been shown to have a direct influence on the population’s physical and mental health. Throughout this period, social media platforms have played a crucial role in the global spread of news about the outbreak, as individuals shared their emotions over them. Based on this overwhelming evidence, we aim to build a powerful system to analyze people’s feedback on Twitter, targeting specific keywords associated with the outbreak, either directly or indirectly. Therefore, we assume that the effectiveness of contextual word vectors generated with Language Models (LMs) can be further enhanced with the inclusion of static word embeddings that are specially trained on social media (tweets about COVID-19). Moreover, we evaluate different approaches to combine static word embeddings in order to take advantage of their complementarity. Furthermore, we proposed a new technique for dealing with the imbalanced dataset problem. As compared to previous studies, the experimental findings proved that our proposed technique improves the efficiency of the COVID-19 Sentiment Analysis system. Furthermore, a fair comparison of both contextual and static embeddings through Sentiment Analysis reveals that our technique beats the static embeddings trained only from scratch or the ones generated from LMs.