A Neuro-symbolic Framework for Hallucination Detection in Image Captioning Using Knuth– Morris–Pratt Pattern Matching

Abstract

The research on image captioning has improved significantly in terms of describe of a scene appears on an image. This
particular task of image caption generation is considered as most vital because of its wide scope multidimensional application
towards various societal usages. However, the existing caption generators could not be deployed with confidence which could serve
100% accuracy because most of them suffers from hallucination. That means a large number of caption generator still struggling
to achieve the human like performance as far the semantic values or context of the sentence is concerned. We could predict the
potentiality of image captioning task that can leverage many other areas of computer science by simply observing the exponentials
growth in this topic. In spite of this popularity, the hallucination detection is most prominent issue nowadays that could finetune
the performance of the image caption generators. The term hallucination refers to the issue of predicting nonexistent object automatically solely due to the biasness of training data. In other words, when the caption generator automatically predicts a non-
existing object from the input image this is called as object hallucination. Although the hallucination are three different types such as object hallucination, spatial hallucination and multimodal hallucination, the object hallucination is the most prominent one in
this field. To address this object hallucination problem, we have proposed a neuro-symbolic framework that not only generates the
semantically correct caption but also successfully detects the object hallucination by integrating the classic Knuth-Morris-Pratt
(KMP) pattern matching algorithm. We have demonstrated this substring matching process by utilizing the KMP algorithm and
found a linear polynomial time complexity with in the controlled environment of experimental setup.