Explainable AI for IoT-Driven Decision Support: Building Trust in Human-AI Collaboration via Deep Learning Interpretability

Abstract

This study was focused on the development and assessment of an explainable artificial intelligence (XAI) system for an Internet of Things (IoT) based decision support system, not only in terms of prediction accuracy, but also in how it impacts human-AI collaboration. A computational experimental methodology was used in which, IoT sensor data were gathered, cleaned, normalized, treated for missing values, and selected features, and then partitioned into training, validation, and testing datasets. A deep learning model was developed to make predictions that supported decision making; subsequently, model agnostic explanation techniques were embedded, post hoc, to uncover and feed-back the most salient features that lead to each prediction. Predictive performance was assessed using accuracy, precision, recall, F1-score and area under the receiver operating characteristic curve compared to an existing deep learning model without explanatory mechanisms. The consistency and the relevance of generated explanations were then evaluated and 160 participants were then exposed to AI-driven decisions with or without explanations to test the impact on trust, perceived understanding, decision confidence, and willingness to trust the system. The findings showed that the explainable model can predict as well as the conventional model, the explanations produced by the two different interpretability techniques were very similar, and participants who received the explanations of the decisions they made were significantly more confident, understood, trusted and willing to rely on the decision made by the explainable model than those who were not given the explanations. Further, the quality of explanations was also determined to be significantly and positively correlated with user trust. The results indicate that interpretability can be integrated into the deep learning-based decision support for IoT without compromising its predictive power and effectively enhance the integration between humans and AI.