Mental Health Risk Detection via Explainable AI (XAI): A Multi-Model Benchmark with Contextual Embeddings

Abstract

Mental health disorders represent a growing global public health concern, yet early detection remains challenging due to the reliance on conventional clinical assessments. The rise of social media provides a unique opportunity to monitor individuals' emotional states through their textual expressions passively. However, existing automated detection approaches are predominantly opaque "black-box" systems, limiting their adoption in clinical and educational settings where interpretability is paramount. This paper presents a comprehensive, multi-modal pipeline for automated mental health risk detection that integrates contextual embeddings from BERT (Bidirectional Encoder Representations from Transformers) with behavioral and linguistic indicators. We benchmark five distinct classifier architectures, including TF-IDF-based and BERT-based models, using Stratified 5-Fold Cross-Validation across multiple metrics: Accuracy, Macro F1-Score, Cohen's Kappa (κ), and Matthews Correlation Coefficient (MCC). The best-performing model, BERT + Logistic Regression, achieves a Macro F1-Score of 0.6464 ± 0.0119 and a Macro AUC-ROC of 0.8726, while BERT + Random Forest achieves the highest accuracy of 0.8742 ± 0.0020. To address the interpretability gap, we implement a multi-faceted Explainable AI (XAI) framework comprising Shapley Additive exPlanations (SHAP) for global and local feature attribution, Local Interpretable Model-agnostic Explanations (LIME) for word-level insights, and Counterfactual Analysis for decision-boundary understanding. Statistical significance is rigorously validated via McNamara’s test(p <0.001) and the Wilcoxon signed-rank test. Our results demonstrate that the integration of rich contextual embeddings with transparent XAI produces a trustworthy, accurate, and actionable system for mental health risk monitoring.