Bayesian-Optimized Hybrid Random Forest and Feature Attention-Based Residual Neural Network for Software Defect Prediction

Abstract

Software defect prediction is an important activity used for identifying fault-prone modules at an early stage of software development. It is used for enhancing software reliability and reducing maintenance costs. Traditional software defect prediction machine learning(ML) models perform well but often fail to capture complex nonlinear relationships in software metrics. Deep learning(DL) models address this limitation by learning hierarchical features but can still be hindered by class imbalance, poor hyperparameter tuning, and insufficient attention to feature importance. To address these challenges, this paper proposes a hybrid software defect prediction model (RF-FA-ResANN) that integrates Random Forest with a Feature Attention-based Residual Artificial Neural Network. This model is evaluated using Stratified 5-Fold Cross-Validation with Accuracy, Precision, Recall, F1-score, and ROC-AUC as performance metrics. Experimental results demonstrate that this model achieved accuracies of 99.83% and 90.29%, F1- scores of 99.15% and 87.1%, with ROC-AUC scores of 0.99998 and 0.94448 on software_defect and NASA JM1 datasets respectively. Experimental results show that proposed RF-FA-ResANN model outperformed conventional Machine Learning models (Support Vector Machine, Naïve Bayes, and Random Forest) as well as Deep Learning models (ANN, Attention-based ANN, Residual ANN, and Attention-based Residual ANN). This model integrates ensemble learning, feature attention, residual learning, and Bayesian optimization, which improves robustness and accuracy in software defect prediction.