AI-Based Adaptive Malware Detection Using Portable Executable (PE) Header Analysis and Hybrid Machine Learning

Abstract

With the rapid evolution of malware and sophisticated evasion techniques used, the efficiency of traditional detection approaches will be increasingly compromised since malware creators frequently leverage polymorphic, metamorphic, packing, encryption, and code obfuscation techniques to circumvent traditional security methods. This work includes an AI-based adaptive malware detection system that utilizes the Portable Executable (PE) headers along with hybrid machine learning models. The system conducts static analysis of malware without the need to run malware in sandboxed environments to gather information regarding the structure of executable files. In total, the system trains and tests three machine learning models - Random Forest, eXtreme Gradient Boosting (XGBoost), and Artificial Neural Network (ANN). Training and testing are conducted on a dataset of 19,611 executable files described with 79 characteristics extracted from Portable Executable headers. To enhance predictive accuracy, a hybrid ensemble model is used, while SHapley Additive exPlanations (SHAP) enable predictability. The experiments conducted yielded an accuracy of 99.13% and an Area Under the Curve (AUC) value of 0.9984 with an extremely low false negative rate. The proposed framework combines a hybrid ensemble of Random Forest and XGBoost classifiers with SHapley Additive exPlanations (SHAP) to provide accurate and interpretable malware detection using static PE-header analysis. Experimental results demonstrate that the proposed approach achieves high classification performance while improving transparency in malware detection decisions, making it suitable for practical cybersecurity applications.