Anomaly Detection in Network Traffic Using Advanced Machine Learning Techniques

Abstract

Recently, network traffic anomaly detection is an important part of increasing cybersecurity and defence against complex attacks. In this paper we present a novel ML based anomaly detection system, with the NSL-KDD dataset. The system tackles both class imbalance and redundant data by employing Synthetic Minority Oversampling Technique and Principal Component Analysis and efficiently reduces dimensionality while improving learning. Various classifiers such as LR, SVM, RF, NB, XGBoost, LightGBM and Isolation Forest are realized to create a solid baseline performance. In addition, an ensemble Voting Classifier is developed, using boosted DT and bagging based RF models, to further improve the resilience. Feature-level interpretations, with transparency, are supported via the application of XAI methodologies like SHAP and LIME to support trustworthy decision making. The experimental results reveal that the VC outperforms individual classifiers with 99.7 per cent accuracy, precision, recall and F1 score. The solution is deployed on Flask framework, with a user signin / signup system based on SQLite in order to use it in the real world. The online interface has interaction with the system, allows the preprocessing of the data, visualises the prediction, and provides interpretable output of faulty/non-defective.