An Efficient Hybrid Machine Learning Framework For Early Skin Cancer Detection Using Deep Learning And Anomaly Detection

Abstract

Skin cancer remains one of the most common and life-threatening malignancies worldwide, where early diagnosis plays a crucial role in improving patient survival. Recent advances in artificial intelligence have enabled the development of automated computer-aided diagnostic systems for accurate skin lesion analysis. This study proposes a hybrid deep learning framework for the early detection of skin cancer using dermoscopic images. The framework integrates image preprocessing, lesion segmentation, feature extraction, deep learning-based classification, and anomaly detection into a unified diagnostic pipeline. Experiments were conducted using the ISIC 2020 benchmark dataset, incorporating image enhancement, hair removal, normalization, and lesion segmentation prior to model training. A Convolutional Neural Network (CNN) was employed as the primary classification model, while Isolation Forest was utilized to identify abnormal lesion patterns and improve diagnostic robustness. The proposed framework was evaluated using accuracy, precision, recall, F1-score, Area Under the ROC Curve (AUC), confusion matrix analysis, and anomaly detection performance. Experimental results demonstrated that the proposed framework achieved superior diagnostic performance compared with conventional machine learning approaches, with the CNN providing the highest classification accuracy and the anomaly detection module enhancing the identification of difficult lesion cases. The findings demonstrate that integrating deep learning with anomaly detection provides a reliable and clinically relevant approach for early skin cancer diagnosis and offers significant potential for assisting dermatologists in computer-aided clinical decision-making.