A Novel 3D Hybrid Transformer-Dense Segmentation Network with Adaptive Capsule-LSTM Classification for Early-Stage Lung Cancer Detection on Kaggle CT Images

Abstract

Background: Early lung cancer detection from CT scans is critical yet challenged by volumetric complexity and multi-site acquisition heterogeneity in benchmarks such as Kaggle Data Science Bowl 2017 (DSB 2017).
Methods: We present HybridSeg-Net comprising: (1) 3D-HTDSeg with novel Attention-Modulated Dense Skip (AMDS) connections applying axial self-attention transformer blocks directly onto skip tensors; (2) ACaps-LSTM combining adaptive dynamic routing capsule encoding with bidirectional LSTM; (3) IQHO — Intensity-Guided Quantum-Inspired Hawk Optimization for joint hyperparameter tuning. Evaluated on Kaggle Data Science Bowl 2017 (DSB 2017) (1,930 CT studies, 70/10/20 split).
Results: 3D-HTDSeg: DSC=0.9738, IoU=0.9612, HD95=1.74 mm (beats UNETR DSC=0.9523). ACaps-LSTM+IQHO: 97.41% accuracy, 97.32% sensitivity, AUC=0.981 (beats IF-RTH-ADNet-LSTM 95.62% by +1.79 pp). 5-fold CV: 97.08+/-0.39% (p<0.005).