Multimodal Deep Learning for Intelligent Condition Monitoring Using Vibration, Acoustic and Thermal Data

Abstract

Industrial condition monitoring increasingly relies on heterogeneous sensors because no single modality completely captures the mechanical, acoustic, and thermal signatures of developing faults. Vibration signals are highly sensitive to impacts, resonance, imbalance, and misalignment; acoustic data provide non-contact evidence of friction, leakage, looseness, and impulsive events; and infrared thermal images reveal heat accumulation, abnormal resistance, lubrication loss, and overload. This study presents a multimodal deep-learning framework that integrates these three sources through modality-specific convolutional encoders and a sample-wise reliability gate. Each vibration and acoustic window is processed by a compact one-dimensional convolutional neural network, while thermal maps are processed by a two-dimensional convolutional branch. The resulting embeddings are fused through softmax-normalized modality weights and a shared representation before four-class diagnosis. A reproducible synthetic benchmark containing 1,200 observations was generated for healthy operation, bearing fault, misalignment, and overheating. The data were divided into 840 training, 180 validation, and 180 test observations; the principal test set included an additional noise shift to approximate sensor degradation and field variability. The proposed attentive fusion network achieved 99.44% accuracy and a macro F1-score of 0.9944, outperforming vibration-only (76.11%), acoustic-only (88.89%), thermal-only (54.44%), and late probability fusion (82.22%) baselines. Noise and missing-modality tests showed that performance declined as corruption increased and that vibration and thermal information were especially important in the present synthetic design. The results demonstrate the methodological value of reliability-aware intermediate fusion, while also emphasizing that synthetic evidence is not a substitute for industrial validation. The paper provides an implementation-oriented architecture, evaluation protocol, deployment blueprint, and research agenda for robust multimodal predictive maintenance.