GPU-Accelerated Edge Inference for COVID-19 Chest X-Ray Classification: A Systematic Benchmarking of CNN Architectures on the NVIDIA Jetson Orin Nano with TensorRT FP16 and INT8 Optimisation

Abstract

Background: Convolutional neural networks (CNNs) have demonstrated strong diagnostic performance for automated COVID-19 classification from chest X-ray (CXR) images; however, published studies evaluate inference almost exclusively on server-grade GPU hardware that is economically and logistically incompatible with point-of-care deployment in resource-constrained clinical settings. GPU-accelerated embedded devices offer a viable alternative, yet a systematic characterisation of CNN inference performance, power consumption, and diagnostic accuracy across multiple precision modes on these platforms remains absent from the medical imaging literature.
Methods: This study presents a systematic benchmarking evaluation of four representative CNN architectures — ResNet18, ResNet50, DenseNet121, and SqueezeNet — deployed on the NVIDIA Jetson Orin Nano (1024-core Ampere GPU, 32 Tensor Cores, 8 GB LPDDR5, <USD 200) for binary COVID-19 CXR classification using the COVID-Xray-5k dataset (5,000 images). Diagnostic performance was assessed via 5-fold stratified cross-validation on an independent 918-image held-out test partition. All architectures were characterised in three precision modes: FP32 baseline, TensorRT FP16 half-precision, and INT8 post-training quantisation (PTQ). Inference latency (mean, P50, P95, σ), throughput (FPS), peak GPU memory, and board-level power were profiled in MAXN mode via tegrastats at 100 ms polling intervals.
Results: All four architectures achieved AUC ≥ 0.978, with diagnostic accuracy fully preserved under FP16 and INT8 TRT (ΔAUC < 0.005). For ResNet18: FP32 latency was 15.7 ms (P50 = 16.0 ms, P95 = 20.1 ms, 63.8 FPS, 7.4 W, 8.6 inf/J); FP16 TRT was 10.8 ms (92.6 FPS, 12.4 inf/J); INT8 TRT was 7.2 ms (138.9 FPS, 6.9 W, 20.1 inf/J). SqueezeNet INT8 TRT achieved up to 909 FPS at approximately 2.6 MB peak GPU memory. FP32 inference was 856–2,178× faster than Raspberry Pi 4 CPU; TensorRT FP16 extended speedups to 1,244–5,290×. A novel empirical finding demonstrates that DenseNet121 exhibits higher FP32 latency than ResNet50 despite 3× fewer parameters, attributable to dense connectivity serialising LPDDR5 memory access.
Conclusions: TensorRT-optimised CNN inference on the Jetson Orin Nano enables real-time CXR screening within a 6.9–7.4 W average power budget. The reproducible three-precision-mode deployment pipeline and hardware-aware architecture selection criteria presented here provide a rigorous empirical basis for deploying CNN-based diagnostics on GPU-accelerated embedded hardware in resource-constrained healthcare settings without cloud infrastructure dependency.