Evaluating the Feasibility of Llama 3.2:3b for Automating Bone Scan Report Structuring

Abstract

Background: Pediatric bone scan reports often contain rich clinical information embedded within lengthy, unstructured narratives, a known challenge in radiology NLP systems [1,2]. Due to inconsistent formatting and variable reporting styles, valuable diagnostic details frequently remain hidden or overlooked, posing major challenges for data integration, research analytics, and automated quality control in nuclear medicine. This growing need for structured, standardized documentation has led to increasing interest in the use of large language models (LLMs) for automated report structuring [3,4].
Objective: This study evaluates the feasibility of locally deployed LLMs for converting unstructured pediatric bone scan reports into structured, machine-readable formats without compromising data privacy, an increasingly important concern in clinical AI applications [5].
Methods: A dataset of 134 de-identified 99mTc-MDP pediatric bone scan reports was analyzed using A dataset of 134 de-identified 99mTc-MDP pediatric bone scan reports was analyzed using the Llama 3.2:3B model under Zero-Shot (ZS), Few-Shot (FS), and Chain-of-Thought (CoT) prompting configurations, reflecting established prompting paradigms in LLM research [6–8]. All experiments were performed through the Ollama framework, ensuring deterministic, on-device inference. Model outputs were benchmarked against expert-annotated ground truth using sensitivity, specificity, PPV, NPV, accuracy, and F1-score, metrics widely used in clinical NLP evaluation [9,10].
Results: The ZS model demonstrated high specificity and NPV (>0.90), indicating strong “rule-out” capability but reduced sensitivity for rare positive findings, consistent with expected behavior under class-imbalanced clinical settings [11]. The FS model achieved balanced improvements in diagnostic extraction (F1 up to 0.91) despite class imbalance. Conversely, CoT prompting amplified conservative bias, increasing false negatives, a known limitation of certain reasoning-style prompts [12].
Conclusion: Local LLM deployment proved technically feasible and clinically valuable, enabling high-throughput, privacy-preserving structuring of pediatric bone scan reports and aligning with current recommendations for privacy-respecting AI in healthcare [5]. These findings establish a reproducible framework for secure, low-cost AI integration into clinical nuclear medicine workflows.