A Comprehensive Review of OAM-Based Free-Space Optical Communication System Design, Atmospheric Challenges and Emerging Techniques

Abstract

Free-space optical (FSO) communication has emerged as a promising high-bandwidth, license-free alternative to conventional radio frequency and fiber-optic systems, and orbital angular momentum (OAM) multiplexing using Laguerre–Gaussian (LG) modes offers a powerful means of scaling FSO capacity by exploiting the spatial dimension of light. However, practical deployment of OAM multiplexed FSO links is hindered by atmospheric impairments such as fog, haze, rain, dust, and turbulence-induced wavefront distortion, all of which cause beam spreading, scintillation, power loss, and inter-modal crosstalk that degrade system performance. This review provides a comprehensive analysis of OAM-based FSO communication by examining the theoretical foundations of LG beam generation and propagation, key performance metrics including channel capacity, bit error rate, and spectral efficiency, and the atmospheric channel models that govern link reliability. Recent advances in modulation format optimization, hybrid multiplexing strategies combining OAM with (WDM, PDM) and (OCDMA) adaptive optics, MIMO equalization, machine-learning-based mode recognition and relay-assisted architectures are critically assessed through a comparative evaluation of representative studies. Finally critical research gaps are identified, including the absence of unified crosstalk models under realistic turbulence, limited real-weather experimental validation and unresolved trade-offs between beam divergence, mode purity and alignment tolerance, and future directions are outlined to support the development of robust, scalable OAM-FSO systems for next-generation wireless and optical networks.