Stochastic Adaptive Linear Unit: Stability and Efficiency in Deep Neural Networks

Abstract

Activation functions are an essential part of deep neural networks. The nature of the activation functions affects the convergence, stability, and generalization of neural networks. Even with significant improvements that were introduced using the activation functions such as ReLU, ELU, Mish, Swish, and GELU, challenges such as gradient explosion, gradient vanishing, and numerical instability are still persistent in deep neural networks. To address the limitations of the current activation functions, we present a novel activation function referred to as Stochastic Adaptive Linear Unit (SALU). It utilizes the damping coefficient (γ) and the stochastic variance (σ).
Based on theoretical analysis, we observe that the presented activation function (SALU) guarantees the bounded gradients and stable convergence even when applied to neural networks of more than 700 layers. Tests performed on the datasets such as CIFAR 10 and CIFAR 100 using models including VGG16, ResNet50, and MobileNet demonstrate that our new activation function SALU outperforms all classical and recent activation functions (ReLU, LeakyReLU, ELU, Mish, Swish, GELU, S2LU).
Moreover, the efficiency analysis highlights the efficiency of SALU because it is known for competitive training time and latency in activation, offering a reasonable balance between computational expenses and efficiency. Cross-dataset consistency and architecture independence ensure the reliability of SALU as an activation function for small models used on mobile devices and for deep architectures.
In terms of the future perspective, there are many promising research directions offered by SALU, including hybrid stochastic activation functions, using SALU in transformer models, testing the application of SALU in large image datasets, such as ImageNet, and hardware-aware implementation to facilitate efficient real-time usage. The strong theoretical foundations and practical advantages of SALU make it an integral part of the further development of deep learning techniques.