Kazry: A New Piecewise Activation Function for Neural Networks
A new neural network activation function called Kazry was presented, which outperformed SiLU in tests, offering improved signal propagation and faster convergence. Kazry and its modification KazGLU showed better loss values on CIFAR-100 in both CNN and Transformer architectures, with only a slight increase in training time compared to SiLU and SwiGLU.
An article on Habr introduces Kazry, a new piecewise activation function for neural networks. The author argues that industry-standard activations like SiLU, while infinitely differentiable, have a derivative that dampens the signal, slowing convergence. Kazry addresses this by using a piecewise formula with negative and positive branches, passing the signal unchanged for non-negative inputs. Tests on CIFAR-100 with CNN and Transformer models (the latter using GLU modifications and NoPE position encoding) showed Kazry achieved lower training and validation loss compared to SiLU, and KazGLU outperformed SwiGLU, with only marginal increases in training time. The article notes potential drawbacks of Kazry: a discontinuity in the second derivative at zero, which may affect more sensitive optimizers, and slightly slower execution speed due to logical branching in code, despite mathematically fewer operations. Source code and training logs are available on GitHub.
- Abbreviations
- CNN = Convolutional Neural Network — Сверточная нейронная сеть
- GLU = Gated Linear Unit — стробированный линейный блок
- GPU = Graphics Processing Unit — графический процессор
- SGD = Stochastic Gradient Descent — стохастический градиентный спуск
Source: Habr — хаб ИИ —
original