Remember Everything: The Weight Spectrum of Neural Networks
PyTorch
The article demonstrates that information about training is preserved in the weight spectrum of a neural network. Even after filtering and posterization, the data is not completely erased, allowing fine-tuning to use these coefficients as initial conditions. Additionally, the posterized spectral coefficients take up significantly less space, which is useful for neural network design.
The article by a Habr author explores the spectral properties of neural network weights. They create a simple neural network in Colab to classify rectangles and squares, generating random rectangles with 8 vertex coordinates as input and binary labels. The model has two linear layers with ReLU activation, trained for 10 epochs achieving over 99% accuracy. After training, the weights are analyzed using the discrete Fourier transform (DFT). The author finds that after applying a smoothing filter and posterization to the spectral coefficients, the essential information about training remains, which can be used for fine-tuning as initial conditions. The posterized spectral representation occupies significantly less space than the original weights, offering a potential compression technique. The author suggests this effect is interesting for neural network design and provides the code in md-summaries as prompts for AI agents.
- Сокращения
- DFT = Discrete Fourier Transform — Дискретное преобразование Фурье
- ReLU = Rectified Linear Unit — Выпрямленная линейная единица
Source: Хабр — Data Mining —
original