Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers †
1
Department of Neuroscience, Columbia University Irving Medical Center, New York, NY 10032, USA
2
Department of Systems Biology, Columbia University Irving Medical Center, New York, NY 10032, USA
†
This paper is an extended version of our paper published in International Workshop on Human Brain and Artificial Intelligence 2019.
Academic Editors: Shu-Tao Xia and Bin Chen
Entropy 2022, 24(1), 59; https://doi.org/10.3390/e24010059
Received: 18 November 2021
/
Revised: 20 December 2021
/
Accepted: 23 December 2021
/
Published: 28 December 2021
(This article belongs to the Special Issue Information Theory and Deep Neural Networks)
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, the regularity normalization constrains the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrate the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in image classification, classic control, procedurally-generated reinforcement learning, generative modeling, handwriting generation and question answering tasks with various neural network architectures. Lastly, the unsupervised attention mechanisms is a useful probing tool for neural networks by tracking the dependency and critical learning stages across layers and recurrent time steps of deep networks.
View Full-Text
Keywords:
neuronal coding; biologically plausible models; minimum description length; deep neural networks; normalization methods
▼
Show Figures
Figure 1
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited
MDPI and ACS Style
Lin, B. Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers. Entropy 2022, 24, 59. https://doi.org/10.3390/e24010059
AMA Style
Lin B. Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers. Entropy. 2022; 24(1):59. https://doi.org/10.3390/e24010059
Chicago/Turabian StyleLin, Baihan. 2022. "Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers" Entropy 24, no. 1: 59. https://doi.org/10.3390/e24010059
Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.
