Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers
release_bb7helcugrehddai22n3zwtfgu
by
Baihan Lin
2021
Abstract
Inspired by the adaptation phenomenon of neuronal firing, we propose the
regularity normalization (RN) as an unsupervised attention mechanism (UAM)
which computes the statistical regularity in the implicit space of neural
networks under the Minimum Description Length (MDL) principle. Treating the
neural network optimization process as a partially observable model selection
problem, the regularity normalization constrains the implicit space by a
normalization factor, the universal code length. We compute this universal code
incrementally across neural network layers and demonstrate the flexibility to
include data priors such as top-down attention and other oracle information.
Empirically, our approach outperforms existing normalization methods in
tackling limited, imbalanced and non-stationary input distribution in image
classification, classic control, procedurally-generated reinforcement learning,
generative modeling, handwriting generation and question answering tasks with
various neural network architectures. Lastly, the unsupervised attention
mechanisms is a useful probing tool for neural networks by tracking the
dependency and critical learning stages across layers and recurrent time steps
of deep networks.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
1902.10658v13
access all versions, variants, and formats of this works (eg, pre-prints)