Constraining Implicit Space with MDL: Regularity Normalization as Unsupervised Attention
release_in4kcsasnzhzhds2y5ou277lh4
by
Baihan Lin
2020
Abstract
Inspired by the adaptation phenomenon of neuronal firing, we propose the
regularity normalization (RN) as an unsupervised attention mechanism (UAM)
which computes the statistical regularity in the implicit space of neural
networks under the Minimum Description Length (MDL) principle. Treating the
neural network optimization process as a partially observable model selection
problem, UAM constrains the implicit space by a normalization factor, the
universal code length. We compute this universal code incrementally across
neural network layers and demonstrated the flexibility to include data priors
such as top-down attention and other oracle information. Empirically, our
approach outperforms existing normalization methods in tackling limited,
imbalanced and non-stationary input distribution in image classification,
classic control, procedurally-generated reinforcement learning, generative
modeling, handwriting generation and question answering tasks with various
neural network architectures. Lastly, UAM tracks dependency and critical
learning stages across layers and recurrent time steps of deep networks.
In text/plain
format
Archived Files and Locations
application/pdf 6.9 MB
file_v6x2y2j4cjgavbt7eswj2pfkci
|
arxiv.org (repository) web.archive.org (webarchive) |
1902.10658v11
access all versions, variants, and formats of this works (eg, pre-prints)