MADE: Masked Autoencoder for Distribution Estimation

Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle
Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:881-889, 2015.

Abstract

There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder’s parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.

Cite this Paper

BibTeX
@InProceedings{pmlr-v37-germain15, title = {MADE: Masked Autoencoder for Distribution Estimation}, author = {Germain, Mathieu and Gregor, Karol and Murray, Iain and Larochelle, Hugo}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning}, pages = {881--889}, year = {2015}, editor = {Bach, Francis and Blei, David}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/germain15.pdf}, url = {https://proceedings.mlr.press/v37/germain15.html}, abstract = {There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder’s parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.} }
Endnote
%0 Conference Paper %T MADE: Masked Autoencoder for Distribution Estimation %A Mathieu Germain %A Karol Gregor %A Iain Murray %A Hugo Larochelle %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-germain15 %I PMLR %P 881--889 %U https://proceedings.mlr.press/v37/germain15.html %V 37 %X There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder’s parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.
RIS
TY - CPAPER TI - MADE: Masked Autoencoder for Distribution Estimation AU - Mathieu Germain AU - Karol Gregor AU - Iain Murray AU - Hugo Larochelle BT - Proceedings of the 32nd International Conference on Machine Learning DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-germain15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 37 SP - 881 EP - 889 L1 - http://proceedings.mlr.press/v37/germain15.pdf UR - https://proceedings.mlr.press/v37/germain15.html AB - There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder’s parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators. ER -
APA
Germain, M., Gregor, K., Murray, I. & Larochelle, H.. (2015). MADE: Masked Autoencoder for Distribution Estimation. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:881-889 Available from https://proceedings.mlr.press/v37/germain15.html.

Related Material