Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
fatcat! About Guide Changelog
Login/Signup

Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers release_rev_6affd87e-2c1a-41ac-abc6-c113f87019f1

by Baihan Lin

Entity Metadata (schema)

abstracts[] {'sha1': 'bee88ed307afc0967641e2e5f6a014e44d688131', 'content': 'Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, the regularity normalization constrains the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrate the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in image classification, classic control, procedurally-generated reinforcement learning, generative modeling, handwriting generation and question answering tasks with various neural network architectures. Lastly, the unsupervised attention mechanisms is a useful probing tool for neural networks by tracking the dependency and critical learning stages across layers and recurrent time steps of deep networks.', 'mimetype': 'application/xml+jats', 'lang': None}
container {'state': 'active', 'ident': '4d3elkqvznfzho6ki7a35bt47u', 'revision': '947a4cda-25c3-4ddf-84f6-fb9c91213992', 'redirect': None, 'extra': {'abbrev': 'Entropy (Basel)', 'country': 'ch', 'default_license': 'CC-BY', 'doaj': {'archive': ['CLOCKSS'], 'as_of': '2022-07-06', 'default_license': 'CC-BY', 'seal': True}, 'kbart': {'clockss': {'year_spans': [[1999, 2022]]}}, 'languages': ['en'], 'publisher_type': 'oa', 'road': {'as_of': '2018-01-24'}, 'sherpa_romeo': {'color': 'green'}, 'urls': ['http://www.e-helvetica.nb.admin.ch/directAccess?callnumber=bel-155606', 'https://www.mdpi.com/journal/entropy', 'http://nbn-resolving.de/urn/resolver.pl?urn=urn:nbn:ch:bel-155606']}, 'edit_extra': None, 'name': 'Entropy', 'container_type': None, 'publication_status': None, 'publisher': 'MDPI AG', 'issnl': '1099-4300', 'issne': '1099-4300', 'issnp': None, 'wikidata_qid': 'Q3011618'}
container_id 4d3elkqvznfzho6ki7a35bt47u
contribs[] {'index': 0, 'creator_id': 'nqmyekhdwbavrjkbmgcujfxyvy', 'creator': None, 'raw_name': 'Baihan Lin', 'given_name': 'Baihan', 'surname': 'Lin', 'role': 'author', 'raw_affiliation': None, 'extra': {'seq': 'first'}}
ext_ids {'doi': '10.3390/e24010059', 'wikidata_qid': None, 'isbn13': None, 'pmid': '35052085', 'pmcid': 'PMC8774926', 'core': None, 'arxiv': None, 'jstor': None, 'ark': None, 'mag': None, 'doaj': None, 'dblp': None, 'oai': None, 'hdl': None}
files
filesets
issue 1
language en
license_slug CC-BY
number
original_title
pages 59
publisher MDPI AG
refs[] {'index': 0, 'target_release_id': 'cv36wghz7zhahlwiv6jraxftza', 'extra': {'pmid': '33434123', 'unstructured': 'IEEE Trans Pattern Anal Mach Intell. 2021 Jan 12;PP:'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 1, 'target_release_id': 'cw4upsrbhfb4jbpjd5sw4vlpqa', 'extra': {'pmid': '29073108', 'unstructured': 'Proc Natl Acad Sci U S A. 2017 Oct 24;114(43):E9115-E9124'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 2, 'target_release_id': 'yikwc4h5yvfj7gwzlimtw5n6ai', 'extra': {'pmid': '27683554', 'unstructured': 'Front Comput Neurosci. 2016 Sep 14;10:94'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 3, 'target_release_id': 'fsoyqta4ozhfrkavoeycr3selu', 'extra': {'pmid': '32275607', 'unstructured': 'IEEE Trans Neural Netw Learn Syst. 2021 Feb;32(2):663-674'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 4, 'target_release_id': 'li2grnvddzg53cu5dp5ncsfg5i', 'extra': {'pmid': '30738835', 'unstructured': 'Prog Neurobiol. 2019 Apr;175:126-137'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 5, 'target_release_id': 'opxh36rdazawtipyqpkurgjjtu', 'extra': {'pmid': '31329548', 'unstructured': 'IEEE Trans Pattern Anal Mach Intell. 2022 Feb;44(2):636-647'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 6, 'target_release_id': 'mjlyu2ptdrdkvjtrcybjarbmku', 'extra': {'pmid': '28728020', 'unstructured': 'Neuron. 2017 Jul 19;95(2):245-258'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 7, 'target_release_id': 'telj2axt7zef5oo2pjq3554hqm', 'extra': {'pmid': '10035458', 'unstructured': 'Phys Rev Lett. 1987 Nov 9;59(19):2229-2232'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 8, 'target_release_id': 'qxgf3nz2tngthbyperlz2mhf2i', 'extra': {'pmid': '5761934', 'unstructured': 'J Physiol. 1969 Jan;200(1):11P-13P'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 9, 'target_release_id': 'wjw5ewsfsfbprgx42yudsrxsse', 'extra': {'pmid': '25247371', 'unstructured': 'Curr Biol. 2014 Sep 22;24(18):R921-R929'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 10, 'target_release_id': 'bk2zayqifva33bzyhvbl7hawme', 'extra': {'pmid': '32663439', 'unstructured': 'Neuron. 2020 Aug 19;107(4):603-616'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
{'index': 11, 'target_release_id': 'lif42luawfenpmpynvn2d6hn4y', 'extra': {'pmid': '11087001', 'unstructured': 'Neuron. 2000 Oct;28(1):287-98'}, 'key': None, 'year': None, 'container_name': None, 'title': None, 'locator': None}
release_date 2021-12-28
release_stage published
release_type article-journal
release_year 2021
subtitle
title Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers
version
volume 24
webcaptures
withdrawn_date
withdrawn_status
withdrawn_year
work_id trtxd5ux6ffltdzvib6sptlfhm

Extra Metadata (raw JSON)

crossref.alternative-id ['e24010059']
crossref.license [{'URL': 'https://creativecommons.org/licenses/by/4.0/', 'content-version': 'vor', 'delay-in-days': 0, 'start': '2021-12-28T00:00:00Z'}]
crossref.subject ['General Physics and Astronomy']
crossref.type journal-article
pubmed.pub_types ['Journal Article']