research-article

Fully Binary Neural Network Model and Optimized Hardware Architectures for Associative Memories

Authors:

Philippe Coussy,

Cyrille Chavet,

Hugues Nono Wouafo,

Laura Conde-CanenciaAuthors Info & Claims

ACM Journal on Emerging Technologies in Computing Systems (JETC), Volume 11, Issue 4

Article No.: 35, Pages 1 - 23

https://doi.org/10.1145/2629510

Published: 27 April 2015 Publication History

Abstract

Brain processes information through a complex hierarchical associative memory organization that is distributed across a complex neural network. The GBNN associative memory model has recently been proposed as a new class of recurrent clustered neural network that presents higher efficiency than the classical models. In this article, we propose computational simplifications and architectural optimizations of the original GBNN. This work leads to significant complexity and area reduction without affecting neither memorizing nor retrieving performance. The obtained results open new perspectives in the design of neuromorphic hardware to support large-scale general-purpose neural algorithms.

References

[1]

L. F. Abbott and S. B. Nelson. 2000. Synaptic plasticity: Taming the beast. Nature Neurosci. 3. 6 pages.

[2]

D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognit. Sci. 9, 1, 147--169.

[3]

A. Annovi, G. Broccolo, A. Ciocci, et al. 2013. Associative memory for L1 track triggering in LHC environment. IEEE Trans. Nucl. Sci. 60, 5, part 2, 3627--3632.

[4]

R. Brette, M. Rudolph, T. Carnevale, et al. 2007. Simulation of networks of spiking neurons: A review of tools and strategies. J. Computat. Neurosci. 23, 349--398.

[5]

C. Chavet, P. Coussy, and N. Charpentier. 2012. Architecture de réseau de neurone, procédé d'obtention et programmes correspondants. Patent n° 1261155.

[6]

P. Frost Gorder. 2008. Computer vision, inspired by the human brain. IEEE J. Comput. Sci. Eng. 10, 2.

Digital Library

[7]

S. Furber and S. Temple. 2007. Neural systems engineering. J. Roy. Soc. Inter. 4.

[8]

T. Giotis, M. A. Christodoulou, and Y. Boutalis. 2011. Identification of combination therapy models using a neuro fuzzy identification scheme. In Proceedings of the 19^th Mediterranean Conference on Control & Automation, 1283--1288.

[9]

R. Granger. 2006. Essential circuits of cognition: The brain's basic operations, architecture, and representations. In AI at 50: The Future of Artificial Intelligence.

[10]

V. Gripon and C. Berrou. 2011. Sparse neural networks with large learning diversity. IEEE Trans. Neural Netw. 22, 7, 1087--1096.

Digital Library

[11]

V. Gripon and C. Berrou. 2012. Nearly-optimal associative memories based on distributed constant weight codes. In Proceedings of Information Theory and Applications Workshop. 269--273.

[12]

V. Gripon, V. Skachek, W. J. Gross, and M. Rabbat. 2012. Random clique codes. In Proceedings of the 7th International Symposium on Turbo Codes and Iterative Information Processing. 121--125.

[13]

J. Hawkins and S. Blakeslee. 2004. On Intelligence. Times Books.

Digital Library

[14]

D. O. Hebb. 1949. The Organization of Behavior. Wiley, New York.

[15]

R. Hecht-Nielsen. 2007. Confabulation Theory.Springer-Verlag, Heidelberg.

Digital Library

[16]

J. Hopfield. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79, 8, 2554--2558.

[17]

A. Jarollahi, N. Onizawa, V. Gripon, and W. J. Gross. 2012. Architecture and implementation of an associative memory using sparse clustered networks. In Proceedings of the IEEE International Symposium on Circuits and Systems.

[18]

A. Jarollahi, V. Gripon, N. Onizawa, and W. J. Gross. 2013. A low-power content-addressable-memory based on clustered-sparse-networks. In Proceedings of the 24^th International Conference on Application-Specific Systems, Architectures and Processors.

Digital Library

[19]

H. Jhuang, T. Serre, L. Wolf, and T. Poggio. 2008. Biologically inspired system for action recognition. In Proceedings of the 11^th IEEE International Conference on Computer Vision, 1--8.

[20]

H. Jiping, M. G. Maltenfort, W. Qingjun, and T. M. Hamm. 2001. Learning from biological systems: Modeling neural control. IEEE J. Cont. Syst. 21, 4.

[21]

E. Kandel, J. Schwartz, T. Jessell, S. Siegelbaum, and A. J. Hudspeth. 2013. Principles of Neural Science 5^th Ed. McGraw-Hill Ryerson.

[22]

T. Kohonen. 1977. Associative Memory. Springer, New York.

[23]

J. E. Laird and Y. Wang. 2007. The importance of action history in decision making and reinforcement learning. In Proceedings of the 8^th International Conference on Cognitive Modeling.

[24]

C. Mead. 1990. Neuromorphic electronic systems. Proc. IEEE 78, 1629--1636.

[25]

NAE Grand Challenges for Engineering. 2013. www.engineeringchallenges.org.

[26]

J. M. Nageswaran, M. Richert, N. Dutt, and J. L. Krichmar. 2010. Towards reverse engineering the brain: Modeling abstractions and simulation frameworks. In Proceedings of the IEEE/IFIP International Conference on VLSI and System-on-Chip.

[27]

G. Palm. 1980. On associative memories. Biol. Cybernet. 36, 19--31.

[28]

G. Palm. 2013. Neural associative memories and sparse coding. Neural Netw. Springer.

Digital Library

[29]

R. Perfetti and E. Ricci. 2008. Recurrent correlation associative memories: A feature space perspective. IEEE Trans. Neural Netw. 19, 2, 333--345.

Digital Library

[30]

Q. Qiu, Q. Wu, M. Bishop, R. E. Pino, and R. W. Linderman. 2013. A parallel neuromorphic text recognition system and its implementation on a heterogeneous high-performance computing cluster. IEEE Trans. Comput. 62, 5, 886--899.

Digital Library

[31]

A. Sandberg. 2003. Bayesian attractor neural network models of memory. Ph.D. dissertation. Stockholm University.

[32]

H. Simon. 2010. Cognitive Science: Relationship of AI to Psychology and Neuroscience. AAAI.

[33]

L. Soo-Young. 2007. Artificial brain based on brain-inspired algorithms for human-like intelligent functions. In Proceedings of the IEEE International Conference on Integration Technology.

[34]

A. Sudo, A. Sato, and O. Hasegawa. 2009. Associative memory for online learning in noisy environments using self-organizing incremental neural network. IEEE Neural Netw. 20, 6, 964--972.

Digital Library

[35]

E. Theordorou and F. J. Valero-Cuevas. 2010. Optimality in neuromuscular systems. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 4510--4516.

[36]

D. Willshaw. 1971. Models of distributed associative memory. Ph.D. dissertation, University of Edinburgh.

[37]

L. Wiskott and T. J. Sejnowski. 2002. Slow feature analysis: Unsupervised learning of invariances. J. Neural Computat. 14, 4, 55 pages.

Digital Library

Cited By

Sarkar MChowdhury SWalling JYi C(2024)An In-Memory Power Efficient Computing Architecture with Emerging VGSOT MRAM Device2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10557835(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10557835
Dick RShang LWolf MYang S(2020)Embedded Intelligence in the Internet-of-ThingsIEEE Design & Test10.1109/MDAT.2019.295735237:1(7-27)Online publication date: Feb-2020
https://doi.org/10.1109/MDAT.2019.2957352
Wouafo HChavet CCoussy P(2019)Clone-Based Encoded Neural Networks to Design Efficient Associative MemoriesIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2018.289065830:10(3186-3199)Online publication date: Oct-2019
https://doi.org/10.1109/TNNLS.2018.2890658
Show More Cited By

Index Terms

Fully Binary Neural Network Model and Optimized Hardware Architectures for Associative Memories
1. Computer systems organization
  1. Architectures
    1. Other architectures
      1. Neural networks
2. Hardware
  1. Very large scale integration design
    1. Application-specific VLSI designs
      1. Application specific processors

Recommendations

Analysis and design of associative memories based on recurrent neural network with discontinuous activation functions

This paper considers a recurrent neural network (RNN) with a special class of discontinuous activation function which is piecewise constants in the state space. One sufficient condition is established to ensure that the novel recurrent neural networks ...
Associative memory for online learning in noisy environments using self-organizing incremental neural network

Associative memory operating in a real environment must perform well in online incremental learning and be robust to noisy data because noisy associative patterns are presented sequentially in a real environment.We propose a novel associative memory ...
Binary Associative Memories with Complemented Operations
Automation and Communication Systems for Autonomous Platforms (Special section, pp. 171-218), Zygmunt Kitowski, Paweł Piskur and Stanisław Hożyń (Eds.)
Abstract
Associative memories based on lattice algebra are of great interest in pattern recognition applications due to their excellent storage and recall properties. In this paper, a class of binary associative memory derived from lattice memories is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Journal on Emerging Technologies in Computing Systems

ACM Journal on Emerging Technologies in Computing Systems Volume 11, Issue 4

Special Issues on Neuromorphic Computing and Emerging Many-Core Systems for Exascale Computing

April 2015

231 pages

ISSN:1550-4832

EISSN:1550-4840

DOI:10.1145/2767119

Editor:
Krishnendu Chakrabarty
Duke University, USA

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

ACM Journals for the Design of Smart and Connected Systems

Publication History

Published: 27 April 2015

Accepted: 01 April 2014

Revised: 01 November 2013

Received: 01 August 2013

Published in JETC Volume 11, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
376
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sarkar MChowdhury SWalling JYi C(2024)An In-Memory Power Efficient Computing Architecture with Emerging VGSOT MRAM Device2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10557835(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10557835
Dick RShang LWolf MYang S(2020)Embedded Intelligence in the Internet-of-ThingsIEEE Design & Test10.1109/MDAT.2019.295735237:1(7-27)Online publication date: Feb-2020
https://doi.org/10.1109/MDAT.2019.2957352
Wouafo HChavet CCoussy P(2019)Clone-Based Encoded Neural Networks to Design Efficient Associative MemoriesIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2018.289065830:10(3186-3199)Online publication date: Oct-2019
https://doi.org/10.1109/TNNLS.2018.2890658
Ling AAbdelfattah MO'Connell SBitar AHan DDicecco RSubhaschandra SJohnson CDenisenko DFender JChiu G(2018)Harnessing Numerical Flexibility for Deep Learning on FPGAsProceedings of the 9th International Symposium on Highly-Efficient Accelerators and Reconfigurable Technologies10.1145/3241793.3241794(1-3)Online publication date: 20-Jun-2018
https://dl.acm.org/doi/10.1145/3241793.3241794
Wouafo HChavet CCoussy PDanilo R(2017)Efficient scalable hardware architecture for highly performant encoded neural networks2017 IEEE International Workshop on Signal Processing Systems (SiPS)10.1109/SiPS.2017.8109986(1-6)Online publication date: Oct-2017
https://doi.org/10.1109/SiPS.2017.8109986
Danilo RWouafo HChavet CGripon VConde-Canencia LCoussy P(2016)Associative Memory based on clustered Neural Networks: Improved model and architecture for Oriented Edge Detection2016 Conference on Design and Architectures for Signal and Image Processing (DASIP)10.1109/DASIP.2016.7853796(51-58)Online publication date: Oct-2016
https://doi.org/10.1109/DASIP.2016.7853796
Wouafo HChavet CCoussy P(2015)Improving storage of patterns in recurrent neural networks: Clone-based model and architecture2015 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS.2015.7168699(577-580)Online publication date: May-2015
https://doi.org/10.1109/ISCAS.2015.7168699

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents