research-article

Public Access

Composable Probabilistic Inference Networks Using MRAM-based Stochastic Neurons

Authors:

Kerem Y. Camsari,

Ronald F. DemaraAuthors Info & Claims

ACM Journal on Emerging Technologies in Computing Systems (JETC), Volume 15, Issue 2

Article No.: 17, Pages 1 - 22

https://doi.org/10.1145/3304105

Published: 26 March 2019 Publication History

All formats PDF

Abstract

Magnetoresistive random access memory (MRAM) technologies with thermally unstable nanomagnets are leveraged to develop an intrinsic stochastic neuron as a building block for restricted Boltzmann machines (RBMs) to form deep belief networks (DBNs). The embedded MRAM-based neuron is modeled using precise physics equations. The simulation results exhibit the desired sigmoidal relation between the input voltages and probability of the output state. A probabilistic inference network simulator (PIN-Sim) is developed to realize a circuit-level model of an RBM utilizing resistive crossbar arrays along with differential amplifiers to implement the positive and negative weight values. The PIN-Sim is composed of five main blocks to train a DBN, evaluate its accuracy, and measure its power consumption. The MNIST dataset is leveraged to investigate the energy and accuracy tradeoffs of seven distinct network topologies in SPICE using the 14nm HP-FinFET technology library with the nominal voltage of 0.8V, in which an MRAM-based neuron is used as the activation function. The software and hardware level simulations indicate that a 784× 200× 10 topology can achieve less than 5% error rates with ∼400pJ energy consumption. The error rates can be reduced to 2.5% by using a 784× 500× 500× 500× 10 DBN at the cost of ∼10× higher energy consumption and significant area overhead. Finally, the effects of specific hardware-level parameters on power dissipation and accuracy tradeoffs are identified via the developed PIN-Sim framework.

References

[1]

{n.d.}. Predictive Technology Model (PTM). Retrieved from http://ptm.asu.edu/.

[2]

David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cogn. Sci. 9, 1 (1985), 147--169.

[3]

A. Ardakani, F. Leduc-Primeau, N. Onizawa, T. Hanyu, and W. J. Gross. 2017. VLSI implementation of deep neural network using integral stochastic computing. IEEE Trans. VLSI Syst. 25, 10 (2017).

Digital Library

[4]

Mukund Bapna and Sara A. Majetich. 2017. Current control of time-averaged magnetization in superparamagnetic tunnel junctions. Appl. Phys. Lett. 111, 24 (2017), 243107.

[5]

I. A. Basheer and M. Hajmeer. 2000. Artificial neural networks: Fundamentals, computing, design, and application. J. Microbiol. Methods 43, 1 (2000), 3--31.

[6]

Behtash Behin-Aein, Vinh Diep, and Supriyo Datta. 2016. A building block for hardware belief networks. Sci. Rep. 6, Article 29893 (2016).

[7]

Sabpreet Bhatti, Rachid Sbiaa, Atsufumi Hirohata, Hideo Ohno, Shunsuke Fukami, and SN Piramanayagam. 2017. Spintronics based random access memory: A review. Materials Today 20, 9 (2017), 530--548. http://www.sciencedirect.com/science/article/pii/S1369702117304285.

[8]

Chris Bishop, Christopher M. Bishop, et al. 1995. Neural Networks for Pattern Recognition. Oxford University Press.

Digital Library

[9]

M. N. Bojnordi and E. Ipek. 2016. Memristive Boltzmann machine: A hardware accelerator for combinatorial optimization and deep learning. In Proceedings of the 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA’16).

[10]

Lars Buesing, Johannes Bill, Bernhard Nessler, and Wolfgang Maass. 2011. Neural dynamics as sampling: A model for stochastic computation in recurrent networks of spiking neurons. PLoS Comput. Biol. 7, 11 (2011), e1002211.

[11]

Kerem Yunus Camsari, Rafatul Faria, Brian M Sutton, and Supriyo Datta. 2017a. Stochastic p-bits for invertible logic. Phys. Rev. X 7, 3 (2017), 031014.

[12]

Kerem Yunus Camsari, Samiran Ganguly, and Supriyo Datta. 2015. Modular approach to spintronics. Sci. Rep. 5, Article 10571 (2015).

[13]

Kerem Yunus Camsari, Sayeef Salahuddin, and Supriyo Datta. 2017b. Implementing p-bits with embedded mtj. IEEE Electr. Dev. Lett. 38, 12 (2017), 1767--1770.

[14]

Miguel A. Carreira-Perpinan and Geoffrey E. Hinton. 2005. On contrastive divergence learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’05), Vol. 10. 33--40.

[15]

Won Ho Choi, Yang Lv, Jongyeon Kim, Abhishek Deshpande, Gyuseong Kang, Jian-Ping Wang, and Chris H. Kim. 2014. A magnetic tunnel junction based true random number generator with conditional perturb and real-time output probability tracking. In Proceedings of the IEEE International Electron Devices Meeting (IEDM’14). IEEE, 12--5.

[16]

Patricia S. Churchland and Terrence J. Sejnowski. 2016. The Computational Brain. MIT Press.

[17]

Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. BinaryConnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (Eds.). Curran Associates, Inc., 3123--3131.

Digital Library

[18]

R. P. Cowburn, D. K. Koltsov, A. O. Adeyeye, M. E. Welland, and D. M. Tricker. 1999. Single-domain circular nanomagnets. Phys. Rev. Lett. 83, 5 (Aug. 1999), 1042--1045.

[19]

Punyashloka Debashis, Rafatul Faria, Kerem Y Camsari, Joerg Appenzeller, Supriyo Datta, and Zhihong Chen. 2016. Experimental demonstration of nanomagnet networks as hardware for ising computing. In Proceedings of the IEEE International Electron Devices Meeting (IEDM’16). IEEE, 34--3.

[20]

P. Debashis, R. Faria, K. Y. Camsari, and Z. Chen. 2018. Design of stochastic nanomagnets for probabilistic spin logic. IEEE Magn. Lett. 9 (2018), 1--5.

[21]

S. B. Eryilmaz, E. Neftci, S. Joshi, S. Kim, M. BrightSky, H. L. Lung, C. Lam, G. Cauwenberghs, and H. S. P. Wong. 2016. Training a probabilistic graphical model with resistive switching electronic synapses. IEEE Trans. Electr. Dev. 63, 12 (Dec. 2016), 5004--5011.

[22]

Akio Fukushima, Takayuki Seki, Kay Yakushiji, Hitoshi Kubota, Hiroshi Imamura, Shinji Yuasa, and Koji Ando. 2014. Spin dice: A scalable truly random number generator based on spintronics. Appl. Phys. Express 7, 8 (2014), 083001.

[23]

Robert Hecht-Nielsen. 1992. Theory of the backpropagation neural network. In Neural Networks for Perception. Elsevier, 65--93.

Digital Library

[24]

Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. Neur. Comput. 18, 7 (2006), 1527--1554.

Digital Library

[25]

Geoffrey E. Hinton, Terrence J. Sejnowski, and David H. Ackley. 1984. Boltzmann Machines: Constraint Satisfaction Networks that Learn. Carnegie-Mellon University, Department of Computer Science Pittsburgh, PA.

[26]

Miao Hu, Hai Li, Qing Wu, and Garrett S. Rose. 2012. Hardware realization of BSB recall function using memristor crossbar arrays. In Proceedings of the 49th Annual Design Automation Conference (DAC’12). ACM, New York, NY, 498--503.

Digital Library

[27]

S. K. Kim, P. L. McMahon, and K. Olukotun. 2010. A large-scale architecture for restricted Boltzmann machines. In Proceedings of the 2010 18th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines. 201--208.

Digital Library

[28]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436.

[29]

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov. 1998), 2278--2324.

[30]

C. J. Lin, S. H. Kang, Y. J. Wang, K. Lee, X. Zhu, W. C. Chen, X. Li, W. N. Hsu, Y. C. Kao, M. T. Liu, et al. 2009. 45nm low power CMOS logic compatible embedded STT MRAM utilizing a reverse-connection 1T/1MTJ cell. In Proceedings of the IEEE International Electron Devices Meeting (IEDM’09). IEEE, 1--4.

[31]

Chamika M. Liyanagedera, Abhronil Sengupta, Akhilesh Jaiswal, and Kaushik Roy. 2017. Stochastic spiking neural networks enabled by magnetic tunnel junctions: From nontelegraphic to telegraphic switching regimes. Phys. Rev. Applied 8, 6 (2017), 064017.

[32]

Nicolas Locatelli, Alice Mizrahi, A. Accioly, Rie Matsumoto, Akio Fukushima, Hitoshi Kubota, Shinji Yuasa, Vincent Cros, Luis Gustavo Pereira, Damien Querlioz, et al. 2014. Noise-enhanced synchronization of stochastic magnetic oscillators. Phys. Rev. Appl. 2, 3 (2014), 034009.

[33]

D. L. Ly and P. Chow. 2010. High-performance reconfigurable hardware architecture for restricted Boltzmann machines. IEEE Trans. Neur. Netw. 21, 11 (Nov. 2010), 1780--1792.

Digital Library

[34]

Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. 2014. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 6197 (2014), 668--673.

[35]

Alice Mizrahi, Tifenn Hirtzlin, Akio Fukushima, Hitoshi Kubota, Shinji Yuasa, Julie Grollier, and Damien Querlioz. 2018. Neural-like computing with populations of superparamagnetic basis functions. Nat. Commun. 9, 1 (2018), 1533.

[36]

Stuart SP Parkin, Christian Kaiser, Alex Panchula, Philip M Rice, Brian Hughes, Mahesh Samant, and See-Hun Yang. 2004. Giant tunnelling magnetoresistance at room temperature with MgO (100) tunnel barriers. Nat. Mater. 3, 12 (2004), 862.

[37]

M. R. Pufall, W. H. Rippard, Shehzaad Kaka, S. E. Russek, T. J. Silva, Jordan Katine, and Matt Carey. 2004. Large-angle, gigahertz-rate random telegraph switching induced by spin-momentum transfer. Phys. Rev. B 69, 21 (Jun. 2004), 214409.

[38]

Kaushik Roy, Abhronil Sengupta, and Yong Shim. 2018. Perspective: Stochastic magnetic devices for cognitive computing. J. Appl. Phys. 123, 21 (2018), 210901.

[39]

Jack C. Sankey, Yong-Tao Cui, Jonathan Z. Sun, John C. Slonczewski, Robert A. Buhrman, and Daniel C. Ralph. 2008. Measurement of the spin-transfer-torque vector in magnetic tunnel junctions. Nat. Phys. 4, 1 (2008), 67.

[40]

R. Sarikaya, G. E. Hinton, and A. Deoras. 2014. Application of deep belief networks for natural language understanding. IEEE/ACM Trans. Aud. Speech Lang. Process. 22, 4 (Apr. 2014), 778--784.

Digital Library

[41]

J. F. Scott. 1998. High-dielectric constant thin films for dynamic random access memories (DRAM). Annu. Rev. Mater. Sci. 28, 1 (1998), 79--100.

[42]

Abhronil Sengupta, Aparajita Banerjee, and Kaushik Roy. 2016a. Hybrid spintronic-CMOS spiking neural network with on-chip learning: Devices, circuits, and systems. Phys. Rev. Appl. 6, 6 (Dec. 2016), 064003.

[43]

Abhronil Sengupta, Priyadarshini Panda, Parami Wijesinghe, Yusung Kim, and Kaushik Roy. 2016b. Magnetic tunnel junction mimics stochastic cortical spiking neurons. Sci. Rep. 6, Article 30039.

[44]

A. Sengupta, M. Parsa, B. Han, and K. Roy. 2016c. Probabilistic deep spiking neural systems enabled by magnetic tunnel junction. IEEE Trans. Electr. Dev. 63, 7 (Jul. 2016), 2963--2970.

[45]

Ahmad Muqeem Sheri, Aasim Rafique, Witold Pedrycz, and Moongu Jeon. 2015. Contrastive divergence for memristor-based restricted Boltzmann machine. Eng. Appl. Artif. Intell. 37 (2015), 336--342. http://www.sciencedirect.com/science/article/pii/S0952197614002334.

[46]

Massimiliano Stengel and Nicola A Spaldin. 2006. Origin of the dielectric dead layer in nanoscale capacitors. Nature 443, 7112 (2006), 679.

[47]

Dmitri B. Strukov, Gregory S. Snider, Duncan R. Stewart, and R. Stanley Williams. 2008. The missing memristor found. Nature 453, 7191 (2008), 80.

[48]

Brian Sutton, Kerem Yunus Camsari, Behtash Behin-Aein, and Supriyo Datta. 2017. Intrinsic optimization using stochastic nanomagnets. Sci. Rep. 7, 44370 (2017).

[49]

M. Tanaka and M. Okutomi. 2014. A novel inference of a restricted Boltzmann machine. In Proceedings of the 2014 22nd International Conference on Pattern Recognition. 1526--1531.

Digital Library

[50]

Mengxing Wang, Wenlong Cai, Kaihua Cao, Jiaqi Zhou, Jerzy Wrona, Shouzhong Peng, Huaiwen Yang, Jiaqi Wei, Wang Kang, Youguang Zhang, et al. 2018. Current-induced magnetization switching in atom-thick tungsten engineered perpendicular magnetic tunnel junctions with large tunnel magnetoresistance. Nat. Commun. 9, 1 (2018), 671.

[51]

Wenhong Wang, Hiroaki Sukegawa, Rong Shan, Seiji Mitani, and Koichiro Inomata. 2009. Giant tunneling magnetoresistance up to 330% at room temperature in sputter deposited Co 2 FeAl/MgO/CoFe magnetic tunnel junctions. Appl. Phys. Lett. 95, 18 (2009), 182502.

[52]

Bo Yuan and Keshab K. Parhi. 2017. VLSI architectures for the restricted Boltzmann machine. J. Emerg. Technol. Comput. Syst. 13, 3, Article 35 (May 2017), 19 pages.

Digital Library

[53]

Shinji Yuasa, Taro Nagahama, Akio Fukushima, Yoshishige Suzuki, and Koji Ando. 2004. Giant room-temperature magnetoresistance in single-crystal Fe/MgO/Fe magnetic tunnel junctions. Nat. Mater. 3, 12 (2004).

[54]

Ramtin Zand, Kerem Yunus Camsari, Steven D. Pyle, Ibrahim Ahmed, Chris H. Kim, and Ronald F. DeMara. 2018. Low-energy deep belief networks using intrinsic sigmoidal spintronic-based probabilistic neurons. In Proceedings of the 2018 on Great Lakes Symposium on VLSI (GLSVLSI’18). ACM, New York, NY, 15--20.

Digital Library

Cited By

Zeng MLi ZSaw JChen B(2024)Effect of stochastic activation function on reconstruction performance of restricted Boltzmann machines with stochastic magnetic tunnel junctionsApplied Physics Letters10.1063/5.0171238124:3Online publication date: 18-Jan-2024
https://doi.org/10.1063/5.0171238
He YLuo SFang CLiang G(2024)Direct design of ground-state probabilistic logic using many-body interactions for probabilistic computingScientific Reports10.1038/s41598-024-65676-z14:1Online publication date: 2-Jul-2024
https://doi.org/10.1038/s41598-024-65676-z
Si JYang SCen YChen JHuang YYao ZKim DCai KYoo JFong XYang H(2024)Energy-efficient superparamagnetic Ising machine and its application to traveling salesman problemsNature Communications10.1038/s41467-024-47818-z15:1Online publication date: 24-Apr-2024
https://doi.org/10.1038/s41467-024-47818-z
Show More Cited By

Index Terms

Composable Probabilistic Inference Networks Using MRAM-based Stochastic Neurons
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
2. Hardware
  1. Emerging technologies
    1. Spintronics and magnetic technologies
  2. Very large scale integration design

Recommendations

Low-Energy Deep Belief Networks Using Intrinsic Sigmoidal Spintronic-based Probabilistic Neurons
GLSVLSI '18: Proceedings of the 2018 Great Lakes Symposium on VLSI

A low-energy hardware implementation of deep belief network (DBN) architecture is developed using near-zero energy barrier probabilistic spin logic devices (p-bits), which are modeled to realize an intrinsic sigmoidal activation function. A CMOS/spin ...
Design of spin-torque transfer magnetoresistive RAM and CAM/TCAM with high sensing and search speed

With a great scalability potential, nonvolatile magnetoresistive memory with spin-torque transfer (STT) programming has become a topic of great current interest. This paper addresses cell structure design for STT magnetoresistive RAM, content ...
Building and Optimizing MRAM-Based Commodity Memories

Emerging non-volatile memory technologies such as MRAM are promising design solutions for energy-efficient memory architecture, especially for mobile systems. However, building commodity MRAM by reusing DRAM designs is not straightforward. The existing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Journal on Emerging Technologies in Computing Systems

ACM Journal on Emerging Technologies in Computing Systems Volume 15, Issue 2

Special Issue on HALO for Energy-Constrained On-Chip Machine Learning

April 2019

184 pages

ISSN:1550-4832

EISSN:1550-4840

DOI:10.1145/3322429

Editor:
Yuan Xie
University of California, Santa Barbara, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

ACM Journals for the Design of Smart and Connected Systems

Publication History

Published: 26 March 2019

Accepted: 01 January 2019

Revised: 01 October 2018

Received: 01 June 2018

Published in JETC Volume 15, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Nanoelectronic Computing Research (nCORE) Centers
Center for Probabilistic Spin Logic for Low-Energy Boolean and Non-Boolean Computing (CAPSL)
Semiconductor Research Corporation (SRC) program
NSF

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
1,260
Total Downloads

Downloads (Last 12 months)195
Downloads (Last 6 weeks)19

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zeng MLi ZSaw JChen B(2024)Effect of stochastic activation function on reconstruction performance of restricted Boltzmann machines with stochastic magnetic tunnel junctionsApplied Physics Letters10.1063/5.0171238124:3Online publication date: 18-Jan-2024
https://doi.org/10.1063/5.0171238
He YLuo SFang CLiang G(2024)Direct design of ground-state probabilistic logic using many-body interactions for probabilistic computingScientific Reports10.1038/s41598-024-65676-z14:1Online publication date: 2-Jul-2024
https://doi.org/10.1038/s41598-024-65676-z
Si JYang SCen YChen JHuang YYao ZKim DCai KYoo JFong XYang H(2024)Energy-efficient superparamagnetic Ising machine and its application to traveling salesman problemsNature Communications10.1038/s41467-024-47818-z15:1Online publication date: 24-Apr-2024
https://doi.org/10.1038/s41467-024-47818-z
Li XWan CZhang RZhao MXiong SKong DLuo XHe BLiu SXia JYu GHan X(2024)Restricted Boltzmann Machines Implemented by Spin–Orbit Torque Magnetic Tunnel JunctionsNano Letters10.1021/acs.nanolett.3c0482024:18(5420-5428)Online publication date: 26-Apr-2024
https://doi.org/10.1021/acs.nanolett.3c04820
Lindsay JZand R(2023)A Novel Stochastic LSTM Model Inspired by Quantum Machine Learning2023 24th International Symposium on Quality Electronic Design (ISQED)10.1109/ISQED57927.2023.10129344(1-8)Online publication date: 5-Apr-2023
https://doi.org/10.1109/ISQED57927.2023.10129344
Yarnell RHossain MDeMara R(2023)Image Quantization Tradeoffs in a YOLO-based FPGA Accelerator Framework2023 24th International Symposium on Quality Electronic Design (ISQED)10.1109/ISQED57927.2023.10129324(1-7)Online publication date: 5-Apr-2023
https://doi.org/10.1109/ISQED57927.2023.10129324
Hossain MTatulian AThummala HDeMara RSalehi S(2023)Energy-/Area-Efficient Spintronic ANN-based Digit Recognition via Progressive Modular Redundancy2023 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS46773.2023.10181529(1-5)Online publication date: 21-May-2023
https://doi.org/10.1109/ISCAS46773.2023.10181529
Cai WHuang YZhang XWang SPan YYin JShi KZhao W(2023)Spintronics intelligent devicesScience China Physics, Mechanics & Astronomy10.1007/s11433-022-2012-266:11Online publication date: 18-Apr-2023
https://doi.org/10.1007/s11433-022-2012-2
Chen BHou YGan CZeng M(2023)Micromagnetic realization of energy-based models using stochastic magnetic tunnel junctionsApplied Physics A10.1007/s00339-023-06931-4129:9Online publication date: 30-Aug-2023
https://doi.org/10.1007/s00339-023-06931-4
Cai BHe YXin YYuan ZZhang XZhu ZLiang G(2023)Unconventional computing based on magnetic tunnel junctionApplied Physics A10.1007/s00339-022-06365-4129:4Online publication date: 3-Mar-2023
https://doi.org/10.1007/s00339-022-06365-4
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents