research-article

Open access

High performance Monte Carlo simulation of ising model on TPU clusters

Authors:

Georgios Roumpos,

John AndersonAuthors Info & Claims

SC '19: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 83, Pages 1 - 15

https://doi.org/10.1145/3295500.3356149

Published: 17 November 2019 Publication History

Abstract

Large-scale deep learning benefits from an emerging class of AI accelerators. Some of these accelerators' designs are general enough for compute-intensive applications beyond AI and Cloud TPU is one such example. In this paper, we demonstrate a novel approach using TensorFlow on Cloud TPU to simulate the two-dimensional Ising Model. TensorFlow and Cloud TPU framework enable the simple and readable code to express the complicated distributed algorithm without compromising the performance. Our code implementation fits into a small Jupyter Notebook and fully utilizes Cloud TPU's efficient matrix operation and dedicated high speed inter-chip connection. The performance is highly competitive: it outperforms the best published benchmarks to our knowledge by 60% in single-core and 250% in multi-core with good linear scaling. When compared to Tesla V100 GPU, the single-core performance maintains a ~10% gain. We also demonstrate that using low precision arithmetic---bfloat16---does not compromise the correctness of the simulation results.

References

[1]

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pages 265--283, 2016.

Digital Library

[2]

Didier Barradas-Bautista, Matias Alvarado-Mentado, Mark Agostino, and Germinal Cocho. Cancer growth and metastasis as a metaphor of go gaming: An ising model approach. PloS one, 13(5):e0195654, 2018.

[3]

Kurt Binder. Finite size scaling analysis of ising model block distribution functions. Zeitschrift fÃijr Physik B Condensed Matter, 43(2):119--140, 1981.

[4]

Kurt Binder, Dieter Heermann, Lyle Roelofs, A. John Mallinckrodt, and Susan McKay. Monte carlo simulation in statistical physics. Computers in Physics, 7(2):156--157, 1993.

[5]

Kurt Binder and Erik Luijten. Monte carlo tests of renormalization-group predictions for critical phenomena in ising models. Physics Reports, 344(4--6):179--253, 2001.

[6]

Benjamin Block, Peter Virnau, and Tobias Preis. Multi-gpu accelerated multi-spin monte carlo simulations of the 2d ising model. Computer Physics Communications, 181(9):1549--1556, 2010.

[7]

Google Cloud. Choosing between a single cloud tpu device and a cloud tpu pod (alpha). https://cloud.google.com/tpu/docs/deciding-pod-versus-tpu, 2019.

[8]

Google Cloud. Performance guide. https://cloud.google.com/tpu/docs/performance-guide, 2019.

[9]

Google Cloud. System architecture. https://cloud.google.com/tpu/docs/system-architecture, 2019.

[10]

Google Cloud. Using bfloat16 with tensorflow models. https://cloud.google.com/tpu/docs/bfloat16, 2019.

[11]

Chase Cook, Hengyang Zhao, Takashi Sato, Masayuki Hiromoto, and Sheldon X.-D. Tan. Gpu based parallel ising computing for combinatorial optimization problems in vlsi physical design. arXiv preprint arXiv:1807.10750, 2018.

[12]

Nvidia NVLink Fabric. Nvlink fabric: Advancing multi-gpu processing. https://www.nvidia.com/en-us/data-center/nvlink/, 2019.

[13]

Ernst Ising. Beitrag zur theorie des ferromagnetismus. Zeitschrift fÃijr Physik, 31(1):253--258, 1925.

[14]

Norm Jouppi. Quantifying the performance of the tpu, our first machine learning chip. Google Cloud: https://cloud.google.com/blog/products/gcp/quantifying-the-performance-of-the-tpu-our-first-machine-learning-chip, 2017.

[15]

Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, and Al Borchers. In-datacenter performance analysis of a tensor processing unit. In Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on, pages 1--12. IEEE, 2017.

Digital Library

[16]

Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Landon, Cliff Young, and David Patterson. A Domain-Specific Supercomputer for Training Deep Neural Networks. Submitted to Communications of the ACM.

[17]

Peter Müller. A generic approach to posterior integration and Gibbs sampling. Purdue University, Department of Statistics, 1991.

[18]

Lars Onsager. Crystal statistics.i. a two-dimensional model with an order-disorder transition. Physical Review, 65(3--4):117, 1944.

[19]

Francisco Ortega-Zamorano, Marcelo A. Montemurro, Sergio Alejandro Cannas, José M. Jerez, and Leonardo Franco. Fpga hardware acceleration of monte carlo simulations for the ising model. IEEE Trans. Parallel Distrib. Syst., 27(9):2618--2627, 2016.

Digital Library

[20]

Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan. Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration. PyTorch: Tensors and dynamic neural networks in Python with strong GPU acceleration, 2017.

[21]

Tobias Preis, Wolfgang Paul, and Johannes J. Schneider. Fluctuation patterns in high-frequency financial asset returns. EPL (Europhysics Letters), 82(6):68005, 2008.

[22]

Tobias Preis, Peter Virnau, Wolfgang Paul, and Johannes J. Schneider. Gpu accelerated monte carlo simulation of the 2d and 3d ising model. Journal of Computational Physics, 228(12):4468--4477, 2009.

Digital Library

[23]

Francisco Prieto-Castrillo, Amin Shokri Gazafroudi, Javier Prieto, and Juan Manuel Corchado. An ising spin-based model to explore efficient flexibility in distributed power systems. Complexity 2018, 2018.

[24]

Alan M. Ferrenberg, Jiahao Xu, and David P. Landau Pushing the Limits of Monte Carlo Simulations for the 3d Ising Model. arXiv preprint arXiv:1806.03558, 2018.

Cited By

Wang HYan HRong CYuan YJiang FHan ZSui HJin DLi Y(2024)Multi-scale Simulation of Complex Systems: A Perspective of Integrating Knowledge and DataACM Computing Surveys10.1145/365466256:12(1-38)Online publication date: 3-Apr-2024
https://dl.acm.org/doi/10.1145/3654662
Singh NKobayashi KCao QSelcuk KHu TNiazi SAadit NKanai SOhno HFukami SCamsari K(2024)CMOS plus stochastic nanomagnets enabling heterogeneous computers for probabilistic inference and learningNature Communications10.1038/s41467-024-46645-615:1Online publication date: 27-Mar-2024
https://doi.org/10.1038/s41467-024-46645-6
Yu LNien C(2023)Physics-Inspired Optimization in the QUBO Framework: Key Concepts and ApproachesSPIN10.1142/S201032472340016713:04Online publication date: 30-Aug-2023
https://doi.org/10.1142/S2010324723400167
Show More Cited By

Index Terms

High performance Monte Carlo simulation of ising model on TPU clusters
1. Applied computing
  1. Physical sciences and engineering
    1. Mathematics and statistics
    2. Physics
2. Computing methodologies
  1. Modeling and simulation
    1. Simulation types and techniques
      1. Distributed simulation
      2. Massively parallel and high-performance simulations

Recommendations

GPU accelerated Monte Carlo simulation of the 2D and 3D Ising model

The compute unified device architecture (CUDA) is a programming approach for performing scientific calculations on a graphics processing unit (GPU) as a data-parallel computing device. The programming interface allows to implement algorithms using ...
Acceleration of Monte-Carlo simulation on high performance computing platforms
RACS '18: Proceedings of the 2018 Conference on Research in Adaptive and Convergent Systems

Monte Carlo methods are often used to solve computational problems with randomness. The random sampling helps avoid the deterministic results, but it requires intensive computations to obtain the results. Several attempts have been made to boost the ...
High-Performance Quasi-Monte Carlo Financial Simulation: FPGA vs. GPP vs. GPU

Quasi-Monte Carlo simulation is a special Monte Carlo simulation method that uses quasi-random or low-discrepancy numbers as random sample sets. In many applications, this method has proved advantageous compared to the traditional Monte Carlo simulation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '19: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2019

1921 pages

ISBN:9781450362290

DOI:10.1145/3295500

General Chair:
Michela Taufer,
Program Chairs:
Pavan Balaji,
Antonio J. Peña

Copyright © 2019 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC '19

Sponsor:

SIGHPC

SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis

November 17 - 19, 2019

Colorado, Denver

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
1,507
Total Downloads

Downloads (Last 12 months)225
Downloads (Last 6 weeks)32

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang HYan HRong CYuan YJiang FHan ZSui HJin DLi Y(2024)Multi-scale Simulation of Complex Systems: A Perspective of Integrating Knowledge and DataACM Computing Surveys10.1145/365466256:12(1-38)Online publication date: 3-Apr-2024
https://dl.acm.org/doi/10.1145/3654662
Singh NKobayashi KCao QSelcuk KHu TNiazi SAadit NKanai SOhno HFukami SCamsari K(2024)CMOS plus stochastic nanomagnets enabling heterogeneous computers for probabilistic inference and learningNature Communications10.1038/s41467-024-46645-615:1Online publication date: 27-Mar-2024
https://doi.org/10.1038/s41467-024-46645-6
Yu LNien C(2023)Physics-Inspired Optimization in the QUBO Framework: Key Concepts and ApproachesSPIN10.1142/S201032472340016713:04Online publication date: 30-Aug-2023
https://doi.org/10.1142/S2010324723400167
Tan HTong GHuang LXiao LXiao N(2023)Multiple-Mode-Supporting Floating-Point FMA Unit for Deep Learning ProcessorsIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2022.322618531:2(253-266)Online publication date: Feb-2023
https://doi.org/10.1109/TVLSI.2022.3226185
Chowdhury SGrimaldi AAadit NNiazi SMohseni MKanai SOhno HFukami STheogarajan LFinocchio GDatta SCamsari K(2023)A Full-Stack View of Probabilistic Computing With p-Bits: Devices, Architectures, and AlgorithmsIEEE Journal on Exploratory Solid-State Computational Devices and Circuits10.1109/JXCDC.2023.32569819:1(1-11)Online publication date: Jun-2023
https://doi.org/10.1109/JXCDC.2023.3256981
Singh NNiazi SChowdhury SSelcuk KKaneko HKobayashi KKanai SOhno HFukami SCamsari K(2023)Hardware Demonstration of Feedforward Stochastic Neural Networks with Fast MTJ-based p-bits2023 International Electron Devices Meeting (IEDM)10.1109/IEDM45741.2023.10413686(1-4)Online publication date: 9-Dec-2023
https://doi.org/10.1109/IEDM45741.2023.10413686
Chowdhury SCamsari KDatta S(2023)Accelerated quantum Monte Carlo with probabilistic computersCommunications Physics10.1038/s42005-023-01202-36:1Online publication date: 27-Apr-2023
https://doi.org/10.1038/s42005-023-01202-3
Liu HLiu YLi KZhao ZSchoenholz SCubuk EGupta PBauchy M(2023)End-to-end differentiability and tensor processing unit computing to accelerate materials’ inverse designnpj Computational Materials10.1038/s41524-023-01080-x9:1Online publication date: 13-Jul-2023
https://doi.org/10.1038/s41524-023-01080-x
Hu RPierce DShafi YBoral AAnisimov VNevo SChen Y(2022)Accelerating physics simulations with tensor processing unitsInternational Journal of High Performance Computing Applications10.1177/1094342022110287336:4(510-523)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1177/10943420221102873
De Maio VAral ABrandic IGeorgiou CSchiller EAli-Eldin AIosup A(2022)A Roadmap To Post-Moore Era for Distributed SystemsProceedings of the 2022 Workshop on Advanced tools, programming languages, and PLatforms for Implementing and Evaluating algorithms for Distributed systems10.1145/3524053.3542747(30-34)Online publication date: 25-Jul-2022
https://dl.acm.org/doi/10.1145/3524053.3542747
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents