research-article

Parallel random numbers: as easy as 1, 2, 3

Authors:

John K. Salmon,

Mark A. Moraes,

David E. ShawAuthors Info & Claims

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 16, Pages 1 - 12

https://doi.org/10.1145/2063384.2063405

Published: 12 November 2011 Publication History

Abstract

Most pseudorandom number generators (PRNGs) scale poorly to massively parallel high-performance computation because they are designed as sequentially dependent state transformations. We demonstrate that independent, keyed transformations of counters produce a large alternative class of PRNGs with excellent statistical properties (long period, no discernable structure or correlation). These counter-based PRNGs are ideally suited to modern multi-core CPUs, GPUs, clusters, and special-purpose hardware because they vectorize and parallelize well, and require little or no memory for state. We introduce several counter-based PRNGs: some based on cryptographic standards (AES, Threefish) and some completely new (Philox). All our PRNGs pass rigorous statistical tests (including TestU01's BigCrush) and produce at least 2⁶⁴ unique parallel streams of random numbers, each with period 2¹²⁸ or more. In addition to essentially unlimited parallel scalability, our PRNGs offer excellent single-chip performance: Philox is faster than the CURAND library on a single NVIDIA GPU.

References

[1]

M. Bellare and P. Rogaway. Pseudorandom functions. In Introduction to Modern Cryptography. UCSD CSE 207 Online Course Notes, 2011. Chap 3. http://cseweb.ucsd.edu/~mihir/cse207/w-prf.pdf.

[2]

D. J. Bernstein and P. Schwabe. New AES software speed records. In D. R. Chowdhury and V. Rijmen, editors, Progress in Cryptology - INDOCRYPT 2008, volume 5365 of Lecture Notes in Computer Science, pages 322--336, Berlin, 2008. Springer-Verlag.

Digital Library

[3]

R. P. Brent. Some long-period random number generators using shifts and xors. ANZIAM Journal, 48:C188--C202, 2007.

[4]

R. G. Brown. Dieharder: A random number test suite. http://phy.duke.edu/~rgb/General/dieharder.php.

[5]

D. S. Cerutti, R. Duke, P. L. Freddolino, H. Fan, and T. P. Lybrand. Vulnerability in popular molecular dynamics packages concerning Langevin and Andersen dynamcs. J. Chem. Theory, 4:1669--1680, 2008.

[6]

P. Coddington. Random number generators for parallel computers. Technical Report 13, Northeast Parallel Architecture Center, 1997.

[7]

A. De Matteis and S. Pagnutti. Parallelization of random number generators and long-range correlations. Numer. Math., 53:595--608, August 1988.

Digital Library

[8]

A. De Matteis and S. Pagnutti. Long-range correlations in linear and non-linear random number generators. Parallel Computing, 14:207--210, 1990.

[9]

M. Dworkin. Recommendation for block cipher modes of operation, methods and techniques. NIST Special Publication 800-38A. National Institute of Standards and Technology (NIST), 2001.

Digital Library

[10]

H. Feistel. Cryptography and computer privacy. Scientific American, 228(5):15--23, 1973.

[11]

N. Ferguson, S. Lucks, B. Schneier, B. Whiting, M. Bellare, T. Kohno, J. Callas, and J. Walker. The Skein hash function family. http://www.schneier.com/skein.pdf, 2010.

[12]

A. M. Ferrenberg, D. P. Landau, and Y. J. Wong. Monte Carlo simulations: Hidden errors from "good" random number generators. Phy. Rev. Lett., 69:3382--3384, 1992.

[13]

G. C. Fox, M. A. Johnson, G. A. Lyzenga, S. W. Otto, J. K. Salmon, and D. W. Walker. Solving Problems on Concurrent Processors; Volume 1: General Techniques and Regular Problems. Prentice-Hall, 1988.

Digital Library

[14]

S. Gueron. Intel Advanced Encryption Standard (AES) instructions set. Technical report, Intel, 2010.

[15]

P. Hellekalek. Don't trust parallel Monte Carlo! In Proc. 12 ^th Workshop on Parallel and Distributed Simulation, PADS '98, pages 82--89, Washington, D. C., 1998. IEEE Computer Society.

Digital Library

[16]

P. Hellekalek. Good random number generators are (not so) easy to find. Math. Comput. Simul., 46:485--505, June 1998.

Digital Library

[17]

P. Hellekalek and S. Wegenkittl. Empirical evidence concerning AES. ACM Trans. Model. Comput. Simul., 13:322--333, October 2003.

Digital Library

[18]

Intel. Vector Statistical Library (VSL) performance data. http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vsl_performance_data.htm.

[19]

M. H. Kalos and P. A. Whitlock. Monte Carlo Methods. Wiley-VCH, 2nd edition, 2008.

[20]

D. E. Knuth. The Art of Computer Programming, Volume 2 (3rd ed.): Seminumerical Algorithms. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997.

Digital Library

[21]

P. L'Ecuyer. Combined multiple recursive random number generators. Operations Research, 44(5):816--822, 1996.

Digital Library

[22]

P. L'Ecuyer. Random number generation. In J. E. Gentle, W. Haerdle, and Y. Mori, editors, Handbook of Computational Statistics, pages 35--70. Springer-Verlag, Berlin, 2004. Chapter II.2.

[23]

P. L'Ecuyer, F. Blouin, and R. Couture. A search for good multiple recursive random number generators. ACM Trans. Model. Comput. Simul., 3(2):87--98, 1993.

Digital Library

[24]

P. L'Ecuyer and R. Simard. TestU01: A C library for empirical testing of random number generators. ACM Trans. Math. Softw., 33, August 2007.

Digital Library

[25]

D. H. Lehmer. Mathematical methods in large-scale computing units. In Proc. 2nd Symp. on Large-Scale Digital Calculating Machinery, pages 141--146. Harvard University Press, 1949.

[26]

G. Marsaglia. DIEHARD: A battery of tests of randomness. http://stat.fsu.edu/~geo/diehard.html.

[27]

G. Marsaglia. Xorshift RNGs. J. Stat. Soft., 8:1--6, 2003.

[28]

M. Mascagni and A. Srinivasan. Algorithm 806: SPRNG: A scalable library for pseudorandom number generation. ACM Transactions on Mathematical Software, 26:436--461, 2000.

Digital Library

[29]

M. Matsumoto and T. Nishimura. Mersenne twister: A 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans. Model. Comput. Simul., 8:3--30, January 1998.

Digital Library

[30]

M. Matsumoto, I. Wada, A. Kuramoto, and H. Ashihara. Common defects in initialization of pseudorandom number generators. ACM Trans. Model. Comput. Simul., 17, September 2007.

Digital Library

[31]

E. H. Mckinney. Generalized birthday problem. The American Mathematical Monthly, 73(4):385--387, April 1966.

[32]

National Bureau of Standards. Data Encryption Standard. FIPS PUB 46--3, 1977.

[33]

National Institute of Standards and Technology. Cryptographic hash algorithm competition website. http://csrc.nist.gov/groups/ST/hash/sha-3/index.html.

[34]

National Institute of Standards and Technology. Advanced Encryption Standard (AES). FIPS PUB 197, 2001.

[35]

F. Panneton, P. L'Ecuyer, and M. Matsumoto. Improved long-period generators based on linear recurrences modulo 2. ACM Trans. Math. Softw., 32:1--16, March 2006.

Digital Library

[36]

S. K. Park and K. W. Miller. Random number generators: good ones are hard to find. Commun. ACM, 31:1192--1201, October 1988.

Digital Library

[37]

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes in C. Cambridge University Press, Cambridge, 2^nd edition, 1992.

[38]

A. Rukhin, J. Soto, J. Nechvatal, M. Smid, E. Barker, M. Levinson, M. Vangel, D. Banks, A. Heckert, J. Dray, and S. Vo. A statistical test suite for random and pseudorandom number generators for cryptographic applications. Special Publication 800-22 Revision 1a, NIST, April 2010.

[39]

C. E. Shannon. Communication theory of secrecy systems. Bell System Technical Journal, 28:656--715, 1949.

[40]

D. E. Shaw, R. O. Dror, J. K. Salmon, J. P. Grossman, K. M. Mackenzie, J. A. Bank, C. Young, M. M. Deneroff, B. Batson, K. J. Bowers, E. Chow, M. P. Eastwood, D. J. Ierardi, J. L. Klepeis, J. S. Kuskin, R. H. Larson, K. Lindorff-Larsen, P. Maragakis, M. A. Moraes, S. Piana, Y. Shan, and B. Towles. Millisecond-scale molecular dynamics simulations on Anton. In Proc. Conf. on High Performance Computing, Networking, Storage and Analysis, SC09, pages 39:1--39:11, New York, NY, 2009. ACM.

Digital Library

[41]

D. E. Shaw, P. Maragakis, K. Lindorff-Larsen, S. Piana, R. O. Dror, M. P. Eastwood, J. A. Bank, J. M. Jumper, J. K. Salmon, Y. Shan, and W. Wriggers. Atomic-level characterization of the structural dynamics of proteins. Science, 330:341--346, 2010.

[42]

D. J. Sindhikara, S. Kim, A. F. Voter, and A. E. Roitberg. Bad seeds sprout perilous dynamics: Stochastic thermostat induced trajectory synchronization in biomolecules. J. Chem. Theory and Comp., 5(6):1624--1631, 2009.

[43]

J. L. Smith. The design of Lucifer, a cryptographic device for data communications. IBM Research Report RC3326, IBM T. J. Watson Research Center, Yorktown Heights, NY 10598, USA, Apr. 1971.

[44]

A. Sorkin. Lucifer, a crytographic algorithm. Cryptologia, 8:22--35, 1984.

[45]

S. Tzeng and L.-Y. Wei. Parallel white noise generation on a GPU via cryptographic hash. In Proc. 2008 Symp. on Interactive 3D graphics and games, I3D '08, pages 79--87, New York, NY, 2008. ACM.

Digital Library

[46]

S. Ulam, R. Richtmeyer, and J. von Neumann. Statistical methods in neutron diffusion. Technical Report LAMS-551, Los Alamos Scientific Laboratory, April 1947.

[47]

J. von Neuman. Various techniques used in connection with random digits. In A. Householder, G. Forsythe, and H. Germond, editors, Monte Carlo Method, Applied Math Series, Volume 11, pages 36--38. National Bureau of Standards, 1951.

[48]

F. Zafar, M. Olano, and A. Curtis. GPU random numbers via the Tiny Encryption Algorithm. In Proc. Conf. High Performance Graphics, HPG '10, pages 133--141, Aire-la-Ville, Switzerland, 2010. Eurographics Association.

Digital Library

Cited By

Romero-Barrera ACruz-Piris LTejedor-Romero MGimenez-Guzman JMarsa-Maestre I(2025)An Environmentally Adaptive CRO-SL Algorithm Based on Dynamic Agents for the Channel Assignment Problem in Wireless NetworksIEEE Access10.1109/ACCESS.2024.352346413(541-561)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3523464
Doignies BCoeurjolly DBonneel NDigne JIehl JOstromoukhov V(2024)Differentiable Owen ScramblingACM Transactions on Graphics10.1145/368776443:6(1-12)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687764
Glatt-Holtz NHolbrook AKrometis JMondaini C(2024)Parallel MCMC algorithms: theoretical foundations, algorithm design, case studiesTransactions of Mathematics and Its Applications10.1093/imatrm/tnae0048:2Online publication date: 22-Aug-2024
https://doi.org/10.1093/imatrm/tnae004
Show More Cited By

Index Terms

Parallel random numbers: as easy as 1, 2, 3

Recommendations

Recycling random bits in parallel
HICSS '95: Proceedings of the 28th Hawaii International Conference on System Sciences

Shows that r pseudo-random bits can be obtained by concatenating t blocks of r/t pseudo-random bits, where the blocks are generated in parallel. This can be considered as a parallel version of R. Impagliazzo and D. Zuckerman's (1989) method of recycling ...
Testing parallel random number generators

Monte Carlo computations are considered easy to parallelize. However, the results can be adversely affected by defects in the parallel pseudorandom number generator used. A parallel pseudorandom number generator must be tested for two types of ...
Random Numbers for Quantum Key Distribution
SYNASC '08: Proceedings of the 2008 10th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing

Quantum Key Distribution (QKD) has seen an important development for the last ten years. Nowadays, different companies offer commercial solutions and key exchanges at pulse rates of up to 1 GHz and over distances of up to 00km have been demonstrated in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

November 2011

866 pages

ISBN:9781450307710

DOI:10.1145/2063384

Conference Chair:
Scott Lathrop
University of Chicago
,
Program Chairs:
Jim Costa
Sandia National Laboratories
,
William Kramer
National Center for Supercomputing Applications

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SC '11

Sponsor:

SIGARCH
IEEE-CS

SC '11: International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 18, 2011

Washington, Seattle

Acceptance Rates

SC '11 Paper Acceptance Rate 74 of 352 submissions, 21%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

148
Total Citations
View Citations
3,035
Total Downloads

Downloads (Last 12 months)195
Downloads (Last 6 weeks)29

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Romero-Barrera ACruz-Piris LTejedor-Romero MGimenez-Guzman JMarsa-Maestre I(2025)An Environmentally Adaptive CRO-SL Algorithm Based on Dynamic Agents for the Channel Assignment Problem in Wireless NetworksIEEE Access10.1109/ACCESS.2024.352346413(541-561)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3523464
Doignies BCoeurjolly DBonneel NDigne JIehl JOstromoukhov V(2024)Differentiable Owen ScramblingACM Transactions on Graphics10.1145/368776443:6(1-12)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687764
Glatt-Holtz NHolbrook AKrometis JMondaini C(2024)Parallel MCMC algorithms: theoretical foundations, algorithm design, case studiesTransactions of Mathematics and Its Applications10.1093/imatrm/tnae0048:2Online publication date: 22-Aug-2024
https://doi.org/10.1093/imatrm/tnae004
Wang TAtif MDong ZLeggett CLin M(2024)A new portable random number generator wrapper libraryEPJ Web of Conferences10.1051/epjconf/202429511001295(11001)Online publication date: 6-May-2024
https://doi.org/10.1051/epjconf/202429511001
Khan SPalmer BEdelmaier CAktulga H(2024)OpenRAND: A performance portable, reproducible random number generation library for parallel computationsSoftwareX10.1016/j.softx.2024.10177327(101773)Online publication date: Sep-2024
https://doi.org/10.1016/j.softx.2024.101773
Park JJu JLee WKang BKachi YSakurai K(2024)A statistical verification method of random permutations for hiding countermeasure against side-channel attacksJournal of Information Security and Applications10.1016/j.jisa.2024.10379784(103797)Online publication date: Aug-2024
https://doi.org/10.1016/j.jisa.2024.103797
Bernaschi MGonzalez-Adalid Pemartin IMartin-Mayor VParisi G(2024)The QISG suite: high-performance codes for studying Quantum Ising Spin GlassesComputer Physics Communications10.1016/j.cpc.2024.109101(109101)Online publication date: Jan-2024
https://doi.org/10.1016/j.cpc.2024.109101
Wu CLiu J(2024)Smart contract assisted secure aggregation scheme for model update in federated learningComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2024.110542250:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.comnet.2024.110542
Boschini MCavallotto GDella Torre SGervasi MLa Vacca GRancoita PTacconi M(2024)Fast and accurate evaluation of deep-space galactic cosmic ray fluxes with HelMod-4/CUDAAdvances in Space Research10.1016/j.asr.2024.04.02174:9(4302-4320)Online publication date: Nov-2024
https://doi.org/10.1016/j.asr.2024.04.021
Onyekwelu ISharda V(2024)A Bayesian Regression Approach for Estimating Photosynthetically Active Radiation Using Satellite Data: Implications for Soybean Yield Prediction using the CROPGRO ModelEarth Systems and Environment10.1007/s41748-024-00391-38:4(1059-1076)Online publication date: 25-Apr-2024
https://doi.org/10.1007/s41748-024-00391-3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents