research-article

Node variability in large-scale power measurements: perspectives from the Green500, Top500 and EEHPCWG

Authors:

Thomas Scogland,

Jonathan Azose,

Suzanne Rivoire,

Daniel HackenbergAuthors Info & Claims

SC '15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 74, Pages 1 - 11

https://doi.org/10.1145/2807591.2807653

Published: 15 November 2015 Publication History

Abstract

The last decade has seen power consumption move from an afterthought to the foremost design constraint of new supercomputers. Measuring the power of a supercomputer can be a daunting proposition, and as a result, many published measurements are extrapolated. This paper explores the validity of these extrapolations in the context of inter-node power variability and power variations over time within a run. We characterize power variability across nodes in systems at eight supercomputer centers across the globe. This characterization shows that the current requirement for measurements submitted to the Green500 and others is insufficient, allowing variations of up to 20% due to measurement timing and a further 10--15% due to insufficient sample sizes. This paper proposes new power and energy measurement requirements for supercomputers, some of which have been accepted for use by the Green500 and Top500, to ensure consistent accuracy.

References

[1]

B. Balaji, J. McCullough, R. K. Gupta, and Y. Agarwal. Accurate characterization of the variability in power consumption in modern mobile processors. In Workshop on Power-Aware Computing Systems, HotPower '12, 2012.

Digital Library

[2]

S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S.-H. Lee, and K. Skadron. Rodinia: A benchmark suite for heterogeneous computing. In IEEE International Symposium on Workload Characterization (IISWC), pages 44--54, 2009.

Digital Library

[3]

J. D. Davis, S. Rivoire, M. Goldszmidt, and E. K. Ardestani. Accounting for variability in large-scale cluster power models. In 2nd Exascale Evaluation and Research Techniques Workshop (EXERT), 2011.

[4]

T. Endo, A. Nukada, and S. Matsuoka. TSUBAME-KFC: Ultra green supercomputing testbed. Presented at International Conference for High Performance Computing, Networking, Storage and Analysis (SuperComputing, SC13), 2013.

[5]

Energy Efficient High Performance Computing Working Group (EE-HPC-WG). Energy efficient high performance computing power measurement methodology, version 1.2rc2. http://www.green500.org/sites/default/files/eehpcwg/EEHPCWG_PowerMeasurementMethodology.pdf.

[6]

X. Fan, W. Weber, and L. A. Barroso. Power provisioning for a warehouse-sized computer. In 34th International Symposium on Computer Architecture (ISCA), June 2007.

Digital Library

[7]

Green500. http://www.green500.org.

[8]

The Green Graph 500. http://green.graph500.org/.

[9]

D. Hackenberg, T. Ilsche, R. Schöne, D. Molka, M. Schmidt, and W. E. Nagel. Power measurement techniques on standard compute nodes: A quantitative comparison. In IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2013.

[10]

D. Hackenberg, R. Oldenburg, D. Molka, and R. Schöne. Introducing FIRESTARTER: A processor stress test utility. In International Green Computing Conference (IGCC), 2013.

[11]

D. Hackenberg, R. Schöne, D. Molka, M. S. Müller, and A. Knüpfer. Quantifying power consumption variations of HPC systems using SPEC MPI benchmarks. Computer Science - R&D, 25(3-4):155--163, 2010.

[12]

C.-H. Hsu and S. W. Poole. Power measurement for high performance computing: State of the art. In International Workshop on Power Measurement and Profiling (PMP), 2011.

Digital Library

[13]

G. Juckeland, W. Brantley, S. Chandrasekaran, B. Chapman, S. Che, M. Colgrove, H. Feng, A. Grund, R. Henschel, W.-M. Hwu, H. Li, M. S. Müller, M. Perminov, P. Shelepugin, K. Skadron, J. Stratton, A. Titov, K. Wang, M. van Waveren, B. Whitney, S. Wienke, R. Xu, and K. Kumaran. SPEC ACCEL -- a standard application suite for measuring hardware accelerator performance. In 5th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, 2014.

[14]

S. Kamil, J. Shalf, and E. Strohmaier. Power efficiency in high performance computing. In IEEE International Symposium on Parallel and Distributed Processing, pages 1--8, April 2008.

[15]

M. S. Müller, J. Baron, W. C. Brantley, H. Feng, D. Hackenberg, R. Henschel, G. Jost, D. Molka, C. Parrott, J. Robichaux, P. Shelepugin, M. van Waveren, B. Whitney, and K. Kumaran. SPEC OMP2012 --- an application benchmark suite for parallel systems using OpenMP. In B. M. Chapman, F. Massaioli, M. S. Müller, and M. Rorro, editors, OpenMP in a Heterogeneous World, volume 7312 of Lecture Notes in Computer Science, pages 223--236. Springer Berlin Heidelberg, 2012.

Digital Library

[16]

D. Rohr, M. Bach, G. Nešković, V. Lindenstruth, C. Pinke, and O. Philipsen. Lattice-CSC: Optimizing and building an efficient supercomputer for Lattice-QCD and to achieve first place in Green500. In Proceedings of the International Supercomputing Conference, 2015.

[17]

B. Rountree, D. H. Ahn, B. R. de Supinski, D. K. Lowenthal, and M. Schulz. Beyond DVFS: A first look at performance under a hardware-enforced power bound. In Workshop on High-Performance, Power-Aware Computing (HPPAC), 2012.

Digital Library

[18]

J. Russell and R. Cohn. Prime95. 2012.

[19]

T. R. Scogland, C. P. Steffen, T. Wilde, F. Parent, S. Coghlan, N. Bates, W. Feng, and E. Strohmaier. A power-measurement methodology for large-scale, high-performance computing. In 5th ACM/SPEC International Conference on Performance Engineering, ICPE '14, pages 149--159. ACM, 2014.

Digital Library

[20]

SPEC Power and Performance Committee. SPEC power and performance benchmark methodology. Technical report, Standard Performance Evaluation Corporation, 2010.

[21]

B. Subramaniam and W.-c. Feng. Understanding power measurement implications in the Green500 list. In IEEE/ACM International Conference on Green Computing and Communications (GreenCom), pages 245--251, Dec 2010.

Digital Library

[22]

Top 500 supercomputing sites. http://www.top500.org.

Cited By

Adhinarayanan VFeng W(2025)Looking Back to Look Forward: 15 Years of the Green500Computer10.1109/MC.2023.333331658:1(76-86)Online publication date: Jan-2025
https://doi.org/10.1109/MC.2023.3333316
Jain RTran BChen KSinclair MVenkataraman S(2024)PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU ClustersProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00032(1-18)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00032
Solórzano ASato KYamamoto KShoji FBrandt JSchwaller BWalton SGreen JTiwari D(2024)Toward Sustainable HPC: In-Production Deployment of Incentive-Based Power Efficiency Mechanism on the Fugaku SupercomputerSC24: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41406.2024.00030(1-16)Online publication date: 17-Nov-2024
https://doi.org/10.1109/SC41406.2024.00030
Show More Cited By

Index Terms

Node variability in large-scale power measurements: perspectives from the Green500, Top500 and EEHPCWG

Recommendations

A power-measurement methodology for large-scale, high-performance computing
ICPE '14: Proceedings of the 5th ACM/SPEC international conference on Performance engineering

Improvement in the energy efficiency of supercomputers can be accelerated by improving the quality and comparability of efficiency measurements. The ability to generate accurate measurements at extreme scale are just now emerging. The realization of ...
Computing on large-scale distributed systems: Xtrem Web architecture, programming models, security, tests and convergence with grid
Special issue: P2P computing and interaction with grids

Global Computing systems belong to the class of large-scale distributed systems. Their properties high computational, storage and communication performance potentials, high resilience make them attractive in academia and industry as computing ...
A large scale grid testbed: the Seoul grid testbed
ICCOMP'06: Proceedings of the 10th WSEAS international conference on Computers

Seoul GRID Center has been building a large scale GRID Testbed, namely Seoul GRID Testbed, which covers whole metropolitan area of Seoul, the Capital City of Korea. In this paper, we introduce the Seoul Grid Testbed with the Seoul Grid portal. The ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2015

985 pages

ISBN:9781450337236

DOI:10.1145/2807591

General Chair:
Jackie Kern
University of Illinois at Urbana-Champaign, Urbana, Illinois
,
Program Chair:
Jeffrey S. Vetter
Oak Ridge National Laboratory and Georgia Institute of Technology, Oak Ridge, Tennessee

Copyright © 2015 ACM.

© 2015 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SC15

Sponsor:

SIGHPC
SIGARCH
IEEE-CS

SC15: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 15 - 20, 2015

Texas, Austin

Acceptance Rates

SC '15 Paper Acceptance Rate 79 of 358 submissions, 22%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
262
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)3

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Adhinarayanan VFeng W(2025)Looking Back to Look Forward: 15 Years of the Green500Computer10.1109/MC.2023.333331658:1(76-86)Online publication date: Jan-2025
https://doi.org/10.1109/MC.2023.3333316
Jain RTran BChen KSinclair MVenkataraman S(2024)PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU ClustersProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00032(1-18)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00032
Solórzano ASato KYamamoto KShoji FBrandt JSchwaller BWalton SGreen JTiwari D(2024)Toward Sustainable HPC: In-Production Deployment of Incentive-Based Power Efficiency Mechanism on the Fugaku SupercomputerSC24: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41406.2024.00030(1-16)Online publication date: 17-Nov-2024
https://doi.org/10.1109/SC41406.2024.00030
Sinha PGuliani AJain RTran BSinclair MVenkataraman SWolf FShende SCulhane CAlam SJagode H(2022)Not all GPUs are created equalProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/3571885.3571971(1-15)Online publication date: 13-Nov-2022
https://dl.acm.org/doi/10.5555/3571885.3571971
Abdelhafez HHalawa HAlmoallim AAhmadi APattabiraman KRipeanu M(2022)Characterizing Variability in Heterogeneous Edge Systems: A Methodology & Case Study2022 IEEE/ACM 7th Symposium on Edge Computing (SEC)10.1109/SEC54971.2022.00016(107-121)Online publication date: Dec-2022
https://doi.org/10.1109/SEC54971.2022.00016
Sinha PGuliani AJain RTran BSinclair MVenkataraman S(2022)Not All GPUs Are Created Equal: Characterizing Variability in Large-Scale, Accelerator-Rich SystemsSC22: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41404.2022.00070(01-15)Online publication date: Nov-2022
https://doi.org/10.1109/SC41404.2022.00070
Hou ZShen HZhou XGu JWang YZhao T(2022)Prediction of job characteristics for intelligent resource allocation in HPC systems: a survey and future directionsFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-022-0625-816:5Online publication date: 1-Oct-2022
https://dl.acm.org/doi/10.1007/s11704-022-0625-8
Patel TLiu ZKettimuthu RRich PAllcock WTiwari DCuicchi CQualters IKramer W(2020)Job characteristics on large-scale systemsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.5555/3433701.3433812(1-17)Online publication date: 9-Nov-2020
https://dl.acm.org/doi/10.5555/3433701.3433812
Patel TLiu ZKettimuthu RRich PAllcock WTiwari D(2020)Job Characteristics on Large-Scale Systems: Long-Term Analysis, Quantification, and ImplicationsSC20: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41405.2020.00088(1-17)Online publication date: Nov-2020
https://doi.org/10.1109/SC41405.2020.00088
Patel TWagenhauser AEibel CHonig TZeiser TTiwari D(2020)What does Power Consumption Behavior of HPC Jobs Reveal? : Demystifying, Quantifying, and Predicting Power Consumption Characteristics2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS47924.2020.00087(799-809)Online publication date: May-2020
https://doi.org/10.1109/IPDPS47924.2020.00087
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten