research-article

Public Access

The Value of Variance

Authors:

Jesun Sahariar Firoz,

Martina Barnas,

Marcin Zalewski,

Andrew LumsdaineAuthors Info & Claims

ICPE '16: Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering

Pages 287 - 295

https://doi.org/10.1145/2851553.2851573

Published: 12 March 2016 Publication History

Abstract

Measurements for distributed algorithms, such as performance results, are usually reported using averages, similarly to prevailing practice in other areas of computer science. We argue that including standard deviations offers additional information and that the minimal burden of providing standard deviations is outweighed by the benefits. We propose a new way of reporting run time speedup that incorporates standard deviation and demonstrate its usefulness in terms of two distributed graph algorithms.

References

[1]

Big Red II at Indiana University. https://kb.iu.edu/d/bcqt#overview.

[2]

HPX website. http://hpx.crest.iu.edu/. Accessed: 2015-05--25.

[3]

The NIST reference on constants, units, and uncertainty. http://physics.nist.gov/cuu/Uncertainty/basic.html, Sept. 2015.

[4]

Combining uncertainty components. http://physics.nist.gov/cgi-bin/cuu/Info/Uncertainty/combination.html, Sept. 2015.

[5]

The Graph500 List. http://www.graph500.org/, June 2015.

[6]

ParalleX Execution Model. https://www.crest.iu.edu/projects/xpress/_media/public/parallex_v3--1_03182013.doc, June 2015.

[7]

A. B. de Oliveira, S. Fischmeister, A. Diwan, M. Hauswirth, and P. F. Sweeney. Why you should care about quantile regression. SIGARCH Comput. Archit. News, 41 (1): 207--218, Mar. 2013. ISSN 0163--5964.

Digital Library

[8]

A. B. de Oliveira, J.-C. Petkovich, T. Reidemeister, and S. Fischmeister. Datamill: Rigorous performance evaluation made easy. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering, ICPE '13, pages 137--148, New York, NY, USA, 2013. ACM. ISBN 978--1--4503--1636--1.

Digital Library

[9]

J. S. Firoz, T. A. Kanewala, M. Zalewski, M. Barnas, and A. Lumsdaine. The anatomy of large-scale distributed graph algorithms. CoRR, abs/1507.06702, 2015. URL http://arxiv.org/abs/1507.06702.

[10]

J. S. Firoz, M. Zalewski, T. A. Kanewala, M. Barnas, and A. Lumsdaine. Importance of runtime considerations in performance engineering of large-scale distributed graph algorithms. In Euro-Par 2015: Parallel Processing Workshops, pages 553--564. Springer International Publishing, 2015.

[11]

D. Guerrera, H. Burkhart, and A. Maffia. Reproducible experiments in parallel computing: Concepts and stencil compiler benchmark study. In Euro-Par 2014: Parallel Processing Workshops - Euro-Par 2014 International Workshops, Porto, Portugal, August 25--26, 2014, Revised Selected Papers, Part I, pages 464--474, 2014.

Digital Library

[12]

A. S. Harji, P. A. Buhr, and T. Brecht. Our troubles with linux and why you should care. In Proceedings of the Second Asia-Pacific Workshop on Systems, APSys '11, pages 2:1--2:5, New York, NY, USA, 2011. ACM. ISBN 978--1--4503--1179--3.

Digital Library

[13]

Harshvardhan, A. Fidel, N. M. Amato, and L. Rauchwerger. KLA: A New Algorithmic Paradigm for Parallel Graph Computations. In Proceedings of the 23rd International Conference on Parallel Architectures and Compilation, pages 27--38. ACM, 2014.

Digital Library

[14]

T. Hoefler and R. Belli. Scientific Benchmarking of Parallel Computing Systems. Nov. 2015. accepted at IEEE/ACM International Conference on High Performance Computing, Networking, Storage and Analysis (SC15).

Digital Library

[15]

S. Hunold and J. L. Traff. On the state and importance of reproducible experimental research in parallel computing. CoRR, abs/1308.3648, 2013. URL http://arxiv.org/abs/1308.3648.

[16]

T. A. Kanewala, M. Zalewski, M. Barnas, J. S. Firoz, and A. Lumsdaine. Abstract graph algorithms with spatial-temporal execution. In preparation.

[17]

J. Levin, J. A. Fox, and D. R. Forde. Elementary statistics in social research. Allyn & Bacon, 2010.

[18]

A. Lumsdaine, D. Gregor, B. Hendrickson, and J. Berry. Challenges in parallel graph processing. Parallel Processing Letters, 17 (1): 5--20, 2007 2007.

[19]

U. Meyer and P. Sanders.Dstepping: A Parallelizable Shortest Path Algorithm. Journal of Algorithms, 49 (1): 114--152, 2003.

Digital Library

[20]

T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney. Producing wrong data without doing anything obviously wrong! SIGPLAN Not., 44 (3): 265--276, Mar. 2009. ISSN 0362--1340.

Digital Library

[21]

L. Peterson and V. S. Pai. Experience-driven experimental systems research. Commun. ACM, 50 (11): 38--44, Nov. 2007. ISSN 0001-0782.

Digital Library

[22]

W. F. Tichy. Should computer scientists experiment more? Computer, 31 (5): 32--40, May 1998. ISSN 0018--9162.

Digital Library

[23]

J. Vitek and T. Kalibera. Repeatability, reproducibility, and rigor in systems research. In Proceedings of the Ninth ACM International Conference on Embedded Software, EMSOFT '11, pages 33--38, New York, NY, USA, 2011. ACM. ISBN 978--1--4503-0714--7.

Digital Library

[24]

J. J. Willcock, T. Hoefler, N. G. Edmonds, and A. Lumsdaine.ampp: A Generalized Active Message Framework. In Proce. 19th Int. Conf. on Parallel Architectures and Compilation Techniques, pages 401--410. ACM, 2010.

Digital Library

[25]

J. J. Willcock, T. Hoefler, N. G. Edmonds, and A. Lumsdaine. Active pebbles: a programming model for highly parallel fine-grained data-driven computations. In Proc. 16th ACM symposium on Principles and practice of parallel programming, pages 305--306. ACM, 2011.

Digital Library

Cited By

Papadopoulos AVersluis LBauer AHerbst NKistowski JAli-Eldin AAbad CAmaral JTuma PIosup A(2021)Methodological Principles for Reproducible Performance Evaluation in Cloud ComputingIEEE Transactions on Software Engineering10.1109/TSE.2019.292790847:8(1528-1543)Online publication date: 1-Aug-2021
https://doi.org/10.1109/TSE.2019.2927908
Papadopoulos AAsadollah SAshjaei MMubeen SPei-Breivold HBehnam M(2017)SLAs for Industrial IoT: Mind the Gap2017 5th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW)10.1109/FiCloudW.2017.70(75-78)Online publication date: Aug-2017
https://doi.org/10.1109/FiCloudW.2017.70
Firoz JZalewski MBarnas MLumsdaine A(2017)Improving Performance of Distributed Graph Traversals via Application-Aware Plug-In Work SchedulerEuro-Par 2016: Parallel Processing Workshops10.1007/978-3-319-58943-5_44(545-556)Online publication date: 28-May-2017
https://doi.org/10.1007/978-3-319-58943-5_44
Show More Cited By

Index Terms

The Value of Variance
1. General and reference
  1. Cross-computing tools and techniques
2. Theory of computation
  1. Design and analysis of algorithms
    1. Distributed algorithms
    2. Parallel algorithms

Recommendations

A parallel vector quantization algorithm for SIMD multiprocessor systems
DCC '95: Proceedings of the Conference on Data Compression

Summary form only given , as follows. This article proposes a parallel vector quantization (VQ) algorithm for an exhaustive search of codebooks on a single-instruction-multiple-data (SIMD) multiprocessor. The proposed parallel VQ algorithm can be ...
Speedup Versus Efficiency in Parallel Systems

The tradeoff between speedup and efficiency that is inherent to a software system is investigated. The extent to which this tradeoff is determined by the average parallelism of the software system, as contrasted with other, more detailed, ...
Interpreting Parallel Processor Performance Measurements

This paper discusses execution time versus number of simultaneous operations in parallel computing systems. The main focus is on shared memory multiprocessors. A model for execution time as a function of the number of processes used in a computation is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '16: Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering

March 2016

346 pages

ISBN:9781450340809

DOI:10.1145/2851553

General Chairs:
Alberto Avritzer
Sonatype, Inc., USA
,
Alexandru Iosup
Delft University of Technology, the Netherlands
,
Program Chairs:
Xiaoyun Zhu
Futurewei Technologies, USA
,
Steffen Becker
Chemnitz University of Technology, Germany

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 March 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Lilly Endowment Inc.
NSF

Conference

ICPE'16

Sponsor:

ICPE'16: ACM/SPEC International Conference on Performance Engineering

March 12 - 16, 2016

Delft, The Netherlands

Acceptance Rates

ICPE '16 Paper Acceptance Rate 23 of 74 submissions, 31%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
188
Total Downloads

Downloads (Last 12 months)62
Downloads (Last 6 weeks)7

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Papadopoulos AVersluis LBauer AHerbst NKistowski JAli-Eldin AAbad CAmaral JTuma PIosup A(2021)Methodological Principles for Reproducible Performance Evaluation in Cloud ComputingIEEE Transactions on Software Engineering10.1109/TSE.2019.292790847:8(1528-1543)Online publication date: 1-Aug-2021
https://doi.org/10.1109/TSE.2019.2927908
Papadopoulos AAsadollah SAshjaei MMubeen SPei-Breivold HBehnam M(2017)SLAs for Industrial IoT: Mind the Gap2017 5th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW)10.1109/FiCloudW.2017.70(75-78)Online publication date: Aug-2017
https://doi.org/10.1109/FiCloudW.2017.70
Firoz JZalewski MBarnas MLumsdaine A(2017)Improving Performance of Distributed Graph Traversals via Application-Aware Plug-In Work SchedulerEuro-Par 2016: Parallel Processing Workshops10.1007/978-3-319-58943-5_44(545-556)Online publication date: 28-May-2017
https://doi.org/10.1007/978-3-319-58943-5_44
Cito JLo DApel SKhurshid S(2016)Developer targeted analytics: supporting software development decisions with runtime informationProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering10.1145/2970276.2975939(892-895)Online publication date: 25-Aug-2016
https://dl.acm.org/doi/10.1145/2970276.2975939

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents