research-article

RMARaceBench: A Microbenchmark Suite to Evaluate Race Detection Tools for RMA Programs

Authors:

Simon Schwitanski,

Sven Klotz, and

Matthias S. MüllerAuthors Info & Claims

SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

November 2023

Pages 205 - 214

https://doi.org/10.1145/3624062.3624087

Published: 12 November 2023 Publication History

Abstract

Parallel programming models with Remote Memory Access (RMA), such as MPI RMA, OpenSHMEM, and GASPI, allow processes to modify the memory of other processes directly. Special care is required to avoid concurrent conflicting accesses that lead to data races across processes resulting in undefined behavior. To detect such RMA races, different race detection tools have been developed. However, there is currently no possibility to compare their effectiveness systematically. In this paper, we present RMARaceBench, a microbenchmark suite to evaluate the detection capabilities of RMA race detection tools for MPI RMA, OpenSHMEM, and GASPI. It consists of about 100 synthetic race test cases for each programming model, aiming to cover all possible race scenarios. Using RMARaceBench, we evaluate two MPI RMA race detectors implemented in the correctness tool suites MUST and PARCOACH. The evaluation shows that RMARaceBench can pinpoint the strengths and weaknesses of RMA race detectors.

References

[1]

Tassadit Célia Aitkaci, Marc Sergent, Emmanuelle Saillard, Denis Barthou, and Guillaume Papauré. 2021. Dynamic Data Race Detection for MPI-RMA Programs. In EuroMPI ’21 - European MPI Users’ Group Meeting.

[2]

Simone Atzeni, Ganesh Gopalakrishnan, Zvonimir Rakamaric, Dong H. Ahn, Ignacio Laguna, Martin Schulz, Gregory L. Lee, Joachim Protze, and Matthias S. Müller. 2016. ARCHER: Effectively Spotting Data Races in Large OpenMP Applications. In 2016 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2016, Chicago, IL, USA, May 23-27, 2016. IEEE, 53–62. https://doi.org/10.1109/IPDPS.2016.68

[3]

Md Abdullah Shahneous Bari, Ujjwal Arora, Varun Hegde, Tony Curtis, and Barbara M. Chapman. 2021. OpenSHMEM Checker - A Clang Based Static Checker for OpenSHMEM. In 20th International Symposium on Parallel and Distributed Computing, ISPDC 2021, Cluj-Napoca, Romania, July 28-30, 2021. IEEE, 41–48. https://doi.org/10.1109/ISPDC52870.2021.9521645

[4]

Zhezhe Chen, James Dinan, Zhen Tang, Pavan Balaji, Hua Zhong, Jun Wei, Tao Huang, and Feng Qin. 2014. MC-Checker: Detecting Memory Consistency Errors in MPI One-Sided Applications. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2014, New Orleans, LA, USA, November 16-21, 2014. IEEE, 499–510. https://doi.org/10.1109/SC.2014.46

Digital Library

[5]

Thanh-Dang Diep, Karl Fürlinger, and Nam Thoai. 2018. MC-CChecker: A Clock-Based Approach to Detect Memory Consistency Errors in MPI One-Sided Applications. In EuroMPI’18: European MPI Users’ Group Meeting, Barcelona, Spain, September 23-26, 2018. ACM, 9:1–9:11. https://doi.org/10.1145/3236367.3236369

Digital Library

[6]

GASPI Forum. 2017. GASPI: Global Address Space Programming Interface 17.1. https://raw.githubusercontent.com/GASPI-Forum/GASPI-Forum.github.io/master/standards/GASPI-17.1.pdf [online; accessed 26-September-2023].

[7]

Tobias Hilbrich, Martin Schulz, Bronis R. de Supinski, and Matthias S. Müller. 2009. MUST: A Scalable Approach to Runtime Error Detection in MPI Programs. In Proceedings of the 3rd International Workshop on Parallel Tools for High Performance Computing, September 2009, ZIH, Dresden. Springer, 53–66. https://doi.org/10.1007/978-3-642-11261-4_5

[8]

Torsten Hoefler, James Dinan, Rajeev Thakur, Brian Barrett, Pavan Balaji, William Gropp, and Keith D. Underwood. 2015. Remote Memory Access Programming in MPI-3. ACM Transactions on Parallel Computing 2, 2 (2015), 9:1–9:26. https://doi.org/10.1145/2780584

Digital Library

[9]

Chung-Hsing Hsu, Neena Imam, Akhil Langer, Sreeram Potluri, and Chris J. Newburn. 2020. An Initial Assessment of NVSHMEM for High Performance Computing. In 2020 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2020, New Orleans, LA, USA, May 18-22, 2020. IEEE, 617–626. https://doi.org/10.1109/IPDPSW50202.2020.00104

[10]

Tim Jammer, Alexander Hück, Jan-Patrick Lehr, Joachim Protze, Simon Schwitanski, and Christian H. Bischof. 2022. Towards a Hybrid MPI Correctness Benchmark Suite. In EuroMPI/USA’22: European MPI Users’ Group Meeting, Chattanooga, TN, USA, September 26-28, 2022. ACM, 46–56. https://doi.org/10.1145/3555819.3555853

Digital Library

[11]

Jithin Jose, Sreeram Potluri, Karen Tomko, and Dhabaleswar K. Panda. 2013. Designing Scalable Graph500 Benchmark with Hybrid MPI+OpenSHMEM Programming Models. In Supercomputing - 28th International Supercomputing Conference, ISC 2013, Leipzig, Germany, June 16-20, 2013. Proceedings, Vol. 7905. Springer, 109–124. https://doi.org/10.1007/978-3-642-38750-0_9

[12]

Olaf Krzikalla. 2018. Neue Ansätze zur Speicherzugriffsanalyse paralleler Anwendungen mit gemeinsam genutztem Adressraum [New approaches for memory access analysis of parallel applications with a shared address space]. Dissertation. Technische Universtität Dresden. https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-332004

[13]

Ignacio Laguna, Ryan J. Marshall, Kathryn M. Mohror, Martin Ruefenacht, Anthony Skjellum, and Nawrin Sultana. 2019. A large-scale study of MPI usage in open-source HPC applications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, Denver, Colorado, USA, November 17-19, 2019. ACM, 31:1–31:14. https://doi.org/10.1145/3295500.3356176

Digital Library

[14]

Mathieu Laurent, Emmanuelle Saillard, and Martin Quinson. 2021. The MPI Bugs Initiative: a Framework for MPI Verification Tools Evaluation. In 5th IEEE/ACM International Workshop on Software Correctness for HPC Applications, Correctness@SC 2021, St. Louis, MO, USA, November 19, 2021. IEEE, 1–9. https://doi.org/10.1109/Correctness54621.2021.00008

[15]

Jan-Patrick Lehr, Tim Jammer, and Christian H. Bischof. 2021. MPI-CorrBench: Towards an MPI Correctness Benchmark Suite. In HPDC ’21: The 30th International Symposium on High-Performance Parallel and Distributed Computing, Virtual Event, Sweden, June 21-25, 2021. ACM, 69–80. https://doi.org/10.1145/3431379.3460652

Digital Library

[16]

Chunhua Liao, Pei-Hung Lin, Joshua Asplund, Markus Schordan, and Ian Karlin. 2017. DataRaceBench: a benchmark suite for systematic evaluation of data race detection tools. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, Denver, CO, USA, November 12 - 17, 2017. ACM, 11. https://doi.org/10.1145/3126908.3126958

Digital Library

[17]

Yiqian Liu, Noushin Azami, Corbin Walters, and Martin Burtscher. 2022. The Indigo Program-Verification Microbenchmark Suite of Irregular Parallel Code Patterns. In International IEEE Symposium on Performance Analysis of Systems and Software, ISPASS 2022, Singapore, May 22-24, 2022. IEEE, 24–34. https://doi.org/10.1109/ISPASS55109.2022.00003

[18]

Glenn R. Luecke, James Coyle, James Hoekstra, Marina Kraeva, Ying Xu, Mi-Young Park, Elizabeth Kleiman, Olga Weiss, André Wehe, and Melissa Yahya. 2009. The Importance of Run-Time Error Detection. In Proceedings of the 3rd International Workshop on Parallel Tools for High Performance Computing, September 2009, ZIH, Dresden. Springer, 145–155. https://doi.org/10.1007/978-3-642-11261-4_10

[19]

Chi-Keung Luk, Robert S. Cohn, Robert Muth, Harish Patil, Artur Klauser, P. Geoffrey Lowney, Steven Wallace, Vijay Janapa Reddi, and Kim M. Hazelwood. 2005. Pin: building customized program analysis tools with dynamic instrumentation. In Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation, Chicago, IL, USA, June 12-15, 2005. ACM, 190–200. https://doi.org/10.1145/1065010.1065034

Digital Library

[20]

Message Passing Interface Forum. 2021. MPI: A Message-Passing Interface Standard Version 4.0. http://mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf [online; accessed 26-September-2023].

[21]

Nicholas Nethercote and Julian Seward. 2007. Valgrind: a framework for heavyweight dynamic binary instrumentation. In Proceedings of the ACM SIGPLAN 2007 Conference on Programming Language Design and Implementation, San Diego, California, USA, June 10-13, 2007. ACM, 89–100. https://doi.org/10.1145/1250734.1250746

Digital Library

[22]

OpenMP Architecture Review Board. 2021. OpenMP Application Programming Interface Version 5.2. https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf [online; accessed 26-September-2023].

[23]

OpenSHMEM Committee. 2020. OpenSHMEM: Application Programming Interface Version 1.5. http://openshmem.org/site/sites/default/site_files/OpenSHMEM-1.5.pdf [online; accessed 26-September-2023].

[24]

Mi-Young Park and Sang-Hwa Chung. 2009. Detecting Race Conditions in One-Sided Communication of MPI Programs. In 8th IEEE/ACIS International Conference on Computer and Information Science, IEEE/ACIS ICIS 2009, June 1-3, 2009, Shanghai, China. IEEE, 867–872. https://doi.org/10.1109/ICIS.2009.170

Digital Library

[25]

Swaroop Pophale, Oscar R. Hernandez, Stephen W. Poole, and Barbara M. Chapman. 2014. Extending the OpenSHMEM Analyzer to Perform Synchronization and Multi-valued Analysis. In OpenSHMEM and Related Technologies. Experiences, Implementations, and Tools - First Workshop, OpenSHMEM 2014, Annapolis, MD, USA, March 4-6, 2014. Proceedings, Vol. 8356. Springer, 134–148. https://doi.org/10.1007/978-3-319-05215-1_10

Digital Library

[26]

Emmanuelle Saillard, Patrick Carribault, and Denis Barthou. 2014. PARCOACH: Combining static and dynamic validation of MPI collective communications. Int. J. High Perform. Comput. Appl. 28, 4 (2014), 425–434. https://doi.org/10.1177/1094342014552204

Digital Library

[27]

Emmanuelle Saillard, Marc Sergent, Célia Tassadit Ait Kaci, and Denis Barthou. 2022. Static Local Concurrency Errors Detection in MPI-RMA Programs. In Sixth IEEE/ACM International Workshop on Software Correctness for HPC Applications, Correctness@SC 2022, Dallas, TX, USA, November 13-18, 2022. IEEE, 18–26. https://doi.org/10.1109/Correctness56720.2022.00008

[28]

Adrian Schmitz, Joachim Protze, Lechen Yu, Simon Schwitanski, and Matthias S. Müller. 2019. DataRaceOnAccelerator - A Micro-benchmark Suite for Evaluating Correctness Tools Targeting Accelerators. In Euro-Par 2019: Parallel Processing Workshops - Euro-Par 2019 International Workshops, Göttingen, Germany, August 26-30, 2019, Revised Selected Papers, Vol. 11997. Springer, 245–257. https://doi.org/10.1007/978-3-030-48340-1_19

Digital Library

[29]

Simon Schwitanski, Joachim Jenke, Felix Tomski, Christian Terboven, and Matthias S. Müller. 2022. On-the-Fly Data Race Detection for MPI RMA Programs with MUST. In Sixth IEEE/ACM International Workshop on Software Correctness for HPC Applications, Correctness@SC 2022, Dallas, TX, USA, November 13-18, 2022. IEEE, 27–36. https://doi.org/10.1109/Correctness56720.2022.00009

[30]

Konstantin Serebryany, Alexander Potapenko, Timur Iskhodzhanov, and Dmitriy Vyukov. 2011. Dynamic Race Detection with LLVM Compiler. In Runtime Verification - Second International Conference, RV 2011, San Francisco, CA, USA, September 27-30, 2011, Revised Selected Papers, Vol. 7186. Springer, 110–114. https://doi.org/10.1007/978-3-642-29860-8_9

Digital Library

[31]

UPC Consortium. 2017. UPC Language Specifications Version 1.3. https://upc.lbl.gov/publications/upc-spec-1.3.pdf [online; accessed 26-September-2023].

Cited By

Liu YAzami NVanAusdal ABurtscher M(2024)Indigo3: A Parallel Graph Analytics Benchmark Suite for Exploring Implementation Styles and Common BugsACM Transactions on Parallel Computing10.1145/3665251Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3665251

Index Terms

RMARaceBench: A Microbenchmark Suite to Evaluate Race Detection Tools for RMA Programs
1. Computing methodologies
  1. Parallel computing methodologies
2. Software and its engineering
  1. Software organization and properties
    1. Software functional properties
      1. Correctness

Recommendations

Mapping High-Level Concurrency from OpenMP and MPI to ThreadSanitizer Fibers
SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

High-level parallel programming paradigms like MPI and OpenMP allow expressing concurrency independent from the execution unit finally executing the code. Most general-purpose data race detection tools perform thread-centric analyses with the operating ...
Read More
Towards a Hybrid MPI Correctness Benchmark Suite
EuroMPI/USA '22: Proceedings of the 29th European MPI Users' Group Meeting

High-performance computing codes often combine the Message-Passing Interface (MPI) with a shared-memory programming model, e.g., OpenMP, for efficient computations. These so-called hybrid models may issue MPI calls concurrently from different threads ...
Read More
MultiRace: efficient on-the-fly data race detection in multithreaded C++ programs: Research Articles
Parallel and Distributed Systems: Testing and Debugging (PADTAD)

Data race detection is highly essential for debugging multithreaded programs and assuring their correctness. Nevertheless, there is no single universal technique capable of handling the task efficiently, since the data race detection problem is ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

November 2023

2180 pages

ISBN:9798400707858

DOI:10.1145/3624062

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SC-W 2023

SC-W 2023: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

November 12 - 17, 2023

CO, Denver, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
47
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)12

Other Metrics

View Author Metrics

Citations

Cited By

Liu YAzami NVanAusdal ABurtscher M(2024)Indigo3: A Parallel Graph Analytics Benchmark Suite for Exploring Implementation Styles and Common BugsACM Transactions on Parallel Computing10.1145/3665251Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3665251

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents