research-article

FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs

Authors:

Philipp Schaad,

Timo Schneider,

Alexandru Calotoiu,

Alexandros Nikolaos Ziogas,

Torsten HoeflerAuthors Info & Claims

SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 88, Pages 1 - 15

https://doi.org/10.1145/3581784.3613214

Published: 11 November 2023 Publication History

Abstract

The current hardware landscape and application scale is driving performance engineers towards writing bespoke optimizations. Verifying such optimizations, and generating minimal failing cases, is important for robustness in the face of changing program conditions, such as inputs and sizes. However, isolation of minimal test-cases from existing applications and generating new configurations are often difficult due to side effects on the system state, mostly related to dataflow. This paper introduces FuzzyFlow: a fault localization and test case extraction framework designed to test program optimizations. We leverage dataflow program representations to capture a fully reproducible system state and area-of-effect for optimizations to enable fast checking for semantic equivalence. To reduce testing time, we design an algorithm for minimizing test inputs, trading off memory for recomputation. We demonstrate FuzzyFlow on example use cases in real-world applications where the approach provides up to 528 times faster optimization testing and debugging compared to traditional approaches.

Supplemental Material

MP4 File - SC23 paper presentation recording for "FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs"

SC23 paper presentation recording for "FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs", by Philipp Schaad, Timo Schneider, Tal Ben-Nun, Alexandru Calotoiu, Alexandros Nikolaos Ziogas and Torsten Hoefler

Download
205.84 MB

References

[1]

Måns Ivar Andersson and Stefano Markidis. 2023. A Case Study on DaCe Portability & Performance for Batched Discrete Fourier Transforms. In Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region. ACM, New York, NY, USA, 55--63.

Digital Library

[2]

Roberto Baldoni, Emilio Coppa, Daniele Cono D'elia, Camil Demetrescu, and Irene Finocchi. 2019. A Survey of Symbolic Execution Techniques. Comput. Surveys 51, 3 (5 2019), 1--39.

Digital Library

[3]

Wenlei Bao, Sriram Krishnamoorthy, Louis-Noël Pouchet, Fabrice Rastello, and P. Sadayappan. 2016. PolyCheck: dynamic verification of iteration space transformations on affine programs. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. ACM, New York, NY, USA, 539--554.

Digital Library

[4]

Tal Ben-Nun, Johannes De Fine Licht, Alexandros N. Ziogas, Timo Schneider, and Torsten Hoefler. 2019. Stateful dataflow multigraphs: A data-centric model for performance portability on heterogeneous architectures. International Conference for High Performance Computing, Networking, Storage and Analysis, SC (2019).

Digital Library

[5]

Tal Ben-Nun, Linus Groner, Florian Deconinck, Tobias Wicky, Eddie Davis, Johann Dahm, Oliver D. Elbert, Rhea George, Jeremy McGibbon, Lukas Trumper, Elynn Wu, Oliver Fuhrer, Thomas Schulthess, and Torsten Hoefler. 2022. Productive Performance Engineering for Weather and Climate Modeling with Python. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1--14.

[6]

Maciej Besta and Torsten Hoefler. 2022. Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis. (5 2022). http://arxiv.org/abs/2205.09702

[7]

A.S. Boujarwah and K. Saleh. 1997. Compiler test case generation methods: a survey and assessment. Information and Software Technology 39, 9 (1 1997), 617--625.

[8]

Ian Briggs and Pavel Panchekha. 2022. Choosing mathematical function implementations for speed and accuracy. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation. ACM, New York, NY, USA, 522--535.

Digital Library

[9]

Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In USENIX Symposium on Operating Systems Design and Implementation (OSDI). 209.

[10]

J. M. Caron and P. A. Darnell. 1990. Bugfind: a tool for debugging optimizing compilers. ACM SIGPLAN Notices 25, 1 (1 1990), 17--22.

Digital Library

[11]

Ines Chami, Bryan Perozzi, Christopher Ré, Kevin Murphy, and Sami Abu-El-Haija. 2022. Machine Learning on Graphs: A Model and Comprehensive Taxonomy. Technical Report. 1--64 pages.

[12]

Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2021. A Survey of Compiler Testing. Comput. Surveys 53, 1 (1 2021), 1--36.

Digital Library

[13]

GB Dantzig and DR Fulkerson. 1955. On The Max Flow Min Cut Theorem of Networks.

[14]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). http://arxiv.org/abs/1810.04805

[15]

Jack Edmonds and Richard M Karp. 1972. Theoretical Improvements in Algorithmic Efficiency for Network Flow Problems. Journal of the ACM (JACM) 19, 2 (1972), 248--264.

Digital Library

[16]

Michael D Ernst. 1994. Practical fine-grained static slicing of optimized code. Technical Report.

[17]

European Centre for Medium-Range Weather Forecasts. 2003. CLOUDSC cloud microphysics scheme. https://github.com/ecmwf-ifs/dwarf-p-cloudsc.

[18]

Azadeh Farzan and Victor Nicolet. 2019. Modular divide-and-conquer parallelization of nested loops. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, New York, NY, USA, 610--624.

Digital Library

[19]

Jeanne Ferrante, Karl J. Ottenstein, and Joe D. Warren. 1987. The program dependence graph and its use in optimization. ACM Transactions on Programming Languages and Systems 9, 3 (7 1987), 319--349.

Digital Library

[20]

Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020. AFL ++ : Combining Incremental Steps of Fuzzing Research. In 14th USENIX Workshop on Offensive Technologies (WOOT 20). https://github.com/AFLplusplushttps://www.usenix.org/conference/woot20/presentation/fioraldi

[21]

Lester Randolph Ford and Delbert R Fulkerson. 1956. Maximal flow through a network. Canadian Journal of Mathematics 8 (1956), 399--404.

[22]

Chris Hathhorn, Chucky Ellison, and Grigore Roşu. 2015. Defining the undefinedness of C. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Vol. 2015-June. ACM, New York, NY, USA, 336--345.

Digital Library

[23]

Susan Horwitz, Thomas Reps, and David Binkley. 1990. Interprocedural slicing using dependence graphs. ACM Transactions on Programming Languages and Systems 12, 1 (1 1990), 26--60.

Digital Library

[24]

Intel. 2003. Intel oneAPI Math Kernel Library. https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl.html.

[25]

Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly. 2007. Dryad. In Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007. ACM, New York, NY, USA, 59--72.

Digital Library

[26]

Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. 2021. Data Movement Is All You Need: A Case Study on Optimizing Transformers. Proceedings of Machine Learning and Systems (2021).

[27]

Fredrik Kjolstad, Stephen Chou, David Lugato, Shoaib Kamil, and Saman Amarasinghe. 2017. Taco: A tool to generate tensor algebra kernels. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 943--948.

[28]

Maria Kotsifakou, Prakalp Srivastava, Matthew D. Sinclair, Rakesh Komuravelli, Vikram Adve, and Sarita Adve. 2018. HPVM. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Vol. 53. ACM, New York, NY, USA, 68--80.

Digital Library

[29]

Ramana Kumar, Magnus O. Myreen, Michael Norrish, and Scott Owens. 2014. CakeML: a verified implementation of ML. In Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. ACM, New York, NY, USA, 179--191.

Digital Library

[30]

C. Lattner and V. Adve. 2004. LLVM: A compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generation and Optimization, 2004. CGO 2004. IEEE, 75--86.

[31]

Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Oleksandr Zinenko. 2021. MLIR: Scaling Compiler Infrastructure for Domain Specific Computation. In 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). IEEE, 2--14.

Digital Library

[32]

Xavier Leroy. 2006. Formal certification of a compiler back-end or: programming a compiler with a proof assistant. ACM SIGPLAN Notices 41, 1 (1 2006), 42--54.

Digital Library

[33]

Xavier Leroy. 2009. Formal verification of a realistic compiler. Commun. ACM 52, 7 (7 2009), 107--115.

Digital Library

[34]

Christian Lindig. 2005. Random testing of C calling conventions. In Proceedings of the Sixth sixth international symposium on Automated analysis-driven debugging - AADEBUG'05. ACM Press, New York, New York, USA, 3--12.

Digital Library

[35]

LLVM Compiler Infrastructure. 2003. LLVM bugpoint tool: design and usage. https://llvm.org/docs/Bugpoint.html.

[36]

Valentin J.M. Manes, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2021. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering 47, 11 (11 2021), 2312--2331.

[37]

William Mansky, Dennis Griffith, and Elsa L. Gunter. 2014. Specifying and Executing Optimizations for Parallel Programs. Electronic Proceedings in Theoretical Computer Science 159 (8 2014), 58--70.

[38]

William Mansky, Elsa L. Gunter, Dennis Griffith, and Michael D. Adams. 2016. Specifying and executing optimizations for generalized control flow graphs. Science of Computer Programming 130 (11 2016), 2--23.

Digital Library

[39]

William M McKeeman. 1998. Differential Testing For Software. Digital Technical Journal 10, 1 (1998), 100--107.

[40]

Ghassan Misherghi and Zhendong Su. 2006. HDD: hierarchical delta debugging. In Proceedings of the 28th international conference on Software engineering. ACM, New York, NY, USA, 142--151.

Digital Library

[41]

Mozilla Security. 2017. funfuzz. https://github.com/MozillaSecurity/funfuzz.

[42]

Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martín Abadi. 2013. Naiad. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles. ACM, New York, NY, USA, 439--455.

Digital Library

[43]

George C. Necula. 2000. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation - PLDI '00. ACM Press, New York, New York, USA, 83--94.

Digital Library

[44]

George C. Necula and Peter Lee. 1998. The design and implementation of a certifying compiler. ACM SIGPLAN Notices 33, 5 (5 1998), 333--344.

Digital Library

[45]

Israt Nisa, Aravind Sukumaran-Rajam, Sureyya Emre Kurt, Changwan Hong, and P. Sadayappan. 2018. Sampled Dense Matrix Multiplication for High-Performance Machine Learning. In 2018 IEEE 25th International Conference on High Performance Computing (HiPC). IEEE, 32--41.

[46]

Karl J. Ottenstein and Linda M. Ottenstein. 1984. The program dependence graph in a software development environment. ACM SIGPLAN Notices 19, 5 (5 1984), 177--184.

Digital Library

[47]

A. Pnueli, O. Shtrichman, and M. Siegel. 1998. The Code Validation Tool (CVT). International Journal on Software Tools for Technology Transfer (STTT) 2, 2 (12 1998), 192--201.

[48]

A. Pnueli, M. Siegel, and E. Singerman. 1998. Translation validation. 151--166.

[49]

Amir Pnueli and Anna Zaks. 2006. Translation Validation of Interprocedural Optimizations. In 4th International Workshop on Software Verification and Validation (SVV 2006).

[50]

David A. Ramos and Dawson R. Engler. 2011. Practical, Low-Effort Equivalence Verification of Real Code. In International Conference on Computer Aided Verification. 669--685.

[51]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, New York, NY, USA, 335--346.

Digital Library

[52]

Martin C Rinard. 2003. Credible Compilation. (2003).

[53]

Xavier Rival. 2004. Symbolic transfer function-based approaches to certified compilation. ACM SIGPLAN Notices 39, 1 (1 2004), 1--13.

Digital Library

[54]

Flash Sheridan. 2007. Practical testing of a C99 compiler using output comparison. Software: Practice and Experience 37, 14 (11 2007), 1475--1488.

[55]

Software Engineering Institute - Carnegie Mellon University. 2010. CERT BFF. https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=507974.

[56]

Chengnian Sun, Yuanbo Li, Qirun Zhang, Tianxiao Gu, and Zhendong Su. 2018. Perses: Syntax-Guided Program Reduction. In Proceedings of the 40th International Conference on Software Engineering, Vol. 2018-January. ACM, New York, NY, USA, 361--371.

Digital Library

[57]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS). http://arxiv.org/abs/1706.03762

[58]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2017. Graph Attention Networks. (10 2017). http://arxiv.org/abs/1710.10903

[59]

Daniel Weise, Roger F. Crew, Michael Ernst, and Bjarne Steensgaard. 1994. Value dependence graphs. In Proceedings of the 21st ACM SIGPLAN-SIGACT symposium on Principles of programming languages - POPL '94. ACM Press, New York, New York, USA, 297--310.

Digital Library

[60]

Mark Weiser. 1984. Program Slicing. IEEE Transactions on Software Engineering SE-10, 4 (1984), 352--357.

Digital Library

[61]

David B. Whalley. 1994. Automatic isolation of compiler errors. ACM Transactions on Programming Languages and Systems 16, 5 (9 1994), 1648--1659.

Digital Library

[62]

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. 2021. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems 32, 1 (1 2021), 4--24.

[63]

Baowen Xu, Ju Qian, Xiaofang Zhang, Zhongqiang Wu, and Lin Chen. 2005. A brief survey of program slicing. ACM SIGSOFT Software Engineering Notes 30, 2 (3 2005), 1--36.

Digital Library

[64]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation - PLDI '11. ACM Press, New York, New York, USA, 283.

Digital Library

[65]

Zhongming Yu, Guohao Dai, Guyue Huang, Yu Wang, and Huazhong Yang. 2021. Exploiting Online Locality and Reduction Parallelism for Sampled Dense Matrix Multiplication on GPUs. In 2021 IEEE 39th International Conference on Computer Design (ICCD), Vol. 2021-October. IEEE, 567--574.

[66]

Anna Zaks and Amir Pnueli. 2008. CoVaC: Compiler Validation by Program Analysis of the Cross-Product. In FM 2008: Formal Methods. Springer Berlin Heidelberg, Berlin, Heidelberg, 35--51.

Digital Library

[67]

Michal Zalewski. 2013. american fuzzy lop. https://lcamtuf.coredump.cx/afl/.

[68]

Andreas Zeller. 1999. Yesterday, my program worked. Today, it does not. Why? ACM SIGSOFT Software Engineering Notes 24, 6 (11 1999), 253--267.

Digital Library

[69]

A. Zeller and R. Hildebrandt. 2002. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering 28, 2 (2002), 183--200.

Digital Library

[70]

Alexandros Nikolaos Ziogas, Tal Ben-Nun, Guillermo Indalecio Fernández, Timo Schneider, Mathieu Luisier, and Torsten Hoefler. 2019. A data-centric approach to extreme-scale ab initio dissipative quantum transport simulations. International Conference for High Performance Computing, Networking, Storage and Analysis, SC (2019).

Digital Library

[71]

Alexandros Nikolaos Ziogas, Tal Ben-Nun, Timo Schneider, and Torsten Hoefler. 2021. NPBench: A Benchmarking Suite for High-Performance NumPy. In Proceedings of the ACM International Conference on Supercomputing. ACM, New York, NY, USA, 63--74.

Digital Library

[72]

Alexandros Nikolaos Ziogas, Grzegorz Kwasniewski, Tal Ben-Nun, Timo Schneider, and Torsten Hoefler. 2022. Deinsum: Practically I/O Optimal Multi-Linear Algebra. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1--15.

[73]

Lenore Zuck, Amir Pnueli, Yi Fang, and Benjamin Goldberg. 2002. VOC: A Translation Validator for Optimizing Compilers. Electronic Notes in Theoretical Computer Science 65, 2 (4 2002), 2--18.

[74]

Lenore Zuck, Amir Pnueli, Benjamin Goldberg, Clark Barrett, Yi Fang, and Ying Hu. 2005. Translation and Run-Time Validation of Loop Transformations. Formal Methods in System Design 27, 3 (11 2005), 335--360.

Digital Library

Cited By

Kale VYan HMukherjee SMayo JTeranishi KRutledge ROrso A(2024)Toward Automated Detection of Portability Bugs in Kokkos Parallel ProgramsSC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SCW63240.2024.00029(180-188)Online publication date: 17-Nov-2024
https://doi.org/10.1109/SCW63240.2024.00029

Index Terms

FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs
1. Software and its engineering
2. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning

Recommendations

Grey-box concolic testing on binary code
ICSE '19: Proceedings of the 41st International Conference on Software Engineering

We present grey-box concolic testing, a novel path-based test case generation method that combines the best of both white-box and grey-box fuzzing. At a high level, our technique systematically explores execution paths of a program under test as in white-...
First international competition on software testing
Abstract
Tool competitions are a special form of comparative evaluation, where each tool has a team of developers or supporters associated that makes sure the tool is properly configured to show its best possible performance. In several research areas, ... $^{}$
Continuous test suite augmentation in software product lines
SPLC '13: Proceedings of the 17th International Software Product Line Conference

Software Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, to generate test cases for individual products in product lines more efficiently. In this paper we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2023

1428 pages

ISBN:9798400701092

DOI:10.1145/3581784

Chair:
Dorian Arnold,
Program Chair:
Rosa M Badia,
Program Co-chair:
Kathryn Mohror

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SC '23

Sponsor:

SIGHPC

SC '23: International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 17, 2023

CO, Denver, USA

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
258
Total Downloads

Downloads (Last 12 months)173
Downloads (Last 6 weeks)4

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kale VYan HMukherjee SMayo JTeranishi KRutledge ROrso A(2024)Toward Automated Detection of Portability Bugs in Kokkos Parallel ProgramsSC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SCW63240.2024.00029(180-188)Online publication date: 17-Nov-2024
https://doi.org/10.1109/SCW63240.2024.00029

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten