Machine-Learning-Based Self-Optimizing Compiler Heuristics✱
Authors: 
Raphael Mosaner, 
David Leopoldseder, Wolfgang Kisling, Lukas Stadler, 
Hanspeter MössenböckAuthors Info & Claims
Raphael Mosaner, David Leopoldseder, Wolfgang Kisling, Lukas Stadler, Hanspeter MössenböckAuthors Info & ClaimsPages 98 - 111
Abstract
Compiler optimizations are often based on hand-crafted heuristics to guide the optimization process. These heuristics are designed to benefit the average program and are otherwise static or only customized by profiling information. We propose machine-learning-based self-optimizing compiler heuristics, a novel approach for fitting optimization decisions in a dynamic compiler to specific environments. This is done by updating a machine learning model with extracted performance data at run time. Related work—which primarily targets static compilers—has already shown that machine learning can outperform hand-crafted heuristics. Our approach is specifically designed for dynamic compilation and uses concepts such as deoptimization for transparently switching between generating data and performing machine learning decisions in single program runs. We implemented our approach in the GraalVM, a high-performance production VM for dynamic compilation. When evaluating our approach by replacing loop peeling heuristics with learned models we encountered speedups larger than 30% for several benchmarks and only few slowdowns of up to 7%.
References
[1]
L. Almagor, Keith D. Cooper, Alexander Grosul, Timothy J. Harvey, Steven W. Reeves, Devika Subramanian, Linda Torczon, and Todd Waterman. 2004. Finding Effective Compilation Sequences. In Proceedings of the 2004 ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems(Washington, DC, USA) (LCTES ’04). Association for Computing Machinery, New York, NY, USA, 231–239. https://doi.org/10.1145/997163.997196
[2]
Amir H. Ashouri, William Killian, John Cavazos, Gianluca Palermo, and Cristina Silvano. 2018. A Survey on Compiler Autotuning Using Machine Learning. ACM Comput. Surv. 51, 5, Article 96 (Sept. 2018), 42 pages. https://doi.org/10.1145/3197978
[3]
Joel Auslander, Matthai Philipose, Craig Chambers, Susan J. Eggers, and Brian N. Bershad. 1996. Fast, Effective Dynamic Compilation. SIGPLAN Not. 31, 5 (may 1996), 149–159. https://doi.org/10.1145/249069.231409
[4]
David F. Bacon, Susan L. Graham, and Oliver J. Sharp. 1994. Compiler Transformations for High-Performance Computing. ACM Comput. Surv. 26, 4 (Dec. 1994), 345–420. https://doi.org/10.1145/197405.197406
[5]
Stephen M. Blackburn, Robin Garner, Chris Hoffmann, Asjad M. Khang, Kathryn S. McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z. Guyer, Martin Hirzel, Antony Hosking, Maria Jump, Han Lee, J. Eliot B. Moss, Aashish Phansalkar, Darko Stefanović, Thomas VanDrunen, Daniel von Dincklage, and Ben Wiedermann. 2006. The DaCapo Benchmarks: Java Benchmarking Development and Analysis. In Proceedings of the 21st Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications (Portland, Oregon, USA) (OOPSLA ’06). Association for Computing Machinery, New York, NY, USA, 169–190. https://doi.org/10.1145/1167473.1167488
[6]
François Bodin, Toru Kisuki, Peter Knijnenburg, Mike OBoyle, and Erven Rohou. 1998. Iterative compilation in a non-linear optimisation space. Workshop on Profile and Feedback-Directed Compilation (03 1998). https://hal.inria.fr/inria-00475919/document
[7]
Alexander Brauckmann, Andrés Goens, Sebastian Ertel, and Jeronimo Castrillon. 2020. Compiler-Based Graph Representations for Deep Learning Models of Code. In Proceedings of the 29th International Conference on Compiler Construction (San Diego, CA, USA) (CC 2020). Association for Computing Machinery, New York, NY, USA, 201–211. https://doi.org/10.1145/3377555.3377894
[8]
John Cavazos and Michael F. P. O’Boyle. 2005. Automatic Tuning of Inlining Heuristics. In Proceedings of the 2005 ACM/IEEE Conference on Supercomputing(SC ’05). IEEE Computer Society, USA, 14. https://doi.org/10.1109/SC.2005.14
[9]
Stefano Cazzulani. 2012. Octane: The JavaScript benchmark suite for the modern web.https://blog.chromium.org/2012/08/octane-javascript-benchmark-suite-for.html retrieved May 25 2022.
[10]
Keith D. Cooper, Philip J. Schielke, and Devika Subramanian. 1999. Optimizing for Reduced Code Space Using Genetic Algorithms. SIGPLAN Not. 34, 7 (May 1999), 1–9. https://doi.org/10.1145/315253.314414
[11]
Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks. Mach. Learn. 20, 3 (sep 1995), 273–297. https://doi.org/10.1023/A:1022627411411
[12]
Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, and Hugh Leather. 2022. CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research. In Proceedings of the 20th IEEE/ACM International Symposium on Code Generation and Optimization (Virtual Event, Republic of Korea) (CGO ’22). IEEE Press, 92–105. https://doi.org/10.1109/CGO53902.2022.9741258
[13]
Gilles Duboscq, Lukas Stadler, Thomas Würthinger, Doug Simon, Christian Wimmer, and Hanspeter Mössenböck. 2013. Graal IR: An Extensible Declarative Intermediate Representation. In Proceedings of the Asia-Pacific Programming Languages and Compilers Workshop. 1–9. https://ssw.jku.at/General/Staff/GD/APPLC-2013-paper_12.pdf
[14]
Gilles Duboscq, Thomas Würthinger, and Hanspeter Mössenböck. 2014. Speculation without Regret: Reducing Deoptimization Meta-Data in the Graal Compiler. In Proceedings of the 2014 International Conference on Principles and Practices of Programming on the Java Platform: Virtual Machines, Languages, and Tools (Cracow, Poland) (PPPJ ’14). Association for Computing Machinery, New York, NY, USA, 187–193. https://doi.org/10.1145/2647508.2647521
[15]
Gilles Duboscq, Thomas Würthinger, Lukas Stadler, Christian Wimmer, Doug Simon, and Hanspeter Mössenböck. 2013. An Intermediate Representation for Speculative Optimizations in a Dynamic Compiler. In Proceedings of the 7th ACM Workshop on Virtual Machines and Intermediate Languages (Indianapolis, Indiana, USA) (VMIL ’13). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/2542142.2542143
[16]
Peng fei Chuang, Howard Chen, Gerolf F. Hoflehner, Daniel M. Lavery, and Wei chung Hsu. 2007. Dynamic profile driven code version selection. In the 11th Annual Workshop on the Interaction between Compilers and Computer Architecture. https://www.researchgate.net/publication/228952289_Dynamic_Profile_Driven_Code_Version_Selection
[17]
Grigori Fursin, Albert Cohen, Michael O’Boyle, and Olivier Temam. 2005. A Practical Method for Quickly Evaluating Program Optimizations. In High Performance Embedded Architectures and Compilers, Nacho Conte, Tomband Navarro, Wen-mei W. Hwu, Mateo Valero, and Theo Ungerer (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 29–46. https://doi.org/10.1007/11587514_4
[18]
Grigori Fursin, Cupertino Miranda, Olivier Temam, Mircea Namolaru, Ayal Zaks, Bilha Mendelson, Edwin Bonilla, John Thomson, Hugh Leather, Chris Williams, Michael O’Boyle, Phil Barnard, Elton Ashton, Eric Courtois, and François Bodin. 2008. MILEPOST GCC: machine learning based research compiler. In Proceedings of the GCC Developers’ Summit 2008. https://hal.inria.fr/inria-00294704
[19]
Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica. 2020. NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning. In Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization (San Diego, CA, USA) (CGO 2020). Association for Computing Machinery, New York, NY, USA, 242–255. https://doi.org/10.1145/3368826.3377928
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity Mappings in Deep Residual Networks. In Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 630–645. https://doi.org/10.1007/978-3-319-46493-0_38
[21]
Urs Hölzle, Craig Chambers, and David Ungar. 1992. Debugging Optimized Code with Dynamic Deoptimization. In Proceedings of the ACM SIGPLAN 1992 Conference on Programming Language Design and Implementation (San Francisco, California, USA) (PLDI ’92). Association for Computing Machinery, New York, NY, USA, 32–43. https://doi.org/10.1145/143095.143114
[22]
Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. 1996. Reinforcement Learning: A Survey. J. Artif. Int. Res. 4, 1 (May 1996), 237–285. https://doi.org/10.1613/jair.301
[23]
Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014).
[24]
P.A. Kulkarni, D.B. Whalley, G.S. Tyson, and J.W. Davidson. 2006. Exhaustive optimization phase order space exploration. In International Symposium on Code Generation and Optimization (CGO’06). IEEE Computer Society, 13 pp.–318. https://doi.org/10.1109/CGO.2006.15
[25]
Jeremy Lau, Matthew Arnold, Michael Hind, and Brad Calder. 2006. Online Performance Auditing: Using Hot Optimizations without Getting Burned. In Proceedings of the 27th ACM SIGPLAN Conference on Programming Language Design and Implementation (Ottawa, Ontario, Canada) (PLDI ’06). Association for Computing Machinery, New York, NY, USA, 239–251. https://doi.org/10.1145/1133981.1134010
[26]
Hugh Leather and Chris Cummins. 2020. Machine Learning in Compilers: Past, Present and Future. In 2020 Forum for Specification and Design Languages (FDL). IEEE Computer Society, 1–8. https://doi.org/10.1109/FDL50818.2020.9232934
[27]
David Leopoldseder, Roland Schatz, Lukas Stadler, Manuel Rigger, Thomas Würthinger, and Hanspeter Mössenböck. 2018. Fast-Path Loop Unrolling of Non-Counted Loops to Enable Subsequent Compiler Optimizations. In Proceedings of the 15th International Conference on Managed Languages & Runtimes (Linz, Austria) (ManLang ’18). Association for Computing Machinery, New York, NY, USA, Article 2, 13 pages. https://doi.org/10.1145/3237009.3237013
[28]
David Leopoldseder, Lukas Stadler, Thomas Würthinger, Josef Eisl, Doug Simon, and Hanspeter Mössenböck. 2018. Dominance-Based Duplication Simulation (DBDS): Code Duplication to Enable Compiler Optimizations. In Proceedings of the 2018 International Symposium on Code Generation and Optimization (Vienna, Austria) (CGO 2018). Association for Computing Machinery, New York, NY, USA, 126–137. https://doi.org/10.1145/3168811
[29]
Shun Long and Michael O’Boyle. 2004. Adaptive Java Optimisation Using Instance-Based Learning. In Proceedings of the 18th Annual International Conference on Supercomputing (Malo, France) (ICS ’04). Association for Computing Machinery, New York, NY, USA, 237–246. https://doi.org/10.1145/1006209.1006243
[30]
Rahim Mammadli, Marija Selakovic, Felix Wolf, and Michael Pradel. 2021. Learning to Make Compiler Optimizations More Effective. In Proceedings of the 5th ACM SIGPLAN International Symposium on Machine Programming (Virtual, Canada) (MAPS 2021). Association for Computing Machinery, New York, NY, USA, 9–20. https://doi.org/10.1145/3460945.3464952
[31]
Charith Mendis, Alex Renda, Dr.Saman Amarasinghe, and Michael Carbin. 2019. Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks. In Proceedings of the 36th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 4505–4515. http://proceedings.mlr.press/v97/mendis19a.html
[32]
Raphael Mosaner. 2020. Machine Learning to Ease Understanding of Data Driven Compiler Optimizations. In Companion Proceedings of the 2020 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (Virtual, USA) (SPLASH Companion 2020). Association for Computing Machinery, New York, NY, USA, 4–6. https://doi.org/10.1145/3426430.3429451
[33]
Raphael Mosaner, David Leopoldseder, Wolfgang Kisling, Lukas Stadler, and Hanspeter Mössenböck. 2022. Compilation Forking: A Fast and Flexible Way of Generating Data for Compiler-Internal Machine Learning Tasks. The Art, Science, and Engineering of Programming 7 (06 2022). https://doi.org/10.22152/programming-journal.org/2023/7/3
[34]
Eunjung Park, John Cavazos, and Marco A. Alvarez. 2012. Using Graph-Based Program Characterization for Predictive Modeling. In Proceedings of the Tenth International Symposium on Code Generation and Optimization (San Jose, California) (CGO ’12). Association for Computing Machinery, New York, NY, USA, 196–206. https://doi.org/10.1145/2259016.2259042
[35]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019), 8026–8037. https://dl.acm.org/doi/10.5555/3454287.3455008
[36]
Filip Pizlo. 2014. JetStream Benchmark Suite. http://browserbench.org/JetStream/ retrieved May 25 2022.
[37]
Ricardo Nabinger Sanchez, Jose Nelson Amaral, Duane Szafron, Marius Pirvu, and Mark Stoodley. 2011. Using Machines to Learn Method-Specific Compilation Strategies. In Proceedings of the 9th Annual IEEE/ACM International Symposium on Code Generation and Optimization(CGO ’11). IEEE Computer Society, USA, 257–266. https://doi.org/10.1109/CGO.2011.5764693
[38]
Andreas Sewe, Mira Mezini, Aibek Sarimbekov, and Walter Binder. 2011. Da Capo Con Scala: Design and Analysis of a Scala Benchmark Suite for the Java Virtual Machine. In Proceedings of the 2011 ACM International Conference on Object Oriented Programming Systems Languages and Applications (Portland, Oregon, USA) (OOPSLA ’11). Association for Computing Machinery, New York, NY, USA, 657–676. https://doi.org/10.1145/2048066.2048118
[39]
Mark Stephenson and Saman Amarasinghe. 2005. Predicting unroll factors using supervised classification. In International Symposium on Code Generation and Optimization. IEEE Computer Society, 123–134. https://doi.org/10.1109/CGO.2005.29
[40]
Michele Tartara and Stefano Crespi Reghizzi. 2013. Continuous Learning of Compiler Heuristics. ACM Trans. Archit. Code Optim. 9, 4, Article 46 (Jan. 2013), 25 pages. https://doi.org/10.1145/2400682.2400705
[41]
Mircea Trofin, Yundi Qian, Eugene Brevdo, Zinan Lin, Krzysztof Choromanski, and David Li. 2021. MLGO: a Machine Learning Guided Compiler Optimizations Framework. CoRR abs/2101.04808(2021). arXiv:2101.04808https://arxiv.org/abs/2101.04808
[42]
Huanting Wang, Zhanyong Tang, Cheng Zhang, Jiaqi Zhao, Chris Cummins, Hugh Leather, and Zheng Wang. 2022. Automating Reinforcement Learning Architecture Design for Code Optimization. In Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction(Seoul, South Korea) (CC 2022). Association for Computing Machinery, New York, NY, USA, 129–143. https://doi.org/10.1145/3497776.3517769
[43]
Zheng Wang and Michael O’Boyle. 2018. Machine Learning in Compiler Optimization. Proc. IEEE 106, 11 (Nov 2018), 1879–1901. https://doi.org/10.1109/JPROC.2018.2817118
[44]
Christian Wimmer, Vojin Jovanovic, Erik Eckstein, and Thomas Würthinger. 2017. One Compiler: Deoptimization to Optimized Code. In Proceedings of the 26th International Conference on Compiler Construction (Austin, TX, USA) (CC 2017). Association for Computing Machinery, New York, NY, USA, 55–64. https://doi.org/10.1145/3033019.3033025
[45]
Christian Wimmer and Thomas Würthinger. 2012. Truffle: A Self-Optimizing Runtime System. In Proceedings of the 3rd Annual Conference on Systems, Programming, and Applications: Software for Humanity (Tucson, Arizona, USA) (SPLASH ’12). Association for Computing Machinery, New York, NY, USA, 13–14. https://doi.org/10.1145/2384716.2384723
[46]
Thomas Würthinger, Christian Wimmer, Christian Humer, Andreas Wöß, Lukas Stadler, Chris Seaton, Gilles Duboscq, Doug Simon, and Matthias Grimmer. 2017. Practical Partial Evaluation for High-performance Dynamic Language Runtimes. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (Barcelona, Spain) (PLDI 2017). Association for Computing Machinery, New York, NY, USA, 662–676. https://doi.org/10.1145/3062341.3062381
[47]
Thomas Würthinger, Christian Wimmer, Andreas Wöß, Lukas Stadler, Gilles Duboscq, Christian Humer, Gregor Richards, Doug Simon, and Mario Wolczko. 2013. One VM to Rule Them All. In Proceedings of the 2013 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software(Indianapolis, Indiana, USA) (Onward! 2013). Association for Computing Machinery, New York, NY, USA, 187–204. https://doi.org/10.1145/2509578.2509581
[48]
Mingzhou Zhou, Xipeng Shen, Yaoqing Gao, and Graham Yiu. 2014. Space-Efficient Multi-Versioning for Input-Adaptive Feedback-Driven Program Optimizations. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications(Portland, Oregon, USA) (OOPSLA ’14). Association for Computing Machinery, New York, NY, USA, 763–776. https://doi.org/10.1145/2660193.2660229
Index Terms
- Machine-Learning-Based Self-Optimizing Compiler Heuristics✱
Recommendations
Improving Vectorization Heuristics in a Dynamic Compiler with Machine Learning Models
VMIL 2022: Proceedings of the 14th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate LanguagesOptimizing compilers rely on many hand-crafted heuristics to guide the optimization process. However, the interactions between different optimizations makes their design a difficult task. We propose using machine learning models to either replace such ...
An Exploratory Study on Machine-Learning-Based Hyper-heuristics for the Knapsack Problem
Pattern RecognitionAbstractHyper-heuristics have risen as a recurrent method to solve combinatorial optimization problems since they use a set of heuristics selectively according to the problem state. Although many ideas have been developed to produce hyper-heuristics, a ...
Continuous learning of compiler heuristics
Special Issue on High-Performance Embedded Architectures and CompilersOptimizing programs to exploit the underlying hardware architecture is an important task. Much research has been done on enabling compilers to find the best set of code optimizations that can build the fastest and less resource-hungry executable for a ...
Comments
Information & Contributors
Information
Published In
September 2022
161 pages
Copyright © 2022 Owner/Author.
This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 November 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Oracle Labs
Conference
MPLR '22
MPLR '22: 19th International Conference on Managed Programming Languages and Runtimes
September 14 - 15, 2022
Brussels, Belgium
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 737Total Downloads
- Downloads (Last 12 months)486
- Downloads (Last 6 weeks)45
Reflects downloads up to 12 Sep 2024
Other Metrics
Citations
Cited By
View allView Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in