research-article

Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation

Authors:

Pedro Valero-Lara,

Keita Teranishi,

Prasanna Balaprakash, and

Jeffrey VetterAuthors Info & Claims

ICPP Workshops '23: Proceedings of the 52nd International Conference on Parallel Processing Workshops

August 2023

Pages 136 - 144

https://doi.org/10.1145/3605731.3605886

Published: 07 September 2023 Publication History

Abstract

We evaluate AI-assisted generative capabilities on fundamental numerical kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV, Jacobi Stencil, and CG. We test the generated kernel codes for a variety of language-supported programming models, including (1) C++ (e.g., OpenMP [including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g., OpenMP [including offload] and OpenACC), (3) Python (e.g., numpy, Numba, cuPy, and pyCUDA), and (4) Julia (e.g., Threads, CUDA.jl, AMDGPU.jl, and KernelAbstractions.jl). We use the GitHub Copilot capabilities powered by the GPT-based OpenAI Codex available in Visual Studio Code as of April 2023 to generate a vast amount of implementations given simple <kernel> + <programming model> + <optional hints> prompt variants. To quantify and compare the results, we propose a proficiency metric around the initial 10 suggestions given for each prompt. Results suggest that the OpenAI Codex outputs for C++ correlate with the adoption and maturity of programming models. For example, OpenMP and CUDA score really high, whereas HIP is still lacking. We found that prompts from either a targeted language such as Fortran or the more general-purpose Python can benefit from adding code keywords, while Julia prompts perform acceptably well for its mature programming models (e.g., Threads and CUDA.jl). We expect for these benchmarks to provide a point of reference for each programming model’s community. Overall, understanding the convergence of large language models, AI, and HPC is crucial due to its rapidly evolving nature and how it is redefining human-computer interactions.

References

[1]

AMD. 2022. AMD ROCm v5.2 Release. https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html#amd-rocm-v5-2-release

[2]

J. W. Backus and W. P. Heising. 1964. Fortran. IEEE Transactions on Electronic Computers EC-13, 4 (1964), 382–385. https://doi.org/10.1109/PGEC.1964.263818

[3]

Tim Besard, Christophe Foket, and Bjorn De Sutter. 2018. Effective Extensible Programming: Unleashing Julia on GPUs. IEEE Transactions on Parallel and Distributed Systems (2018). https://doi.org/10.1109/TPDS.2018.2872064 arxiv:1712.03112 [cs.PL]

Digital Library

[4]

Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. 2017. Julia: A Fresh Approach to Numerical Computing. SIAM Rev. 59, 1 (Jan. 2017), 65–98. https://doi.org/10.1137/141000671 arXiv:http://dx.doi.org/10.1137/141000671

Digital Library

[5]

Robert W. Brennan and Jonathan Lesage. 2023. Exploring the Implications of OpenAI Codex on Education for Industry 4.0. In Service Oriented, Holonic and Multi-Agent Manufacturing Systems for Industry of the Future, Theodor Borangiu, Damien Trentesaux, and Paulo Leitão (Eds.). Springer International Publishing, Cham, 254–266.

[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

[7]

H. Carter Edwards, Christian R. Trott, and Daniel Sunderland. 2014. Kokkos: Enabling manycore performance portability through polymorphic memory access patterns. J. Parallel and Distrib. Comput. 74, 12 (2014), 3202–3216. https://doi.org/10.1016/j.jpdc.2014.07.003 Domain-Specific Languages and High-Level Frameworks for High-Performance Computing.

Digital Library

[8]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).

[9]

Valentin Churavy, Dilum Aluthge, Lucas C Wilcox, James Schloss, Simon Byrne, Maciej Waruszewski, Julian Samaroo, Ali Ramadhan, Meredith, Simeon Schaub, Jake Bolewski, Anton Smirnov, Charles Kawczynski, Chris Hill, Jinguo Liu, Oliver Schulz, Oscar, Páll Haraldsson, Takafumi Arakaki, and Tim Besard. 2022. JuliaGPU/KernelAbstractions.jl: v0.8.3. https://doi.org/10.5281/zenodo.6742177

[10]

Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt, and Wannes Meert. 2022. Code Generation Using Machine Learning: A Systematic Review. IEEE Access 10 (2022), 82434–82455. https://doi.org/10.1109/ACCESS.2022.3196347

[11]

Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 1136–1142. https://doi.org/10.1145/3545945.3569823

Digital Library

[12]

Jack Dongarra, Robert Graybill, William Harrod, Robert Lucas, Ewing Lusk, Piotr Luszczek, Janice Mcmahon, Allan Snavely, Jeffrey Vetter, Katherine Yelick, Sadaf Alam, Roy Campbell, Laura Carrington, Tzu-Yi Chen, Omid Khalili, Jeremy Meredith, and Mustafa Tikir. 2008. DARPA’s HPCS Program: History, Models, Tools, Languages. In Advances in COMPUTERS. Advances in Computers, Vol. 72. Elsevier, 1–100. https://doi.org/10.1016/S0065-2458(08)00001-6

[13]

Jack Dongarra et al.2011. The International Exascale Software Project roadmap. The International Journal of High Performance Computing Applications 25, 1 (2011), 3–60. https://doi.org/10.1177/1094342010391989 arXiv:https://doi.org/10.1177/1094342010391989

Digital Library

[14]

Li Fei-Fei, R. Fergus, and P. Perona. 2006. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 4 (2006), 594–611. https://doi.org/10.1109/TPAMI.2006.79

Digital Library

[15]

Michael Fink. 2004. Object Classification from a Single Example Utilizing Class Relevance Metrics. In Advances in Neural Information Processing Systems, L. Saul, Y. Weiss, and L. Bottou (Eds.). Vol. 17. MIT Press. https://proceedings.neurips.cc/paper_files/paper/2004/file/ef1e491a766ce3127556063d49bc2f98-Paper.pdf

[16]

James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). Association for Computing Machinery, New York, NY, USA, 10–19. https://doi.org/10.1145/3511861.3511863

Digital Library

[17]

Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines 30 (2020), 681–694.

Digital Library

[18]

William F. Godoy, Pedro Valero-Lara, T. Elise Dettling, Christian Trefftz, Ian Jorquera, Thomas Sheehy, Ross G. Miller, Marc Gonzalez-Tallada, Jeffrey S. Vetter, and Valentin Churavy. 2023. Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes. arxiv:2303.06195 [cs.DC]

[19]

Thomas Helmuth and Peter Kelly. 2021. PSB2: The Second Program Synthesis Benchmark Suite. In Proceedings of the Genetic and Evolutionary Computation Conference (Lille, France) (GECCO ’21). Association for Computing Machinery, New York, NY, USA, 785–794. https://doi.org/10.1145/3449639.3459285

Digital Library

[20]

Julia Hirschberg and Christopher D. Manning. 2015. Advances in natural language processing. Science 349, 6245 (2015), 261–266. https://doi.org/10.1126/science.aaa8685 arXiv:https://www.science.org/doi/pdf/10.1126/science.aaa8685

[21]

Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-Programming? An Empirical Study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 319–321. https://doi.org/10.1145/3510454.3522684

Digital Library

[22]

Zheming Jin. 2021. The Rodinia Benchmarks in SYCL. Technical Report. Oak Ridge National Lab.(ORNL), Oak Ridge, TN (United States).

[23]

Zhemin Jin. 2023. Hecbench. https://github.com/zjin-lcf/HeCBench.

[24]

Srinath Kailasa, Tingyu Wang, Lorena A. Barba, and Timo Betcke. 2023. PyExaFMM: an exercise in designing high-performance software with Python and Numba. arxiv:2303.08394 [cs.SE]

[25]

Andreas Klöckner. [n. d.]. pycuda 2022.2.2 documentation. https://documen.tician.de/pycuda/. Accessed: 2023-04-20.

[26]

Andreas Klöckner, Nicolas Pinto, Yunsup Lee, Bryan Catanzaro, Paul Ivanov, and Ahmed Fasih. 2012. PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation. Parallel Comput. 38, 3 (2012), 157–174. https://doi.org/10.1016/j.parco.2011.09.001

Digital Library

[27]

Tobias Knopp. 2014. Experimental multi-threading support for the Julia programming language. In 2014 First Workshop for High Performance Technical Computing in Dynamic Languages. IEEE, 1–5.

Digital Library

[28]

Douglas Kothe, Stephen Lee, and Irene Qualters. 2019. Exascale Computing in the United States. Computing in Science & Engineering 21, 1 (2019), 17–29. https://doi.org/10.1109/MCSE.2018.2875366

Digital Library

[29]

Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015. Numba: A LLVM-based Python JIT compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC. 1–6.

Digital Library

[30]

Dongdong Lu, Jie Wu, Yongxiang Sheng, Peng Liu, and Mengmeng Yang. 2020. Analysis of the popularity of programming languages in open source software communities. In 2020 International Conference on Big Data and Social Sciences (ICBDSS). 111–114. https://doi.org/10.1109/ICBDSS51270.2020.00033

[31]

Nhan Nguyen and Sarah Nadi. 2022. An Empirical Evaluation of GitHub Copilot’s Code Suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR ’22). Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/3524842.3528470

Digital Library

[32]

ROYUD Nishino and Shohei Hido Crissman Loomis. 2017. Cupy: A numpy-compatible library for nvidia gpu calculations. 31st confernce on neural information processing systems 151, 7 (2017).

[33]

NVIDIA. 2022. CUDA Toolkit Documentation - v11.7.0. https://developer.nvidia.com/cuda-toolkit

[34]

NVIDIA. 2022. The API reference guide for Thrust, the CUDA C++ template library.https://docs.nvidia.com/cuda/thrust/index.html

[35]

OpenACC Architecture Review Board. 2020. OpenACC Application Program Interface Version 3.1. https://www.openacc.org/sites/default/files/inline-images/Specification/OpenACC-3.1-final.pdf

[36]

OpenMP Architecture Review Board. 2021. OpenMP Application Program Interface Version 5.2. https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf

[37]

Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In 2022 IEEE Symposium on Security and Privacy (SP). 754–768. https://doi.org/10.1109/SP46214.2022.9833571

[38]

Inc. Preferred Networks and Inc. Preferred Infrastructure. [n. d.]. CuPy – NumPy & SciPy for GPU. https://docs.cupy.dev/en/stable/. Accessed: 2023-04-20.

[39]

Julian Samaroo, Valentin Churavy, Wiktor Phillips, Ali Ramadhan, Jason Barmparesos, Julia TagBot, Ludovic Räss, Michel Schanen, Tim Besard, Anton Smirnov, Takafumi Arakaki, Stephan Antholzer, Alessandro, Chris Elrod, Matin Raayai, and Tom Hu. 2022. JuliaGPU/AMDGPU.jl: v0.4.1. https://doi.org/10.5281/zenodo.6949520

[40]

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models(ICER ’22). Association for Computing Machinery, New York, NY, USA, 27–43. https://doi.org/10.1145/3501385.3543957

Digital Library

[41]

Dominik Sobania, Martin Briesch, and Franz Rothlauf. 2022. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of Github Copilot and Genetic Programming. In Proceedings of the Genetic and Evolutionary Computation Conference (Boston, Massachusetts) (GECCO ’22). Association for Computing Machinery, New York, NY, USA, 1019–1027. https://doi.org/10.1145/3512290.3528700

Digital Library

[42]

Bjarne Stroustrup. 2013. The C++ programming language. Pearson Education.

[43]

Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages. https://doi.org/10.1145/3491101.3519665

Digital Library

[44]

Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. 2011. The NumPy array: a structure for efficient numerical computation. Computing in science & engineering 13, 2 (2011), 22–30.

[45]

Guido Van Rossum 2007. Python Programming Language. In USENIX annual technical conference, Vol. 41. Santa Clara, CA, 1–36.

[46]

J. S. Vetter, R. Brightwell, M. Gokhale, P. McCormick, R. Ross, J. Shalf, K. Antypas, D. Donofrio, T. Humble, C. Schuman, B. Van Essen, S. Yoo, A. Aiken, D. Bernholdt, S. Byna, K. Cameron, F. Cappello, B. Chapman, A. Chien, M. Hall, R. Hartman-Baker, Z. Lan, M. Lang, J. Leidel, S. Li, R. Lucas, J. Mellor-Crummey, P. Peltz Jr., T. Peterka, M. Strout, and J. Wilke. 2018. Extreme Heterogeneity 2018 - Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity. Technical Report. USDOE Office of Science (SC) (United States). https://doi.org/10.2172/1473756

[47]

Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from a Few Examples: A Survey on Few-Shot Learning. ACM Comput. Surv. 53, 3, Article 63 (jun 2020), 34 pages. https://doi.org/10.1145/3386252

Digital Library

[48]

Michel Wermelinger. 2023. Using GitHub Copilot to Solve Simple Programming Problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 172–178. https://doi.org/10.1145/3545945.3569830

Digital Library

[49]

Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the Quality of GitHub Copilot’s Code Generation. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering (Singapore, Singapore) (PROMISE 2022). Association for Computing Machinery, New York, NY, USA, 62–71. https://doi.org/10.1145/3558489.3559072

Digital Library

Cited By

Palkowski MGruzewski M(2024)GPT-Driven Source-to-Source Transformation for Generating Compilable Parallel CUDA Code for Nussinov’s AlgorithmElectronics10.3390/electronics1303048813:3(488)Online publication date: 24-Jan-2024
https://doi.org/10.3390/electronics13030488
Zeng FGan WWang YYu P(2023)Distributed Training of Large Language Models2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00126(840-847)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00126
Kadosh THasabnis NMattson TPinter YOren G(2023)Quantifying OpenMP: Statistical Insights into Usage and Adoption2023 IEEE High Performance Extreme Computing Conference (HPEC)10.1109/HPEC58863.2023.10363459(1-7)Online publication date: 25-Sep-2023
https://doi.org/10.1109/HPEC58863.2023.10363459

Index Terms

Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
1. Computer systems organization
  1. Architectures
    1. Other architectures
      1. Heterogeneous (hybrid) systems
    2. Parallel architectures
      1. Multicore architectures
  2. Embedded and cyber-physical systems
    1. System on a chip

Recommendations

Evaluation of directive-based performance portable programming models

We present an extended exploration of the performance portability of directives provided by OpenMP 4 and OpenACC to program various types of node architectures with attached accelerators. To do this, we use examples of algorithms with varying ...
Read More
Parallel Programming Models for Dense Linear Algebra on Heterogeneous Systems

We present a review of the current best practices in parallel programming models for dense linear algebra DLA on heterogeneous architectures. We consider multicore CPUs, stand alone manycore coprocessors, GPUs, and combinations of these. Of interest is ...
Read More
Many Cores, Many Models: GPU Programming Model vs. Vendor Compatibility Overview
SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

In recent history, GPUs became a key driver of compute performance in HPC. With the installation of the Frontier supercomputer, they became the enablers of the Exascale era; further largest-scale installations are in progress (Aurora, El Capitan, ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICPP Workshops '23: Proceedings of the 52nd International Conference on Parallel Processing Workshops

August 2023

217 pages

ISBN:9798400708428

DOI:10.1145/3605731

Copyright © 2023 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 September 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICPP-W 2023

ICPP-W 2023: 52nd International Conference on Parallel Processing Workshops

August 7 - 10, 2023

UT, Salt Lake City, USA

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
249
Total Downloads

Downloads (Last 12 months)249
Downloads (Last 6 weeks)14

Other Metrics

View Author Metrics

Citations

Cited By

Palkowski MGruzewski M(2024)GPT-Driven Source-to-Source Transformation for Generating Compilable Parallel CUDA Code for Nussinov’s AlgorithmElectronics10.3390/electronics1303048813:3(488)Online publication date: 24-Jan-2024
https://doi.org/10.3390/electronics13030488
Zeng FGan WWang YYu P(2023)Distributed Training of Large Language Models2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00126(840-847)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00126
Kadosh THasabnis NMattson TPinter YOren G(2023)Quantifying OpenMP: Statistical Insights into Usage and Adoption2023 IEEE High Performance Extreme Computing Conference (HPEC)10.1109/HPEC58863.2023.10363459(1-7)Online publication date: 25-Sep-2023
https://doi.org/10.1109/HPEC58863.2023.10363459

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents