Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
Authors:
Sanjay Vishwakarma,
Francis Harkins,
Siddharth Golecha,
Vishal Sharathchandra Bajpe,
Nicolas Dupuis,
Luca Buratti,
David Kremer,
Ismael Faro,
Ruchir Puri,
Juan Cruz-Benito
Abstract:
Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability…
▽ More
Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit - a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
Teaching quantum computing with an interactive textbook
Authors:
James R. Wootton,
Francis Harkins,
Nicholas T. Bronn,
Almudena Carrera Vazquez,
Anna Phan,
Abraham T. Asfaw
Abstract:
Quantum computing is a technology that promises to offer significant advantages during the coming decades. Though the technology is still in a prototype stage, the last few years have seen many of these prototype devices become accessible to the public. This has been accompanied by the open-source development of the software required to use and test quantum hardware in increasingly sophisticated w…
▽ More
Quantum computing is a technology that promises to offer significant advantages during the coming decades. Though the technology is still in a prototype stage, the last few years have seen many of these prototype devices become accessible to the public. This has been accompanied by the open-source development of the software required to use and test quantum hardware in increasingly sophisticated ways. Such tools provide new education opportunities, not just for quantum computing specifically, but also more broadly for quantum information science and even quantum physics as a whole. In this paper we present a case study of one education resource which aims to take advantage of the opportunities: the open-source online textbook `Learn Quantum Computation using Qiskit'. An overview of the topics covered is given, as well as an explanation of the approach taken for each.
△ Less
Submitted 16 December, 2020;
originally announced December 2020.
