research-article

RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Model

Authors:

Haoxing RenAuthors Info & Claims

DAC '24: Proceedings of the 61st ACM/IEEE Design Automation Conference

Article No.: 53, Pages 1 - 6

https://doi.org/10.1145/3649329.3657353

Published: 07 November 2024 Publication History

Abstract

This paper presents RTLFixer, a novel framework enabling automatic syntax errors fixing for Verilog code with Large Language Models (LLMs). Despite LLM's promising capabilities, our analysis indicates that approximately 55% of errors in LLM-generated Verilog are syntax-related, leading to compilation failures. To tackle this issue, we introduce a novel debugging framework that employs Retrieval-Augmented Generation (RAG) and ReAct prompting, enabling LLMs to act as autonomous agents in interactively debugging the code with feedback. This framework demonstrates exceptional proficiency in resolving syntax errors, successfully correcting about 98.5% of compilation errors in our debugging dataset, comprising 212 erroneous implementations derived from the VerilogEval benchmark. Our method leads to 32.3% and 10.1% increase in pass@1 success rates in the VerilogEval-Machine and VerilogEval-Human benchmarks, respectively. The source code and benchmark are available at https://github.com/NVlabs/RTLFixer.

References

[1]

Jason Blocklove, et al. 2023. Chip-Chat: Challenges and Opportunities in Conversational Hardware Design. arXiv preprint arXiv:2305.13243 (2023).

[2]

Mark Chen, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).

[3]

Xinyun Chen, et al. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023).

[4]

Nat Friedman. 2021. Introducing GitHub Copilot: your Al pair programmer. (2021). https://github.blog/2021-06-29-introducing-github-copilot-ai-pair-programmer

[5]

Ziwei Ji, et al. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (March 2023), 1--38.

Digital Library

[6]

Shuyang Jiang, et al. 2023. SelfEvolve: A Code Evolution Framework via Large Language Models. arXiv preprint arXiv:2306.02907 (2023).

[7]

Patrick Lewis, et al. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:cs.CL/2005.11401

[8]

Mingjie Liu, et al. 2023. ChipNeMo: Domain-Adapted LLMs for Chip Design. arXiv:cs.CL/2311.00176

[9]

Mingjie Liu, et al. 2023. VerilogEval: Evaluating Large Language Models for Verilog Code Generation. arXiv preprint arXiv:2309.07544 (2023).

[10]

Shuai Lu, et al. 2022. ReACC: A Retrieval-Augmented Code Completion Framework. arXiv:cs.SE/2203.07722

[11]

Yao Lu, et al. 2023. RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model. arXiv preprint arXiv:2308.05345 (2023).

[12]

Suphakit Niwattanakul, et al. 2013. Using of Jaccard coefficient for keywords similarity. In Proceedings of the international multiconference of engineers and computer scientists, Vol. 1. 380--384.

[13]

OpenAI. 2023. OpenAI models api. (2023). https://platform.openai.com/docs/models

[14]

Hammond Pearce, et al. 2020. Dave: Deriving automatically verilog from english. In Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD. 27--32.

Digital Library

[15]

Yujia Qin, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 (2023).

[16]

Erich Schubert, et al. 2017. DBSCAN revisited, revisited: why and how you should (still) use DBSCAN. ACM Transactions on Database Systems (TODS) 42, 3 (2017), 1--21.

Digital Library

[17]

Shailja Thakur, et al. 2023. VeriGen: A Large Language Model for Verilog Code Generation. arXiv preprint arXiv:2308.00708 (2023).

[18]

Jason Wei, et al. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:cs.CL/2201.11903

[19]

Stephen Williams et al. 2002. Icarus verilog: open-source verilog more than a year later. Linux Journal 2002, 99 (2002), 3.

[20]

Shunyu Yao, et al. 2022. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations.

[21]

Fengji Zhang, et al. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570 (2023).

Cited By

Xiao ZHe XWu HYu BGuo Y(2025)EDA-Copilot: A RAG-Powered Intelligent Assistant for EDA ToolsACM Transactions on Design Automation of Electronic Systems10.1145/3715326Online publication date: 27-Jan-2025
https://doi.org/10.1145/3715326
Reddi VYazdanbakhsh A(2025)Architecture 2.0: Foundations of Artificial Intelligence Agents for Modern Computer System DesignComputer10.1109/MC.2024.352164158:2(116-124)Online publication date: Feb-2025
https://doi.org/10.1109/MC.2024.3521641
Paria SDasgupta ABhunia S(2024)Navigating SoC Security Landscape on LLM-Guided PathsProceedings of the Great Lakes Symposium on VLSI 202410.1145/3649476.3660393(252-257)Online publication date: 12-Jun-2024
https://doi.org/10.1145/3649476.3660393
Show More Cited By

Index Terms

RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Model
1. Human-centered computing
  1. Visualization
    1. Visualization techniques
2. Software and its engineering

Index terms have been assigned to the content through auto-classification.

Recommendations

A statistical analysis of syntax errors

A study of errors made by Pascal programmers is described. The results of this study are discussed in relation to compiler syntax error recovery procedures. It is found that syntax errors made in practice are quite simple and occur relatively ...
Syntax errors just aren't natural: improving error reporting with language models
MSR 2014: Proceedings of the 11th Working Conference on Mining Software Repositories

A frustrating aspect of software development is that compiler error messages often fail to locate the actual cause of a syntax error. An errant semicolon or brace can result in many errors reported throughout the file. We seek to find the actual source ...
SynShine: Improved Fixing of Syntax Errors
Novice programmers struggle with the complex syntax of modern programming languages like <sc>Java</sc>, and make lot of syntax errors. The diagnostic syntax error messages from compilers and IDEs are sometimes useful, but often the messages are cryptic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DAC '24: Proceedings of the 61st ACM/IEEE Design Automation Conference

June 2024

2159 pages

ISBN:9798400706011

DOI:10.1145/3649329

Chair:
Vivek De

Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGDA: ACM Special Interest Group on Design Automation
IEEE-CEDA

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2024

Check for updates

Qualifiers

Research-article

Conference

DAC '24

Sponsor:

SIGDA

DAC '24: 61st ACM/IEEE Design Automation Conference

June 23 - 27, 2024

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 1,770 of 5,499 submissions, 32%

Upcoming Conference

DAC '25

Sponsor:
sigda

62nd ACM/IEEE Design Automation Conference

June 22 - 26, 2025

San Francisco , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
221
Total Downloads

Downloads (Last 12 months)221
Downloads (Last 6 weeks)54

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xiao ZHe XWu HYu BGuo Y(2025)EDA-Copilot: A RAG-Powered Intelligent Assistant for EDA ToolsACM Transactions on Design Automation of Electronic Systems10.1145/3715326Online publication date: 27-Jan-2025
https://doi.org/10.1145/3715326
Reddi VYazdanbakhsh A(2025)Architecture 2.0: Foundations of Artificial Intelligence Agents for Modern Computer System DesignComputer10.1109/MC.2024.352164158:2(116-124)Online publication date: Feb-2025
https://doi.org/10.1109/MC.2024.3521641
Paria SDasgupta ABhunia S(2024)Navigating SoC Security Landscape on LLM-Guided PathsProceedings of the Great Lakes Symposium on VLSI 202410.1145/3649476.3660393(252-257)Online publication date: 12-Jun-2024
https://doi.org/10.1145/3649476.3660393
Akyash MM Kamali H(2024)Evolutionary Large Language Models for Hardware Security: A Comparative SurveyProceedings of the Great Lakes Symposium on VLSI 202410.1145/3649476.3660390(496-501)Online publication date: 12-Jun-2024
https://doi.org/10.1145/3649476.3660390
Chaudhuri JThapar DChaudhuri AFirouzi FChakrabarty K(2024)SPICED: Syntactical Bug and Trojan Pattern Identification in A/MS Circuits using LLM-Enhanced Detection2024 IEEE Physical Assurance and Inspection of Electronics (PAINE)10.1109/PAINE62042.2024.10792717(1-7)Online publication date: 12-Nov-2024
https://doi.org/10.1109/PAINE62042.2024.10792717
Xu KZhang GYin XZhuo CSchlichtmann ULi B(2024)Automated C/C++ Program Repair for High-Level Synthesis via Large Language Models2024 ACM/IEEE 6th Symposium on Machine Learning for CAD (MLCAD)10.1109/MLCAD62225.2024.10740262(1-9)Online publication date: 9-Sep-2024
https://doi.org/10.1109/MLCAD62225.2024.10740262
Sharma UWu BKankipati SChhabria VRovinski A(2024)OpenROAD-Assistant: An Open-Source Large Language Model for Physical Design Tasks2024 ACM/IEEE 6th Symposium on Machine Learning for CAD (MLCAD)10.1109/MLCAD62225.2024.10740242(1-7)Online publication date: 9-Sep-2024
https://doi.org/10.1109/MLCAD62225.2024.10740242
Wu BSharma UKankipati SYadav AGeorge BGuntupalli SRovinski AChhabria V(2024)EDA Corpus: A Large Language Model Dataset for Enhanced Interaction with OpenROAD2024 IEEE LLM Aided Design Workshop (LAD)10.1109/LAD62341.2024.10691774(1-5)Online publication date: 28-Jun-2024
https://doi.org/10.1109/LAD62341.2024.10691774
Alsaqer SAlajmi SAhmad IAlfailakawi M(2024)The potential of LLMs in hardware designJournal of Engineering Research10.1016/j.jer.2024.08.001Online publication date: Aug-2024
https://doi.org/10.1016/j.jer.2024.08.001

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten