Software Engineering
See recent articles
Showing new listings for Friday, 27 September 2024
- [1] arXiv:2409.17166 [pdf, html, other]
-
Title: ScriptSmith: A Unified LLM Framework for Enhancing IT Operations via Automated Bash Script Generation, Assessment, and RefinementOishik Chatterjee, Pooja Aggarwal, Suranjana Samanta, Ting Dai, Prateeti Mohapatra, Debanjana Kar, Ruchi Mahindru, Steve Barbieri, Eugen Postea, Brad Blancett, Arthur De MagalhaesComments: Under ReviewSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
In the rapidly evolving landscape of site reliability engineering (SRE), the demand for efficient and effective solutions to manage and resolve issues in site and cloud applications is paramount. This paper presents an innovative approach to action automation using large language models (LLMs) for script generation, assessment, and refinement. By leveraging the capabilities of LLMs, we aim to significantly reduce the human effort involved in writing and debugging scripts, thereby enhancing the productivity of SRE teams. Our experiments focus on Bash scripts, a commonly used tool in SRE, and involve the CodeSift dataset of 100 tasks and the InterCode dataset of 153 tasks. The results show that LLMs can automatically assess and refine scripts efficiently, reducing the need for script validation in an execution environment. Results demonstrate that the framework shows an overall improvement of 7-10% in script generation.
- [2] arXiv:2409.17434 [pdf, html, other]
-
Title: Harnessing the Potential of Gen-AI Coding Assistants in Public Sector Software DevelopmentSubjects: Software Engineering (cs.SE)
The study on GitHub Copilot by GovTech Singapore's Engineering Productivity Programme (EPP) reveals significant potential for AI Code Assistant tools to boost developer productivity and improve application quality in the public sector. Highlighting the substantial benefits for the public sector, the study observed an increased productivity (coding / tasks speed increased by 21-28%), which translates into accelerated development, and quicker go-to-market, with a notable consensus (95%) that the tool increases developer satisfaction. Particularly, junior developers experienced considerable efficiency gains and reduced coding times, illustrating Copilot's capability to enhance job satisfaction by easing routine tasks. This advancement allows for a sharper focus on complex projects, faster learning, and improved code quality. Recognising the strategic importance of these tools, the study recommends the development of an AI Framework to maximise such benefits while cautioning against potential over-reliance without solid foundational programming skills. It also advises public sector developers to classify their code as "Open" to use Gen-AI Coding Assistant tools on the Cloud like GitHub Copilot and to consider self-hosted tools like Codeium or Code Llama for confidential code to leverage technology efficiently within the public sector framework. With up to 8,000 developers, comprising both public officers and vendors developing applications for the public sector and its customers, there is significant potential to enhance productivity.
- [3] arXiv:2409.17473 [pdf, html, other]
-
Title: An exploratory analysis of Community-based Question-Answering Platforms and GPT-3-driven Generative AI: Is it the end of online community-based learning?Subjects: Software Engineering (cs.SE)
The advent of Large Language Model-driven tools like ChatGPT offers software engineers an interactive alternative to community question-answering (CQA) platforms like Stack Overflow. While Stack Overflow provides benefits from the accumulated crowd-sourced knowledge, it often suffers from unpleasant comments, reactions, and long waiting times. In this study, we assess the efficacy of ChatGPT in providing solutions to software engineering questions by analyzing its performance specifically against human answers on 2564 Python and JavaScript questions posted between January 2022 and December 2022 in Stack Overflow. We parse the questions and answers from Stack Overflow, then collect the answers to the same questions from ChatGPT through API, and employ four textual and four cognitive metrics to compare the answers generated by ChatGPT with the accepted answers provided by human subject matter experts to find out the potential reasons for which future knowledge seekers may prefer ChatGPT over CQA platforms. Our analysis indicates that ChatGPT's responses are 66\% shorter and share 35\% more words with the questions, showing a 25\% increase in positive sentiment compared to human responses. ChatGPT's answers' accuracy rate is between 71 to 75\%, with a variation in response characteristics between JavaScript and Python. Additionally, our findings suggest a recent 38\% decrease in comment interactions on Stack Overflow, indicating a shift in community engagement patterns. A supplementary survey with 14 Python and JavaScript professionals validated these findings. While ChatGPT offers quicker, more concise responses, the implications of reduced community involvement warrant further investigation.
- [4] arXiv:2409.17561 [pdf, html, other]
-
Title: TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language ModelsSubjects: Software Engineering (cs.SE)
Software testing is a crucial phase in the software life cycle, helping identify potential risks and reduce maintenance costs. With the advancement of Large Language Models (LLMs), researchers have proposed an increasing number of LLM-based software testing techniques, particularly in the area of test case generation. Despite the growing interest, limited efforts have been made to thoroughly evaluate the actual capabilities of LLMs in this task.
In this paper, we introduce TestBench, a benchmark for class-level LLM-based test case generation. We construct a dataset of 108 Java programs from 9 real-world, large-scale projects on GitHub, each representing a different thematic domain. We then design three distinct types of prompts based on context descriptions, including self-contained context, full context, and simple context. Besides, we propose a fine-grained evaluation framework that considers five aspects of test cases: syntactic correctness, compilation correctness, test correctness, code coverage rate, and defect detection rate. Furthermore, we propose a heuristic algorithm to repair erroneous test cases generated by LLMs. We evaluate CodeLlama-13b, GPT-3.5, and GPT-4 on the TestBench, and our experimental results indicate that larger models demonstrate a greater ability to effectively utilize contextual information, thus generating higher-quality test cases. Smaller models may struggle with the noise introduced by the extensive information contained within the full context. However, when using the simplified version, namely the simple context, which is derived from the full context via abstract syntax tree analysis, the performance of these models improves significantly. Our analysis highlights the current progress and pinpoints future directions to further enhance the effectiveness of models by handling contextual information for test case generation. - [5] arXiv:2409.17844 [pdf, html, other]
-
Title: Software Security Analysis in 2030 and Beyond: A Research RoadmapComments: 25 pages single-column, Invited article for ACM TOSEMSubjects: Software Engineering (cs.SE); Cryptography and Security (cs.CR)
As our lives, our businesses, and indeed our world economy become increasingly reliant on the secure operation of many interconnected software systems, the software engineering research community is faced with unprecedented research challenges, but also with exciting new opportunities. In this roadmap paper, we outline our vision of Software Security Analysis for the software systems of the future. Given the recent advances in generative AI, we need new methods to evaluate and maximize the security of code co-written by machines. As our software systems become increasingly heterogeneous, we need practical approaches that work even if some functions are automatically generated, e.g., by deep neural networks. As software systems depend evermore on the software supply chain, we need tools that scale to an entire ecosystem. What kind of vulnerabilities exist in future systems and how do we detect them? When all the shallow bugs are found, how do we discover vulnerabilities hidden deeply in the system? Assuming we cannot find all security flaws, how can we nevertheless protect our system? To answer these questions, we start our research roadmap with a survey of recent advances in software security, then discuss open challenges and opportunities, and conclude with a long-term perspective for the field.
- [6] arXiv:2409.17885 [pdf, html, other]
-
Title: Sentiment Analysis of ML Projects: Bridging Emotional Intelligence and Code QualitySubjects: Software Engineering (cs.SE)
This study explores the intricate relationship between sentiment analysis (SA) and code quality within machine learning (ML) projects, illustrating how the emotional dynamics of developers affect the technical and functional attributes of software projects. Recognizing the vital role of developer sentiments, this research employs advanced sentiment analysis techniques to scrutinize affective states from textual interactions such as code comments, commit messages, and issue discussions within high-profile ML projects. By integrating a comprehensive dataset of popular ML repositories, this analysis applies a blend of rule-based, machine learning, and hybrid sentiment analysis methodologies to systematically quantify sentiment scores. The emotional valence expressed by developers is then correlated with a spectrum of code quality indicators, including the prevalence of bugs, vulnerabilities, security hotspots, code smells, and duplication instances. Findings from this study distinctly illustrate that positive sentiments among developers are strongly associated with superior code quality metrics manifested through reduced bugs and lower incidence of code smells. This relationship underscores the importance of fostering positive emotional environments to enhance productivity and code craftsmanship. Conversely, the analysis reveals that negative sentiments correlate with an uptick in code issues, particularly increased duplication and heightened security risks, pointing to the detrimental effects of adverse emotional conditions on project health.
- [7] arXiv:2409.18048 [pdf, html, other]
-
Title: Next-Gen Software Engineering: AI-Assisted Big ModelsComments: 18 pages, 6 figures, preprintSubjects: Software Engineering (cs.SE); Emerging Technologies (cs.ET)
The effectiveness of model-driven software engineering (MDSE) has been demonstrated in the context of complex software; however, it has not been widely adopted due to the requisite efforts associated with model development and maintenance, as well as the specific modelling competencies required for MDSE. Concurrently, artificial intelligence (AI) methods, particularly machine learning (ML) methods, have demonstrated considerable abilities when applied to the huge code bases accessible on open-source coding platforms. The so-called big code provides the basis for significant advances in empirical software engineering, as well as in the automation of coding processes and improvements in software quality with the use of AI. The objective of this paper is to facilitate a synthesis between these two significant domains of software engineering (SE), namely models and AI in SE. The paper provides an overview of the current status of AI-assisted software engineering. In light of the aforementioned considerations, a vision of AI-assisted Big Models in SE is put forth, with the aim of capitalising on the advantages inherent to both approaches in the context of software development. Finally, the new paradigm of pair modelling in MDSE is proposed.
New submissions (showing 7 of 7 entries)
- [8] arXiv:2409.17513 (cross-list from cs.CR) [pdf, html, other]
-
Title: Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted CodeComments: 6 pages, 2 figuresSubjects: Cryptography and Security (cs.CR); Computation and Language (cs.CL); Machine Learning (cs.LG); Software Engineering (cs.SE)
Ransomware and other forms of malware cause significant financial and operational damage to organizations by exploiting long-standing and often difficult-to-detect software vulnerabilities. To detect vulnerabilities such as buffer overflows in compiled code, this research investigates the application of unidirectional transformer-based embeddings, specifically GPT-2. Using a dataset of LLVM functions, we trained a GPT-2 model to generate embeddings, which were subsequently used to build LSTM neural networks to differentiate between vulnerable and non-vulnerable code. Our study reveals that embeddings from the GPT-2 model significantly outperform those from bidirectional models of BERT and RoBERTa, achieving an accuracy of 92.5% and an F1-score of 89.7%. LSTM neural networks were developed with both frozen and unfrozen embedding model layers. The model with the highest performance was achieved when the embedding layers were unfrozen. Further, the research finds that, in exploring the impact of different optimizers within this domain, the SGD optimizer demonstrates superior performance over Adam. Overall, these findings reveal important insights into the potential of unidirectional transformer-based approaches in enhancing cybersecurity defenses.
- [9] arXiv:2409.17876 (cross-list from cs.CY) [pdf, html, other]
-
Title: Why Companies "Democratise" Artificial Intelligence: The Case of Open Source Software DonationsComments: 30 pages, 1 figure, 5 tablesSubjects: Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Companies claim to "democratise" artificial intelligence (AI) when they donate AI open source software (OSS) to non-profit foundations or release AI models, among others, but what does this term mean and why do they do it? As the impact of AI on society and the economy grows, understanding the commercial incentives behind AI democratisation efforts is crucial for ensuring these efforts serve broader interests beyond commercial agendas. Towards this end, this study employs a mixed-methods approach to investigate commercial incentives for 43 AI OSS donations to the Linux Foundation. It makes contributions to both research and practice. It contributes a taxonomy of both individual and organisational social, economic, and technological incentives for AI democratisation. In particular, it highlights the role of democratising the governance and control rights of an OSS project (i.e., from one company to open governance) as a structural enabler for downstream goals, such as attracting external contributors, reducing development costs, and influencing industry standards, among others. Furthermore, OSS donations are often championed by individual developers within companies, highlighting the importance of the bottom-up incentives for AI democratisation. The taxonomy provides a framework and toolkit for discerning incentives for other AI democratisation efforts, such as the release of AI models. The paper concludes with a discussion of future research directions.
- [10] arXiv:2409.18060 (cross-list from cs.HC) [pdf, html, other]
-
Title: Infering Alt-text For UI Icons With Large Language Models During App DevelopmentSubjects: Human-Computer Interaction (cs.HC); Software Engineering (cs.SE)
Ensuring accessibility in mobile applications remains a significant challenge, particularly for visually impaired users who rely on screen readers. User interface icons are essential for navigation and interaction and often lack meaningful alt-text, creating barriers to effective use. Traditional deep learning approaches for generating alt-text require extensive datasets and struggle with the diversity and imbalance of icon types. More recent Vision Language Models (VLMs) require complete UI screens, which can be impractical during the iterative phases of app development. To address these issues, we introduce a novel method using Large Language Models (LLMs) to autonomously generate informative alt-text for mobile UI icons with partial UI data. By incorporating icon context, that include class, resource ID, bounds, OCR-detected text, and contextual information from parent and sibling nodes, we fine-tune an off-the-shelf LLM on a small dataset of approximately 1.4k icons, yielding IconDesc. In an empirical evaluation and a user study IconDesc demonstrates significant improvements in generating relevant alt-text. This ability makes IconDesc an invaluable tool for developers, aiding in the rapid iteration and enhancement of UI accessibility.
Cross submissions (showing 3 of 3 entries)
- [11] arXiv:2402.18401 (replaced) [pdf, html, other]
-
Title: DevPhish: Exploring Social Engineering in Software Supply Chain Attacks on DevelopersHossein Siadati, Sima Jafarikhah, Elif Sahin, Terrence Brent Hernandez, Elijah Lorenzo Tripp, Denis KhryashchevComments: 7 pages, 2 figuresSubjects: Software Engineering (cs.SE); Cryptography and Security (cs.CR)
The Software Supply Chain (SSC) has captured considerable attention from attackers seeking to infiltrate systems and undermine organizations. There is evidence indicating that adversaries utilize Social Engineering (SocE) techniques specifically aimed at software developers. That is, they interact with developers at critical steps in the Software Development Life Cycle (SDLC), such as accessing Github repositories, incorporating code dependencies, and obtaining approval for Pull Requests (PR) to introduce malicious code. This paper aims to comprehensively explore the existing and emerging SocE tactics employed by adversaries to trick Software Engineers (SWEs) into delivering malicious software. By analyzing a diverse range of resources, which encompass established academic literature and real-world incidents, the paper systematically presents an overview of these manipulative strategies within the realm of the SSC. Such insights prove highly beneficial for threat modeling and security gap analysis.
- [12] arXiv:2403.15676 (replaced) [pdf, html, other]
-
Title: AC4: Algebraic Computation Checker for Circuit Constraints in ZKPsComments: 24 pages, 5 figuresSubjects: Software Engineering (cs.SE); Computation and Language (cs.CL); Cryptography and Security (cs.CR)
Zero-knowledge proof (ZKP) systems have surged attention and held a fundamental role in contemporary cryptography. Zero-knowledge succinct non-interactive argument of knowledge (zk-SNARK) protocols dominate the ZKP usage, implemented through arithmetic circuit programming paradigm. However, underconstrained or overconstrained circuits may lead to bugs. The former refers to circuits that lack the necessary constraints, resulting in unexpected solutions and causing the verifier to accept a bogus witness, and the latter refers to circuits that are constrained excessively, resulting in lacking necessary solutions and causing the verifier to accept no witness. This paper introduces a novel approach for pinpointing two distinct types of bugs in ZKP circuits. The method involves encoding the arithmetic circuit constraints to polynomial equation systems and solving them over finite fields by the computer algebra system. The classification of verification results is refined, greatly enhancing the expressive power of the system. A tool, AC4, is proposed to represent the implementation of the method. Experiments show that AC4 demonstrates a increase in the checked ratio, showing a 29% improvement over Picus, a checker for Circom circuits, and a 10% improvement over halo2-analyzer, a checker for halo2 circuits. Within a solvable range, the checking time has also exhibited noticeable improvement, demonstrating a magnitude increase compared to previous efforts.
- [13] arXiv:2408.16151 (replaced) [pdf, html, other]
-
Title: Automatic Library Migration Using Large Language Models: First ResultsComments: Accepted at 18th International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1-7, 2024Subjects: Software Engineering (cs.SE)
Despite being introduced only a few years ago, Large Language Models (LLMs) are already widely used by developers for code generation. However, their application in automating other Software Engineering activities remains largely unexplored. Thus, in this paper, we report the first results of a study in which we are exploring the use of ChatGPT to support API migration tasks, an important problem that demands manual effort and attention from developers. Specifically, in the paper, we share our initial results involving the use of ChatGPT to migrate a client application to use a newer version of SQLAlchemy, an ORM (Object Relational Mapping) library widely used in Python. We evaluate the use of three types of prompts (Zero-Shot, One-Shot, and Chain Of Thoughts) and show that the best results are achieved by the One-Shot prompt, followed by the Chain Of Thoughts. Particularly, with the One-Shot prompt we were able to successfully migrate all columns of our target application and upgrade its code to use new functionalities enabled by SQLAlchemy's latest version, such as Python's asyncio and typing modules, while preserving the original code behavior.
- [14] arXiv:2409.13590 (replaced) [pdf, html, other]
-
Title: Toward Interactive Optimization of Source Code Differences: An Empirical Study of Its PerformanceComments: (C) 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksJournal-ref: Proceedings of the 24th IEEE International Conference on Source Code Analysis and Manipulation, 224-234, 2024Subjects: Software Engineering (cs.SE)
A source code difference (diff) indicates changes made by comparing new and old source codes, and it can be utilized in code reviews to help developers understand the changes made to the code. Although many diff generation methods have been proposed, existing automatic methods may generate nonoptimal diffs, hindering reviewers from understanding the changes. In this paper, we propose an interactive approach to optimize diffs. Users can provide feedback for the points of a diff that should not be matched but are or parts that should be matched but are not. The edit graph is updated based on this feedback, enabling users to obtain a more optimal diff. We simulated our proposed method by applying a search algorithm to empirically assess the number of feedback instances required and the amount of diff optimization resulting from the feedback to investigate the potential of this approach. The results of 23 GitHub projects confirm that 92% of nonoptimal diffs can be addressed with less than four feedback actions in the ideal case.
- [15] arXiv:2409.15228 (replaced) [pdf, html, other]
-
Title: A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language ModelsSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Large language models (LLMs) like GitHub Copilot and ChatGPT have emerged as powerful tools for code generation, significantly enhancing productivity and accelerating software development. However, existing benchmarks primarily focus on general code generation without considering API-oriented code generation, i.e., generating code that invokes APIs from specific libraries. Given the growing demand for API-oriented code generation, there is a pressing need for a systematic and automated approach to evaluate LLM on API-oriented code generation. To address this gap, we propose AutoAPIEval, a lightweight and automated framework designed to evaluate the capabilities of LLMs in API-oriented code generation. Our framework works with any library that provides API documentation and focuses on two unit tasks: API recommendation and code example generation, along with four metrics to evaluate the generated APIs and code examples, such as the proportion of incorrect API recommendations for Task 1, and the proportion of code examples where no specific API is invoked and uncompilable/unexecutable code examples for Task 2. In addition, we conducted a case study on three LLMs (ChatGPT, MagiCoder, and DeepSeek Coder) and Java Runtime Environment 8 to demonstrate the framework's effectiveness. Our findings reveal substantial variability in LLM performance across tasks, with ChatGPT adhering better to instructions, while sharing similar effectiveness in code example generation with its counterparts (i.e., MagiCoder and DeekSeek Coder). We also identify key factors associated with code quality, such as API popularity and model confidence, and build classifiers that achieve high accuracy in detecting incorrect API recommendations and erroneous code examples. Retrieval-augmented generation enhances the quality of code generated by LLMs, though its effectiveness varies across different LLMs.
- [16] arXiv:2404.09486 (replaced) [pdf, other]
-
Title: MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming ProblemsComments: EMNLP 2024Subjects: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Software Engineering (cs.SE)
Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at this https URL.
- [17] arXiv:2409.14697 (replaced) [pdf, html, other]
-
Title: QueenV2: Future of Quantum Circuit SimulationSubjects: Quantum Physics (quant-ph); Software Engineering (cs.SE)
A state vector-based quantum circuit simulation can provide accurate results for the development and validation of quantum computing algorithms, without being affected by noise interference. However, existing quantum circuit simulators have consistently underperformed due to inadequate integration with quantum circuits and high-performance computing architectures. To tackle the challenges in quantum computing, we propose QueenV2, which builds upon the design principles of Queen and elevates performance to a new level. Experimental results on the NVIDIA RTX-4090 demonstrate that QueenV2 achieves up to a 40x improvement in gate performance and a 5x improvement in circuit performance compared to hyQuas. Furthermore, QueenV2 realizes a 137x speedup in gate benchmarks and a 14x speedup in circuit performance relative to NVIDIA cuQuantum, enabled by gate fusion via the IBM Qiskit toolkit. By eliminating reliance on third-party libraries, QueenV2 is positioned to significantly accelerate quantum circuit simulation, thus promoting the development of innovative accelerators and quantum algorithms.
- [18] arXiv:2409.16341 (replaced) [pdf, html, other]
-
Title: Quality Matters: Evaluating Synthetic Data for Tool-Using LLMsSubjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Software Engineering (cs.SE)
Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.