Computer Science > Software Engineering
[Submitted on 8 Jan 2024 (v1), last revised 23 Jul 2024 (this version, v4)]
Title:An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks
View PDF HTML (experimental)Abstract:Stakeholders constantly make assumptions in the development of deep learning (DL) frameworks. These assumptions are related to various types of software artifacts (e.g., requirements, design decisions, and technical debt) and can turn out to be invalid, leading to system failures. Existing approaches and tools for assumption management usually depend on manual identification of assumptions. However, assumptions are scattered in various sources (e.g., code comments, commits, and issues) of DL framework development, and manually identifying assumptions has high costs (e.g., time and resources). The objective of the study is to evaluate different classification models for the purpose of identification with respect to assumptions from the point of view of developers and users in the context of DL framework projects (i.e., issues, pull requests, and commits) on GitHub. We constructed a new and largest dataset (i.e., AssuEval) of assumptions collected from the TensorFlow and Keras repositories on GitHub; explored the performance of seven non-transformers based models (e.g., Support Vector Machine, Classification and Regression Trees), the ALBERT model, and three large language models (i.e., ChatGPT, Claude, and Gemini) for identifying assumptions on the AssuEval dataset. The study results show that ALBERT achieves the best performance (f1-score: 0.9584) for identifying assumptions on the AssuEval dataset, which is much better than the other models (the 2nd best f1-score is 0.8858, achieved by the Claude 3.5 Sonnet model). Though ChatGPT, Claude, and Gemini are popular large language models, we do not recommend using them to identify assumptions in DL framework development because of their low performance. This study provides researchers with the largest dataset of assumptions for further research and helps practitioners better understand assumptions and how to manage them in their projects.
Submission history
From: Peng Liang [view email][v1] Mon, 8 Jan 2024 03:50:03 UTC (3,589 KB)
[v2] Wed, 10 Jan 2024 00:46:31 UTC (3,589 KB)
[v3] Wed, 20 Mar 2024 08:06:33 UTC (3,883 KB)
[v4] Tue, 23 Jul 2024 00:50:15 UTC (4,372 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.