Analyzing the Rework Time and Severity of Code Debt: A Case Study Using Technical Debt Catalogs

Analyzing the Rework Time and Severity of Code Debt: A Case Study Using Technical Debt Catalogs Bruno Santos de Lima, Rogério Eduardo Garcia arXiv:2002.04695v1 [cs.SE] 11 Feb 2020 Department of Mathematics and Computation, São Paulo State University (UNESP), Faculty of Science and Technology Presidente Prudente, São Paulo, Brazil E-mail: {bruno.s.lima, rogerio.garcia}@unesp.br Abstract—This paper presents a case study analyzing Hibernate ecosystem software projects to investigate and demonstrate Code Debt behavior in relation to severity and rework time. The case study carried out revealed that the Code Debt with severity related to impact on software maintenance is the most representative and has the largest rework times to be paid in the Hibernate ecosystem. Besides, it was found that the distributions of rework times of Code Debt for all severities undergo variations in the initial versions of the development. Keywords—Code Debt, Technical Debt, Technical Debt Analysis, Software Evolution, Mining Software Repositories I. I NTRODUCTION In software development, several standards are followed aiming at achieving the best possible quality in the final product. In this context, Software Engineering is essential, however, it is also necessary to pay attention to how the software source code is implemented so that it is efficient and readable avoiding difficulties in the maintenance stage. For this, to follow technical activities and good programming practices is fundamental. which problems are most frequent, which phases of development are more likely to accumulate these problems, or the same, which aspects have more influence for insertion of new debt instances. However, each problem has a grade of impact and time required for its correction, these two characteristics being variant according to the type of problem. In this context, the objective of this paper is to analyze the behavior of Code Debt from the perspective of severity and time required for resolution, and the possible relations between these two aspects. To enable this analysis, we conducted a case study in which we identified and cataloged the Technical Debt present in the source code belonging to the Hibernate ecosystem software. To guide our analysis we define as research questions: RQ1: What is the distribution of rework time for each severity group of Code Debt? Is this distribution similar in all analyzed software of this ecosystem? RQ2: Is the distribution of rework time for each severity group experiencing significant variation across the multiple versions of each software of the Hibernate ecosystem? In the literature, the Technical Debt metaphor is presented describing the cost-benefit relationship between postponing of technical software development activities to obtain shortterm profits and the long-term consequences [1]. The term was originally used with the focus on coding practices, the metaphor subsequently had its concept expanded including architecture, testing, and even software documentation [2]. RQ3: Analyzing the distribution of the total amount of Code Debt of each severity group, which groups are most representative in the Hibernate ecosystem? The presence of Technical Debt and their accumulation make the maintenance phase more complex and costly, besides degrading the quality of the software. The effective management Technical Debt is a fundamental activity to achieve and maintain the quality of the software [3]. Some cases are reported in which the inattention in the monitoring of Technical Debt, make the maintenance process infeasible in such a way that each new functionality added to the software will undermine existing implementations [4] [5]. Thus, consider information about Technical Debt, analyzing its behavior, evolution, and impacts become important to subsidize its management. This paper is organized as follows: Section II background is described. Section III the related works are presented. Section IV outlines the case study setup followed for the accomplishment of this work. Section V the results obtained after the study of the case are presented and discussed. Section VI presents the final remarks and future works. In the source code perspective, Technical Debt is described as poorly written code that violates good practices or coding rules, denominates Code Debt [6] [3]. In the literature are found works that analyze Technical Debt present in the source code from different perspectives, for example, an identity RQ4: Observing the evolution of the total rework time of each severity group, is the Code Debt controlled in the Hibernate Ecosystem? II. BACKGROUND The Technical Debt metaphor was evidenced in 1992 through an experience report, describing a situation where code quality was exchanged for general short-term gains [7]. The software development community uses the term Technical Debt to label internal software quality problems, future development, and maintenance risks, as well as the costs required to solve these problems which often come from improper shortcuts made by software engineers to bring business benefits [8]. Considering the financial point of view, an analogy with the financial debt can be perceived, because the Technical Debt also results in interest, extra efforts, that need to be paid in the future during the software development process. The term was originally used with a focus on coding practices, and then the metaphor had its concept expanded to include a requirement, architectural, test, defect, documentation, among others [6]. Thus, a Technical Debt may be related to a specific type of artifact software [2]. In the perspective of source code, the Technical Debt is described as poorly written code, duplicates, with low readability, which infringe to some extent the good coding practices [6]. Also, Code smells, classes or complex methods, violations of modularity and the principles of Object Orientation, are commonly called Code Debt [3]. These problems can be identified through the Automatic Static Analysis (ASA) of source code. This type of analysis looks for violations of recommended programming practices that may cause failures or degrade software quality to some extent, this analysis is performed without the source code being executed [9]. III. R ELATED W ORKS In the literature are present studies that analyze from a different perspective the behavior of problems present in source code characterized as Code Debt. Tufanoet al. [10] analyzed the changing history of a set of open-source software projects to understand why Code Smells were introduced and their life cycle. It has been identified that most Code Smells are introduced in the process of creating classes and files or implementing and enhancing functionality. In a complementary study, Jesus et al. [11] identified that the aspects: the size of the project, number of employees and speed of development are the most related with montant the Technical Debt present in the software. Digkas et al. [12] analyzed the evolution of Code Debt present in the Apache ecosystem to investigate how the Code Debt evolves considering each type of problem identified. Was identified 232 different types of problems incurred in Technical Debt, using the SonarQube tool, among them were pointed out the most frequent and those that take longer to be corrected over time. Posteriorly, Digkas et al. [13] identified what types of problems get paid overtime. This work differs from those found in the literature, once, through a case study, aimed at analyzing the behavior of Code Debt regarding severity and rework time, relating these two aspects. IV. C ASE S TUDY S ETUP For the accomplishment of the case study was used as a set of software belonging to the Hibernate ecosystem to have their Technical Debt cataloged. For each software repository a Technical Debt Catalog of source code was constructed, tools were used in this process: SonarQube to identify the problems present in the source code and TD-Tracker for the cataloging of Technical Debt. Then, the analysis stage was carried out. A. Hibernate Ecosystem: Selecting Software Projects In the case study we used project software belonging to the Hibernate ecosystem1 , not yet explored in the literature. To 1 https://github.com/hibernate/ enable an analysis to be carried out, the software of this ecosystem must follow the following criteria that restrict our study: The software should follow the Object-Oriented Paradigm and use the Java programming language in its development; The software project must be under development for a significant period, and may already have been finalized or still under development; The software must have several releases, since one of our analyzes aims to investigate the variation in the distribution Code Debt rework time to each severity group over the development time. The Hibernate ecosystem consists of 36 software repositories, however, it was necessary to disregard some of this software in our case study because they did not meet the criteria established for the scope of this work. Therefore, we selected 10 software repositories of the Hibernate ecosystem (Table I) to compose our case study. TABLE I. S OFTWARE PROJECTS USED IN THE CASE STUDY Project Asciidoctor Extensions Commons Annotations HQL Parser Metamodelgen OGM ORM Search Shards Tools Validator Developers 2 15 12 8 55 461 67 3 10 77 LOC 292 2K 5K 6K 111K 748K 334K 23K 24K 98K B. SonarQube The SonarQube2 , is a code management open platform dedicated to analyzing and measuring the technical quality of source code. It also allows reporting a listing and information on potential issues, to support continuous improvements in the quality of code. There are other tools available to identify problems in source code, however, SonarQube has the characteristic of detecting a large number of problems [13]. Also, was used to identify code debt in other related works [11] [12] [13]. The source code inspection performed by SonarQube indicates to a Issues whenever a part of the code breaks some coding rule. The Issues are classified in the categories: Bugs, Vulnerabilities and Code Smells. Besides being associated to a degree of severity: Blocker: Bug with high probability of impacting software behavior, must be corrected immediately; Critical: Bug with low probability of impacting software behavior, code should be reviewed; Major: Quality failure that may severely affect developer productivity; Minor: Quality failure that may slightly affect developer productivity; It is still estimated to each Issue a time of rework (in minutes), that is, the time needed to solve the problem and thus pay the Code Debt present in a certain part of the source code. Thus, the SonarQube tool was used in this work to perform the static analysis by inspecting the multiple versions of the source code of the software projects, identifying the issues characterized as Code Debt. 2 https://www.sonarqube.org C. TD-Tracker TD-Tracker is a tool that supports the process of identification and mainly of cataloging Technical Debt. It makes use of an integrated catalog used to record the properties of each identified Technical Debt [14] [15]. TD-Tracker allows support for the cataloging of different types of Technical Debt, being they debts of architecture, code, test, documentation, and defects. TD-Tracker was used in conjunction with SonarQube to perform the cataloging of Technical Debt. While SonarQube performs the inspection in the source code and points out the existing problems, in turn, TD-Tracker catalogs these problems in the form of Code Debt. In Figure 1 is illustrated the scheme that characterizes the cataloging process. D. Technical Debt Catalog A catalog was constructed for each Hibernate ecosystem software selected to be part of the case study, totaling ten Technical Debt Catalogs. Each Code Debt instance has a set of attributes that describe its characteristics. The following are each of the attributes of a Code Debt present in the catalogs: • Description: Description of the problem that caused the Technical Debt. • Type: Describes the type of Technical Debt, in this case, only Code Debt is being studied. • Project id: Identifies the version code that the Technical Debt belongs to. • Severity: Identifies the severity of the problem that caused Code Debt, the greater the severity the greater the impact it can cause. • Debt: Identifies the time (in minutes) required for payment of Code Debt, time estimated by SonarQube can be configured, however for this work the standard settings of the SonarQube tool have been maintained. • Location: Identifies the path (package, class, and line) in the Code Debt is located within a version of the software. Technical Debt Catalogs are one of the contributions of this work and can be used in other studies directed to the Technical Debt metaphor, such as: be used as a data set for research in Technical Debt Visualization, since it is reported in literature mappings [3] [16] that this is still an area that needs exploitation. The Technical Debt Catalogs are available at [BLIND]. V. C ASE S TUDY: R ESULTS AND D ISCUSSIONS In this section, we present and discuss the results obtained by the case study, considering the research questions presented in Section I. RQ1: What is the distribution of rework time for each severity group of Code Debt? Is this distribution similar in all analyzed software of this ecosystem? The distribution of the rework time of each Code Debt severity group is presented in Figure 2. Regarding the Code Debt with greatest impact to software functionalities – Blocker and Critical severity groups – (see Figure 2a), three software do not present Blocker severity, they are: Asciidoctor, HQL, Metamodelgen. Regarding other software, most Blocker severity code debt have rework times of 20 minutes; we still observe outliers that indicate Blocker, with rework time of 5, 15 and 30 minutes – the minimum time to solve a Code Debt of Blocker severity, in this ecosystem, is 5 minutes and maximum at 30 minutes. In the Critical severity group – Figure 2b – two software do not present such severity, they are Asciidoctor and HQL; regarding other software, the Critical severity present rework time average between 5 and 10 minutes. Differently from the previous group, for Critical severity there is a significant variation of rework time between different software analyzed. However, the minimum rework time is 5 minutes and the maximum is 30 minutes for this group. Major and Minor severity groups impacted negatively software maintenance capacity. One may observe that Major severity group presents high rework times in comparison to the other severity groups – Figure 2c. We investigate the sort of Code Debt and they are mostly related to Code Duplication problems; the most duplications are in a code snippet, the longer the rework time is to solve this problem. Ignoring the outliers present in the severity group – Figure 3. Major Code Debt present average rework times 10 and 20 minutes (discarding outliers), and minimum and maximum rework time 1 and 30 minutes, respectively, for most software. At last, the Minor severity group has on average 2 minute rework times, a minimum of 1 minute and a maximum of 20 minutes, and in the Hibernate ecosystem, the Code Debt belonging to this group of severity have the lowest maximum rework time. Major Code Debt severity have high rework time, therefore, they will be hardly paid, considering the high effort required to perform this action and the low profit obtained. Solving those problems results in improved maintainability, however, the priority is to solve problems that directly affect the software functionality. Managers and Technical Debt management team, aware of problems that affect the capacity of maintenance and the high rework time, could perform training and guide developers to avoid problems, such as code duplication. Finally, we emphasize that Code Debt that impairs software functionality directly – Blocker and Critical severity – should be the first to be paid, since they have low rework times (maximum of 30 minutes identified) and would result in a higher quality gain for the software. RQ2: Is the distribution of rework time for each severity group experiencing significant variation across the multiple versions of each software of the Hibernate ecosystem? Analyzing each version of software projects belonging to the Hibernate ecosystem, we delineate the rework time distribution of each severity group and, thus, we observe the variation of this distribution along the development time of each software. Fig. 1. Schematization of the process of cataloging Technical Debt (a) (b) (c) (d) Fig. 2. Distribution of the rework time of each severity group: Blocker (a), Critical (b), Major (c) and Minor (d), of the Code Debt present in the last version of the Hibernate ecosystem softwares. variations at initial versions or no variation in their distribution of rework time. RQ3: Analyzing the distribution of the total amount of Code Debt of each severity group, which groups are most representative in the Hibernate ecosystem? Fig. 3. Distribution of rework time of Code Debt with Major severity present in the latest version of the Hibernate ecosystem softwares, disregarding outliers. In general, we observed that most analyzed software presents a little variation on rework time distribution over time. In general, the variations happen mainly at initial versions, and throughout the development, it tends to stabilize and maintain a similar distribution. This behavior happens because the initial versions undergo many modifications, generating a variation to rework time distribution of Code Debt. Over time, the acquired maturity in the development of each software, a smaller amount of modifications is necessary what results in the stabilization of rework time distribution. Therefore, all severity groups, in general, exhibit the same behavior: To answer this question, we verified the total amount of Code Debt of each severity group for each Hibernate ecosystem software. Then, we compared this distribution among all software to this ecosystem – Figure 4. Analyzing the distribution of Code Debt of each severity group, we found that most of the Code Debt is related to problems that are harmful to software maintenance (Major and Minor severity), being the groups of the severity of greater representativeness in the Hibernate ecosystem. The positive aspect was the almost insignificant number of Code Debt of greater severity (Blocker and Critical), that directly impact the functionality of the software, only the Hibernate Shards and Hibernate Commons software present a significant amount of problems with such severities. This behavior is analogous in all versions of each of the analyzed software. RQ1 has pointed out that Major severity requires longer software versions (Hibernate Asciidoctor, Commons and Metamodelgen). Other software presents an evolution characteristic followed by decreasing points. This situation may indicate that problems with this severity are being controlled and solved. VI. Fig. 4. Distribution of the total amount of Code Debt distributed by severity groups. rework time to be solved in comparison to other severities. RQ3 pointed out the majority to Major – high number of occurrence (see Figure 4). This Code Debt appears in large numbers in the multiple versions. The large amount and high rework time indicate difficulty to manually pay Code Debt. Thus, the use of automated refactoring techniques are necessary to pay Major severity. The analysis of the effectiveness and possible techniques to be applied can be considered future work. RQ4: Observing the evolution of the total rework time of each severity group, is the Code Debt controlled in the Hibernate Ecosystem? The focus to this research question is the evolution of the total rework time required for the solution of all the Code Debt present in each analyzed software and, consequently, to verify if the Technical Debt related to each severity group is controlled, being paid or accumulated in this ecosystem. For that, considering each of the severity groups, the total rework time of each software version was obtained. This time was normalized between 0 and 1 to highlight not the amount but rather the evolution factor of the reworking time required to solve the problems of each severity group. The Figures 5 e 6 illustrate, respectively, the evolution of some software of the Hibernate ecosystem in perspective of the total rework time of Code Debt detrimental to maintenance (Major and Minor severity) and Code Debt that directly affect the operation of the software (Blocker and Critical severity). In general, observing the evolution of the Code Debt rework time of Major and Minor severity, the increase of rework time was observed during the software development. To more than 50% of the analyzed software, the rework time of both severities tends to increase. This leads one to believe that they are not controlled by the development team, being generated and not paid over the development period. The minority of the ecosystem software has presented decreasing rework time of one severity, they are Hibernate Commons, Shards and Tools. Regarding the rework time evolution of Blocker and Critical severity Code Debt, both harmful to the software operation, we observed a stabilized evolution to the minority of analyzed software, with low and constant rework times to all F INAL R EMARKS The Technical Debt accumulation results in increased cost of the maintenance phase and a decrease in software quality. Knowing the behavior of debt can support to create efficient strategies in activities of prevention or payment of Technical Debt. We describe a case study carried out with the Hibernate ecosystem, analyze the behavior of Code Debt with the focus on aspects of severity and rework time. Technical Debt Catalogs were built which were later analyzed. The results of the study revealed that most Code Debt is detrimental to code maintainability, Major and Minor severity, and in particular, those of Major severity are those that need more time to be solved. We also investigated the variation of the rework time distribution of the Code Debt of each severity group and found that when there is variation, in general, it occurs in the early versions of development. Technical Debt Catalogs are positive contributions of this study since they can be used in other works involving Technical Debt. As future work we also intend to use the catalogs as a database, to investigate the possible benefits that visualization techniques can provide to support the monitoring and management of Technical Debt, since this is a lacuna pointed out in the literature, requiring further investigation [3] [16]. ACKNOWLEDGEMENT We are grateful to [BLIND] Research Agency for financial support to development of this work. R EFERENCES [1] P. Kruchten, R. L. Nord, and I. Ozkaya, “Technical debt: From metaphor to theory and practice,” IEEE Software, vol. 29, no. 6, pp. 18–21, Nov 2012. [2] Y. Guo and C. Seaman, “Measuring and monitoring technical debt,” in Advances in Computers, vol. 82, 2011, pp. 25–46. [3] N. Rios, M. G. de Mendonça Neto, and R. O. Spı́nola, “A tertiary study on technical debt: Types, management strategies, research trends, and base information for practitioners,” Information and Software Technology, vol. 102, pp. 117 – 145, 2018. [4] C. Seaman, Y. Guo, C. Izurieta, Y. Cai, N. Zazworka, F. Shull, and A. Vetre, Using technical debt data in decision making: Potencial decision approaches. In: Proceedings of the Third International Workshop on Managing Technical Debt, Piscataway, NJ, USA: IEEE Press, p. 45-48(MTD ’12, V.1), 2015. [5] A. Nugroho, J. Visser, and T. Kuipers, “An empirical model of technical debt and interest,” in Proceedings of the 2Nd Workshop on Managing Technical Debt, ser. MTD ’11. New York, NY, USA: ACM, 2011, pp. 1–8. [Online]. Available: http://doi.acm.org/10.1145/1985362.1985364 [6] Z. Li, P. Avgeriou, and P. Liang, “A systematic mapping study on technical debt and its management,” Journal of Systems and Software, vol. 101, no. C, pp. 193–220, Mar. 2015. [7] W. Cunningham, “The wycash portfolio management system,” SIGPLAN OOPS Mess., vol. 4, no. 2, pp. 29–30, Dec. 1992. [Online]. Available: http://doi.acm.org/10.1145/157710.157715 [8] N. Ramasubbu and C. F. Kemerer, “Managing technical debt in enterprise software packages,” IEEE Transactions on Software Engineering, vol. 40, no. 8, pp. 758–772, Aug 2014. Fig. 5. Fig. 6. (a) (b) (c) (d) (e) (f) Rework time evolution to Major and Minor severity: HQL (a), Metamodelgen (b), OGM (c), Search (d), Tools (e) and Validator (f) (a) (b) (c) (d) (e) (f) Rework time evolution to Blocker and Critical severity: Commons (a), OGM (b), ORM (c), Shards (d), Tools (e) and Validator (f) [9] R. Alfayez, P. Behnamghader, K. Srisopha, and B. Boehm, “An exploratory study on the influence of developers in technical debt,” in 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), May 2018, pp. 1–10. [12] G. Digkas, M. Lungu, A. Chatzigeorgiou, and P. Avgeriou, “The evolution of technical debt in the apache ecosystem,” in Software Architecture, A. Lopes and R. de Lemos, Eds. Cham: Springer International Publishing, 2017, pp. 51–66. [10] M. Tufano, F. Palomba, G. Bavota, R. Oliveto, M. D. Penta, A. D. Lucia, and D. Poshyvanyk, “When and why your code starts to smell bad,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1, May 2015, pp. 403–414. [13] [11] J. S. d. Jesus and A. C. V. d. Melo, “Technical debt and the software project characteristics. a repository-based exploratory analysis,” in 2017 IEEE 19th Conference on Business Informatics (CBI), vol. 01, July 2017, pp. 444–453. G. Digkas, M. Lungu, P. Avgeriou, A. Chatzigeorgiou, and A. Ampatzoglou, “How do developers fix issues and pay back technical debt in the apache ecosystem?” in 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), March 2018, pp. 153–163. [14] L. B. Foganholi, R. E. Garcia, D. M. Eler, R. C. M. Correira, and C. O. Junior, Td-manager: a tool for managing technical debt through integrated catalog. Int’l Conf. Software Eng. Research and Pratice, p. 132-138, 2015. ——, “Supporting technical debt cataloging with td-tracker tool,” Advances in Software Engineering, vol. 2015, 2015. [16] N. S. Alves, T. S. Mendes, M. G. de Mendonça, R. O. Spı́nola, F. Shull, and C. Seaman, “Identification and management of technical debt: A systematic mapping study,” Information and Software Technology, vol. 70, pp. 100 – 121, 2016. [15]