Learning a Metric for Code Readability

Reviewer: Ned Chapin

Buse et al. had 120 subjects-mostly undergraduates majoring in computer science-rate the readability (not complexity) of 100 snippets of Java source code from five major open-source programs. Then, they statistically analyzed the resulting 12,000 ratings: "This analysis seems to confirm the widely held belief that humans agree significantly on what readable code looks like, but not to an overwhelming extent." This paper, a continuation of the authors' ongoing research work on this project, reports their results, which are drawn mostly from their correlation analyses. As a part of their project, the authors also produced an unnamed software metric that is based on and uses counts of 25 specific features of Java source code. This software metric is the focus of the paper. The authors report that their metric is somewhat better than their human subjects at assessing readability. According to metric runs on more than two million lines of Java open-source code, including some longitudinal studies, their metric "dramatically" correlated with code changes and bug reports for 15 major open-source programs. Furthermore, the paper states that their metric can be obtained quickly, in only a few minutes, which is far faster than the human response time. The paper's major points are summarized in three figures. Figure 9 shows that only three Java source code features increase readability; the two most powerful features are average comment frequency and average blank line ratio (the most powerful). Figure 12 shows the metric's flagging of source code defect potential, for 13 open-source Java programs. Figure 15 illustrates the metric's relationship to source code program maturity, for 12 open-source Java programs. The authors report positive results for their metric, which reflects favorably on their work. Hopefully, they will continue this project and publish additional reports. That being said, the paper suffers from trying to cover too much in too few pages. For example, the authors do not report the usual basic statistics for their metric, such as standard deviations and levels of statistical significance; instead, they prefer to use qualitative terms such as "dramatically." Although the authors acknowledge that statistical correlations are a poor fit for their data, they still use and report them. While principal component analysis is briefly mentioned, any specifics seem to be withheld for a future paper. The authors mention relative predictive power, but fail to defend or adequately explain its use. A statistician should have advised the authors on how to strengthen their analysis work and present the results in a more sharply focused manner; perhaps it was just a problem of balancing the paper's length and its content. Online Computing Reviews Service