On the techniques we create, the tools we build, and their misalignments: A study of KLEE

EF Rizzi, S Elbaum, MB Dwyer - … of the 38th International Conference on …, 2016 - dl.acm.org
EF Rizzi, S Elbaum, MB Dwyer
Proceedings of the 38th International Conference on Software Engineering, 2016dl.acm.org
Our community constantly pushes the state-of-the-art by introducing" new" techniques.
These techniques often build on top of, and are compared against, existing systems that
realize previously published techniques. The underlying assumption is that existing systems
correctly represent the techniques they implement. This paper examines that assumption
through a study of KLEE, a popular and well-cited tool in our community. We briefly describe
six improvements we made to KLEE, none of which can be considered" new" techniques …
Our community constantly pushes the state-of-the-art by introducing "new" techniques. These techniques often build on top of, and are compared against, existing systems that realize previously published techniques. The underlying assumption is that existing systems correctly represent the techniques they implement. This paper examines that assumption through a study of KLEE, a popular and well-cited tool in our community. We briefly describe six improvements we made to KLEE, none of which can be considered "new" techniques, that provide order-of-magnitude performance gains. Given these improvements, we then investigate how the results and conclusions of a sample of papers that cite KLEE are affected. Our findings indicate that the strong emphasis on introducing "new" techniques may lead to wasted effort, missed opportunities for progress, an accretion of artifact complexity, and questionable research conclusions (in our study, 27% of the papers that depend on KLEE can be questioned). We conclude by revisiting initiatives that may help to realign the incentives to better support the foundations on which we build.
ACM Digital Library