research-article

JaConTeBe: a benchmark suite of real-world Java concurrency bugs

Authors:

Jianjun ZhaoAuthors Info & Claims

ASE '15: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering

Pages 178 - 189

https://doi.org/10.1109/ASE.2015.87

Published: 09 November 2015 Publication History

Abstract

Researchers have proposed various approaches to detect concurrency bugs and improve multi-threaded programs, but performing evaluations of these approaches still remains a substantial challenge. We survey the existing evaluations and find out that they often use code or bugs not representative of real world. To improve representativeness, we have prepared JaConTeBe, a benchmark suite of 47 confirmed concurrency bugs from 8 popular open-source projects, supplemented with test cases for reproducing buggy behaviors. Running three approaches on JaConTeBe shows that our benchmark suite confirms some limitations of the three approaches. We submitted JaConTeBe to the SIR repository (a software-artifact repository for rigorous controlled experiments), and it was included as a part of SIR.

References

[1]

ASM homepage.{online}.available: http://asm.ow2.org.

[2]

FindBugs website.{online}.available: http://findbugs.sourceforge.net/bugDescriptions.html.

[3]

Mockito homepage.{online}.available: http://site.mockito.org.

[4]

S. Bensalem and K. Havelund. Dynamic deadlock analysis of multithreaded programs. In HVC, 2005.

[5]

S. Bindal, S. Bansal, and A. Lal. Variable and thread bounding for systematic testing of multithreaded programs. In ISSTA, 2013.

Digital Library

[6]

C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu. Fair and balanced?: Bias in bug-fix datasets. In ESEC/FSE, 2009.

Digital Library

[7]

S. Biswas, J. Huang, A. Sengupta, and M. D. Bond. DoubleChecker: Efficient sound and precise atomicity checking. In PLDI, 2014.

Digital Library

[8]

S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khan, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. L. Hosking, M. Jump, H. B. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovic, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In OOPSLA, 2006.

Digital Library

[9]

M. D. Bond, K. E. Coons, and K. S. McKinley. PACER: Proportional detection of data races. In PLDI, 2010.

Digital Library

[10]

J. S. Bradbury, I. Segall, E. Farchi, K. Jalbert, and D. Kelk. Using combinatorial benchmark construction to improve the assessment of concurrency bug detection tools. In PADTAD, 2012.

Digital Library

[11]

J. M. Bull, L. A. Smith, M. D. Westhead, D. S. Henty, and R. A. Davey. A benchmark suite for high performance Java. Concurrency - Practice and Experience, 12(6):375--388, 2000.

[12]

J. Burnim, T. Elmas, G. C. Necula, and K. Sen. NDSeq: Runtime checking for nondeterministic sequential specifications of parallel correctness. In PLDI, 2011.

Digital Library

[13]

Y. Cai and W. Chan. Magiclock: Scalable detection of potential deadlocks in large-scale multithreaded programs. IEEE Transactions on Software Engineering, 40(3):266--281, 2014.

Digital Library

[14]

Y. Cai, S. Wu, and W. K. Chan. ConLock: A constraint-based approach to dynamic checking on deadlocks in multithreaded programs. In ICSE, 2014.

Digital Library

[15]

J. Choi, K. Lee, A. Loginov, R. O'Callahan, V. Sarkar, and M. Sridharan. Efficient and precise datarace detection for multithreaded object-oriented programs. In PLDI, 2002.

Digital Library

[16]

M. d'Amorim, A. Sobeih, and D. Marinov. Optimized execution of deterministic blocks in Java PathFinder. In ICFEM, 2006.

Digital Library

[17]

D. Dimitrov, V. Raychev, M. T. Vechev, and E. Koskinen. Commutativity race detection. In PLDI, 2014.

Digital Library

[18]

H. Do, S. G. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering: An International Journal, 10(4):405--435, 2005.

Digital Library

[19]

M. B. Dwyer, S. Person, and S. Elbaum. Controlling factors in evaluating path-sensitive error detection techniques. In FSE, 2006.

Digital Library

[20]

T. Elmas, J. Burnim, G. Necula, and K. Sen. CONCURRIT: A domain specific language for reproducing concurrency bugs. In PLDI, 2013.

Digital Library

[21]

M. Eslamimehr and J. Palsberg. Sherlock: Scalable deadlock detection for concurrent programs. In FSE, 2014.

Digital Library

[22]

Y. Eytani, K. Havelund, S. D. Stoller, and S. Ur. Towards a framework and a benchmark for testing tools for multi-threaded programs. Concurrency and Computation: Practice and Experience, 19(3):267--279, 2007.

Digital Library

[23]

Y. Eytani, R. Tzoref, and S. Ur. Experience with a concurrency bugs benchmark. In TESTBENCH, 2008.

Digital Library

[24]

Y. Eytani and S. Ur. Compiling a benchmark of documented multithreaded bugs. In PADTAD, 2004.

[25]

E. Farchi, Y. Nir, and S. Ur. Concurrent bug patterns and how to test them. In IPDPS, 2003.

Digital Library

[26]

A. Farzan, P. Madhusudan, N. Razavi, and F. Sorrentino. Predicting null-pointer dereferences in concurrent programs. In FSE, 2012.

Digital Library

[27]

C. Flanagan and S. N. Freund. FastTrack: Efficient and precise dynamic race detection. In PLDI, 2009.

Digital Library

[28]

C. Flanagan and S. N. Freund. Adversarial memory for detecting destructive races. In PLDI, 2010.

Digital Library

[29]

C. Flanagan, S. N. Freund, and J. Yi. Velodrome: A sound and complete dynamic atomicity checker for multithreaded programs. In PLDI, 2008.

Digital Library

[30]

M. K. Ganai. Scalable and precise symbolic analysis for atomicity violations. In ASE, 2011.

Digital Library

[31]

M. K. Ganai. Efficient data race prediction with incremental reasoning on time-stamped lock history. In ASE, 2013.

Digital Library

[32]

M. Gligoric, L. Zhang, C. Pereira, and G. Pokam. Selective mutation testing for concurrent code. In ISSTA, 2013.

Digital Library

[33]

G. Golan-Gueta, G. Ramalingam, M. Sagiv, and E. Yahav. Concurrent libraries with foresight. In PLDI, 2013.

Digital Library

[34]

C. Hammer, J. Dolby, M. Vaziri, and F. Tip. Dynamic detection of atomic-set-serializability violations. In ICSE, 2008.

Digital Library

[35]

S. Hong, J. Ahn, S. Park, M. Kim, and M. J. Harrold. Testing concurrent programs to achieve high synchronization coverage. In ISSTA, 2012.

Digital Library

[36]

S. Hong and M. Kim. A survey of race bug detection techniques for multithreaded programmes. Software Testing, Verification and Reliability, 25(3):191--217, 2015.

Digital Library

[37]

D. Hovemeyer and W. Pugh. Finding bugs is easy. In OOPSLA, 2004.

Digital Library

[38]

J. Huang, P. Liu, and C. Zhang. LEAP: Lightweight deterministic multiprocessor replay of concurrent Java programs. In FSE, 2010.

Digital Library

[39]

J. Huang, P. O. Meredith, and G. Rosu. Maximal sound predictive race detection with control flow abstraction. In PLDI, 2014.

Digital Library

[40]

J. Huang and C. Zhang. Persuasive prediction of concurrency access anomalies. In ISSTA, 2011.

Digital Library

[41]

J. Huang, C. Zhang, and J. Dolby. CLAP: Recording local executions to reproduce concurrency failures. In PLDI, 2013.

Digital Library

[42]

V. Jagannath, M. Gligoric, D. Jin, Q. Luo, and G. Rosu. Improved multithreaded unit testing. In ESEC/FSE, 2011.

Digital Library

[43]

V. Jagannath, Q. Luo, and D. Marinov. Change-aware preemption prioritization. In ISSTA, 2011.

Digital Library

[44]

P. Joshi, M. Naik, C.-S. Park, and K. Sen. CalFuzzer: An extensible active testing framework for concurrent programs. In CAV, 2009.

Digital Library

[45]

P. Joshi, M. Naik, K. Sen, and D. Gay. An effective dynamic analysis for detecting generalized deadlocks. In FSE, 2010.

Digital Library

[46]

P. Joshi, C.-S. Park, K. Sen, and M. Naik. A randomized dynamic program analysis technique for detecting real deadlocks. In PLDI, 2009.

Digital Library

[47]

R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In FSE, 2014.

Digital Library

[48]

Z. Lai, S. Cheung, and W. K. Chan. Detecting atomic-set serializability violations in multithreaded programs through active randomized testing. In ICSE, 2010.

Digital Library

[49]

Z. Letko, T. Vojnar, and B. Krena. AtomRace: Data race and atomicity violation detector and healer. In PADTAD, 2008.

Digital Library

[50]

S. Li, Y. D. Liu, and G. Tan. JATO: Native code atomicity for Java. In APLAS, 2012.

[51]

T. Lindholm and F. Yellin. Java virtual machine specification. Addison-Wesley Longman Publishing Co., Inc., 1999.

Digital Library

[52]

P. Liu, J. Dolby, and C. Zhang. Finding incorrect compositions of atomicity. In ESEC/FSE, 2013.

Digital Library

[53]

S. Lu, Z. Li, F. Qin, L. Tan, and P. Zhou. BugBench: Benchmarks for evaluating bug detection tools. In Workshop on the Evaluation of Software Defect Detection Tools, 2005.

[54]

S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes: A comprehensive study on real world concurrency bug characteristics. In ASPLOS, 2008.

Digital Library

[55]

B. Lucia, B. P. Wood, and L. Ceze. Isolating and understanding concurrency errors using reconstructed execution fragments. In PLDI, 2011.

Digital Library

[56]

J. Manson, W. Pugh, and S. V. Adve. The Java memory model. In POPL, 2005.

Digital Library

[57]

D. Marino, C. Hammer, J. Dolby, M. Vaziri, F. Tip, and J. Vitek. Detecting deadlock in programs with data-centric synchronization. In ICSE, 2013.

Digital Library

[58]

M. Naik, A. Aiken, and J. Whaley. Effective static race detection for Java. In PLDI, 2006.

Digital Library

[59]

M. Naik, C.-S. Park, K. Sen, and D. Gay. Effective static deadlock detection. In ICSE, 2009.

Digital Library

[60]

A. Nistor, Q. Luo, M. Pradel, T. R. Gross, and D. Marinov. Ballerina: Automatic generation and clustering of efficient random unit tests for multithreaded code. In ICSE, 2012.

Digital Library

[61]

C.-S. Park and K. Sen. Randomized active atomicity violation detection in concurrent programs. In FSE, 2008.

Digital Library

[62]

S. Park. Debugging non-deadlock concurrency bugs. In ISSTA, 2013.

Digital Library

[63]

S. Park, R. W. Vuduc, and M. J. Harrold. Falcon: Fault localization in concurrent programs. In ICSE, 2010.

Digital Library

[64]

C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In ISSTA, 2011.

Digital Library

[65]

C. S. Păsăreanu, M. B. Dwyer, and W. Visser. Finding feasible abstract counter-examples. International Journal on Software Tools for Technology Transfer, 5(1):34--48, 2003.

Digital Library

[66]

W. Pugh and T. Lindholm. JSR-133: Java memory model and thread specification. Java Community Process, 2004.

[67]

C. Radoi and D. Dig. Practical static race detection for Java parallel loops. In ISSTA, 2013.

Digital Library

[68]

R. Raman, J. Zhao, V. Sarkar, M. T. Vechev, and E. Yahav. Scalable and precise dynamic datarace detection for structured parallelism. In PLDI, 2012.

Digital Library

[69]

J. Rößler, A. Zeller, G. Fraser, C. Zamfir, and G. Candea. Reconstructing core dumps. In ICST, 2013.

Digital Library

[70]

M. Samak and M. K. Ramanathan. Omen+: A precise dynamic deadlock detector for multithreaded Java libraries. In FSE, 2014.

Digital Library

[71]

M. Samak and M. K. Ramanathan. Trace driven dynamic deadlock detection and reproduction. In PPOPP, 2014.

Digital Library

[72]

K. Sen. Effective random testing of concurrent programs. In ASE, 2007.

Digital Library

[73]

K. Sen. Race directed random testing of concurrent programs. In PLDI, 2008.

Digital Library

[74]

O. Shacham, E. Yahav, G. Golan-Gueta, A. Aiken, N. G. Bronson, M. Sagiv, and M. T. Vechev. Verifying atomicity via data independence. In ISSTA, 2014.

Digital Library

[75]

N. Shafiei and F. van Breugel. Automatic handling of native methods in Java PathFinder. In SPIN, 2014.

Digital Library

[76]

S. E. Sim, S. M. Easterbrook, and R. C. Holt. Using benchmarking to advance research: A challenge to software engineering. In ICSE, 2003.

Digital Library

[77]

L. A. Smith, J. M. Bull, and J. Obdrizalek. A parallel Java grande benchmark suite. In Supercomputing, 2001.

Digital Library

[78]

F. Sorrentino, A. Farzan, and P. Madhusudan. PENELOPE: Weaving threads to expose atomicity violations. In FSE, 2010.

Digital Library

[79]

W. Visser, K. Havelund, G. Brat, S. Park, and F. Lerda. Model checking programs. Automated Software Engineering, 10(2):203--232, 2003.

Digital Library

[80]

J. W. Voung, R. Jhala, and S. Lerner. RELAY: Static race detection on millions of lines of code. In ESEC/FSE, 2007.

Digital Library

[81]

J. Yi, T. Disney, S. N. Freund, and C. Flanagan. Cooperative types for controlling thread interference in Java. In ISSTA, 2012.

Digital Library

[82]

T. Yu, W. Srisa-an, and G. Rothermel. SimRT: An automated framework to support regression testing for data races. In ICSE, 2014.

Digital Library

[83]

C. Zamfir and G. Candea. Execution synthesis: A technique for automated software debugging. In Eurosys, 2010.

Digital Library

[84]

K. Zhai, B. Xu, W. K. Chan, and T. H. Tse. CARISMA: A context-sensitive approach to race-condition sample-instance selection for multithreaded applications. In ISSTA, 2012.

Digital Library

[85]

L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, and H. Mei. Is operator-based mutant selection superior to random mutant selection? In ICSE, 2010.

Digital Library

Cited By

Silva ASaavedra NMonperrus MSpinellis DConstantinou EBacchelli A(2024)GitBug-Java: A Reproducible Java Benchmark of Recent BugsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644884(118-122)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644884
Misu MAchar RLopes C(2024)SourcererJBF: A Java Build Framework For Large-Scale CompilationACM Transactions on Software Engineering and Methodology10.1145/363571033:3(1-35)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3635710
Liu TJi WDong XYao WWang YLiu HPeng HWang YRoychoudhury APaiva AAbreu RStorey M(2024)JLeaks: A Featured Resource Leak Repository Collected From Hundreds of Open-Source Java ProjectsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639162(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639162
Show More Cited By

JaConTeBe: a benchmark suite of real-world Java concurrency bugs
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

DataRaceBench: a benchmark suite for systematic evaluation of data race detection tools
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Data races in multi-threaded parallel applications are notoriously damaging while extremely difficult to detect. Many tools have been developed to help programmers find data races. However, there is no dedicated OpenMP benchmark suite to systematically ...
Savina - An Actor Benchmark Suite: Enabling Empirical Evaluation of Actor Libraries
AGERE! '14: Proceedings of the 4th International Workshop on Programming based on Actors Agents & Decentralized Control

This paper introduces the Savina benchmark suite for actor-oriented programs. Our goal is to provide a standard benchmark suite that enables researchers and application developers to compare different actor implementations and identify those that ...
Bugs4Q: a benchmark of real bugs for quantum programs
ASE '21: Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering

Realistic benchmarks of reproducible bugs and fixes are vital to good experimental evaluation of debugging and testing approaches. However, there is no suitable benchmark suite that can systematically evaluate the debugging and testing methods of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '15: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering

November 2015

935 pages

ISBN:9781509000241

General Chair:
Myra Cohen
University of Nebraska-Lincoln
,
Program Chairs:
Lars Grunske
University of Stuttgart, Germany
,
Michael Whalen
University of Minnesota

Sponsors

In-Cooperation

IEEE CS

Publisher

IEEE Press

Publication History

Published: 09 November 2015

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASE '15

Sponsor:

ASE '15: ACM/IEEE International Conference on Automated Software Engineering

November 9 - 15, 2015

Nebraska, Lincoln

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
74
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Silva ASaavedra NMonperrus MSpinellis DConstantinou EBacchelli A(2024)GitBug-Java: A Reproducible Java Benchmark of Recent BugsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644884(118-122)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644884
Misu MAchar RLopes C(2024)SourcererJBF: A Java Build Framework For Large-Scale CompilationACM Transactions on Software Engineering and Methodology10.1145/363571033:3(1-35)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3635710
Liu TJi WDong XYao WWang YLiu HPeng HWang YRoychoudhury APaiva AAbreu RStorey M(2024)JLeaks: A Featured Resource Leak Repository Collected From Hundreds of Open-Source Java ProjectsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639162(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639162
Wang MSrikant SSamak MO’Reilly UFerrara PHadarean L(2023)RaceInjector: Injecting Races to Evaluate and Learn Dynamic Race Detection AlgorithmsProceedings of the 12th ACM SIGPLAN International Workshop on the State Of the Art in Program Analysis10.1145/3589250.3596142(63-70)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3589250.3596142
Liang JYuan MDing ZMa SHan XZhang C(2023)RaceBench: A Triggerable and Observable Concurrency Bug BenchmarkProceedings of the 2023 ACM Asia Conference on Computer and Communications Security10.1145/3579856.3595787(415-428)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3579856.3595787
Costea ATiwari AChianasta SR KRoychoudhury ASergey I(2023)Hippodrome: Data Race Repair Using Static Analysis SummariesACM Transactions on Software Engineering and Methodology10.1145/354694232:2(1-33)Online publication date: 31-Mar-2023
https://dl.acm.org/doi/10.1145/3546942
Rahman SLi CShi AGrundy J(2023)TSVD4J: Thread-Safety Violation Detection for JavaProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00029(78-82)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00029
Hirsch THofer B(2022)A systematic literature review on benchmarks for evaluating debugging approachesJournal of Systems and Software10.1016/j.jss.2022.111423192:COnline publication date: 25-Aug-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111423
Salehnamadi NAlshayban AAhmed IMalek Sde Lara EMohomed INieh JBelding E(2020)A benchmark for event-race analysis in android appsProceedings of the 18th International Conference on Mobile Systems, Applications, and Services10.1145/3386901.3396602(466-467)Online publication date: 15-Jun-2020
https://dl.acm.org/doi/10.1145/3386901.3396602
Tu TLiu XSong LZhang YBahar IHerlihy MWitchel ELebeck A(2019)Understanding Real-World Concurrency Bugs in GoProceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3297858.3304069(865-878)Online publication date: 4-Apr-2019
https://dl.acm.org/doi/10.1145/3297858.3304069
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents