research-article

<u>T</u>i<u>d</u>d<u>l</u>e: a <u>t</u>race <u>d</u>escription <u>l</u>anguage for generating concurrent benchmarks to test dynamic analyses

Authors:

Caitlin Sadowski,

Jaeheon YiAuthors Info & Claims

WODA '09: Proceedings of the Seventh International Workshop on Dynamic Analysis

Pages 15 - 21

https://doi.org/10.1145/2134243.2134248

Published: 20 July 2009 Publication History

Abstract

Dynamic analysis is a promising technique for finding concurrency bugs in multithreaded programs. However, testing a dynamic analysis tool can be difficult. Researchers end up writing large amounts of small benchmark programs. Since the benchmarks themselves are concurrent programs, they may execute nondeterministically, complicating testing of the analysis tool.

We propose testing dynamic analyses by writing traces in a simple trace description language, Tiddle. Our implementation, written in Haskell, generates deterministic multithreaded Java programs for testing dynamic analyses. We report that it is substantially easier to write programs with incriminating bugs such as race conditions in Tiddle than the corresponding Java source code version, reducing the amount of source code to maintain and understand. Although our implementation is targeted towards Java, the ideas extend to any other languages which support mutable fields and multiple threads.

References

[1]

Hackage. http://hackage.haskell.org.

[2]

S. Boroday, A. Petrenko, J. Singh, and H. Hallal. Dynamic analysis of Java applications for multithreaded antipatterns. In International Workshop on Dynamic Analysis (WODA). ACM, 2005.

Digital Library

[3]

R. Brown, K. Driesen, D. Eng, L. Hendren, J. Jorgensen, C. Verbrugge, and Q. Wang. STEP: A framework for the efficient encoding of general trace data. In ACM SIGPLAN-SIGSOFT Workshop on Program Anaylsis for Software Tools and Engineering (PASTE), Nov. 2002.

Digital Library

[4]

R. Carver and K. Tai. Replay and testing for concurrent programs. IEEE Software, 8(2):66--74, 1991.

Digital Library

[5]

S. Chandra, B. Richards, and J. Larus. Teapot: a domain-specific language for writing cache coherence protocols. Software Engineering, IEEE Transactions on, 25(3):317--333, 1999.

Digital Library

[6]

F. Chang and J. Ren. Validating system properties exhibited in execution traces. In International Conference on Automated Software Engineering (ASE). ACM, 2007.

Digital Library

[7]

F. Chen and G. Roşu. Mop: an efficient and generic runtime verification framework. SIGPLAN Notices, 42(10):569--588, 2007.

Digital Library

[8]

J. Choi and H. Srinivasan. Deterministic replay of Java multithreaded applications. In SIGMETRICS Symposium on Parallel and Distributed Tools (SPDT). ACM, 1998.

Digital Library

[9]

M. d'Amorim and K. Havelund. Event-based runtime verification of java programs. In International Workshop on Dynamic Analysis (WODA), pages 1--7, New York, NY, USA, 2005. ACM.

Digital Library

[10]

L. Eeckhout, K. de Bosschere, and H. Neefs. Performance analysis through synthetic trace generation. In IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2000.

Digital Library

[11]

T. Elmas, S. Qadeer, and S. Tasiran. Goldilocks: a race and transaction-aware Java runtime. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 245--255, 2007.

Digital Library

[12]

Y. Eytani, R. Tzoref, and S. Ur. Experience with a Concurrency Bugs Benchmark. In IEEE International Conference on Software Testing Verification and Validation Workshop (ICSTW), 2008.

Digital Library

[13]

Y. Eytani and S. Ur. Compiling a benchmark of documented multi-threaded bugs. In Parallel and Distributed Processing Symposium, 2004.

[14]

A. Farzan and P. Madhusudan. Monitoring atomicity in concurrent programs. In Computer Aided Verification (CAV), 2008.

Digital Library

[15]

C. Flanagan and S. Freund. FastTrack: Efficient and precise dynamic race detection. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2009. To appear.

Digital Library

[16]

C. Flanagan and S. N. Freund. Atomizer: A dynamic atomicity checker for multithreaded programs. In ACM SIGPLAN - SIGACT Symposium on Principles of Programming Languages (POPL), pages 256--267, 2004.

Digital Library

[17]

C. Flanagan, S. N. Freund, and J. Yi. Velodrome: A sound and complete dynamic atomicity checker for multithreaded programs. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2008.

Digital Library

[18]

C. Hammer, J. Dolby, M. Vaziri, and F. Tip. Dynamic detection of atomic-set-serializability violations. In International Conference on Software Engineering. ACM, 2008.

Digital Library

[19]

A. Hamou-Lhadj and T. C. Lethbridge. An efficient algorithm for detecting patterns in traces of procedure calls. In International Workshop on Dynamic Analysis (WODA). ACM, 2003.

[20]

D. Harmanci, P. Felber, V. Gramoli, and C. Fetzer. TMunit: Testing Transactional Memories. 4th ACM SIGPLAN Workshop on Transactional Computing (TRANSACT), 2009.

[21]

P. Hudak, J. Hughes, S. Peyton Jones, and P. Wadler. A history of Haskell: being lazy with class. In HOPL III: Proceedings of the third ACM SIGPLAN conference on History of programming languages. ACM, 2007.

Digital Library

[22]

E. Jahier, M. Ducassé, and O. Ridoux. Specifying prolog trace models with a continuation semantics. In LOPSTR: Selected Papers form the 10th International Workshop on Logic Based Program Synthesis and Transformation, pages 165--182, London, UK, 2001. Springer-Verlag.

Digital Library

[23]

C.-W. Jeon, I.-G. Kim, and J.-Y. Choi. Automatic generation of the c# code for security protocols verified with casper/fdr. In AIN A '05: Proceedings of the 19th International Conference on Advanced Information Networking and Applications, pages 507--510, Washington, DC, USA, 2005. IEEE Computer Society.

Digital Library

[24]

S. Joshi and A. Orso. Scarpe: A technique and tool for selective capture and replay of program executions. In International Conference on Software Maintenance (ICSM), pages 234--243. IEEE, 2007.

[25]

J. Levine, T. Mason, and D. Brown. lex & yacc. O'Reilly, 2nd edition, 1992.

[26]

B. Long, D. Hoffman, and P. Strooper. Tool support for testing concurrent java components. IEEE Transactions on Software Engineering, 29(6):555--566, 2003.

Digital Library

[27]

S. Lu, J. Tucek, F. Qin, and Y. Zhou. Avio: Detecting atomicity violations via access-interleaving invariants. IEEE Micro, 27(1):26--35, 2007.

Digital Library

[28]

S. Marlow. Happy, a parser-generator for Haskell. http://www.haskell.org/happy.

[29]

S. Marlow. A lexical analyser generator for Haskell. http://www.haskell.org/alex.

[30]

R. O'Callahan and J.-D. Choi. Hybrid dynamic data race detection. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), 2003.

Digital Library

[31]

S. Peyton Jones. A pretty printer library in Haskell. Part of the GHC distribution at http://www.haskell.org/ghc.

[32]

E. Pozniansky and A. Schuster. MultiRace: efficient on-the-fly data race detection in multithreaded C++ programs. Concurrency and Computation: Practice & Experience, 19(3):327--340, 2007.

Digital Library

[33]

D. Pozza, R. Sisto, and L. Durante. Spi2java: Automatic cryptographic protocol java code generation from spi calculus. In AINA '04: Proceedings of the 18th International Conference on Advanced Information Networking and Applications, page 400, Washington, DC, USA, 2004. IEEE Computer Society.

Digital Library

[34]

W. Pugh and N. Ayewah. Unit testing concurrent software. In International Conference on Automated Software Engineering (ASE). ACM, 2007.

Digital Library

[35]

M. Ronsse and K. D. Bosschere. Recplay: a fully integrated practical record/replay system. ACM Transactions on Computer Systems, 17(2):133--152, 1999.

Digital Library

[36]

C. Sadowski, S. N. Freund, and C. Flanagan. Singletrack: A dynamic determinism checker for multithreaded programs. In European Symposium on Programming (ESOP).

Digital Library

[37]

C. Sadowski and J. Yi. Sidetrack: Generalizing dynamic atomicity analysis. In Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging (PADTAD), 2009. To appear.

Digital Library

[38]

S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. E. Anderson. Eraser: A dynamic data race detector for multi-threaded programs. ACM Transactions on Computer Systems, 15(4):391--411, 1997.

Digital Library

[39]

E. Schonberg. On-the-fly detection of access anomalies. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 285--297, 1989.

Digital Library

[40]

V. Schuppan, M. Baur, and A. Biere. Jvm independent replay in java. Electronic Notes in Theoretical Compututer Science, 113:85--104, 2005.

Digital Library

[41]

J. Steven, P. Chandra, B. Fleck, and A. Podgurski. jrapture: A capture/replay tool for observation-based testing. SIGSOFT Software Engineering Notes, 25(5):158--167, 2000.

Digital Library

[42]

L. Wang and S. D. Stoller. Run-time analysis for atomicity. volume 89(2) of Electronic Notes in Theoretical Computer Science. Elsevier, 2003.

[43]

L. Wang and S. D. Stoller. Accurate and efficient runtime detection of atomicity errors in concurrent programs. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), pages 137--146, 2006.

Digital Library

[44]

D. Wile. Supporting the DSL Spectrum. Journal of Computing and Information Technology, 9(4):263--288, 2001.

[45]

M. Xu, R. Bodík, and M. D. Hill. A serializability violation detector for shared-memory server programs. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 1--14, 2005.

Digital Library

[46]

Y. Yu, T. Rodeheffer, and W. Chen. Racetrack: efficient detection of data race conditions via adaptive tracking. In ACM Symposium on Operating Systems Principles (SOSP), 2005.

Digital Library

Cited By

Abbaspour Asadollah SSundmark DEldh SHansson HAfzal W(2017)10 Years of research on debugging concurrent and multicore softwareSoftware Quality Journal10.1007/s11219-015-9301-725:1(49-82)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s11219-015-9301-7
Melo SSouza SSilva RSouza PVos TEldh SPrasetya W(2015)Concurrent software testing in practice: a catalog of toolsProceedings of the 6th International Workshop on Automating Test Case Design, Selection and Evaluation10.1145/2804322.2804328(31-40)Online publication date: 30-Aug-2015
https://dl.acm.org/doi/10.1145/2804322.2804328
Yi JSadowski CFlanagan CGopalakrishnan GFarchi EMercer E(2009)SideTrackProceedings of the 7th Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging10.1145/1639622.1639630(1-10)Online publication date: 19-Jul-2009
https://dl.acm.org/doi/10.1145/1639622.1639630

Index Terms

Tiddle: a trace description language for generating concurrent benchmarks to test dynamic analyses

Recommendations

FastTrack: efficient and precise dynamic race detection
PLDI '09

\begin{abstract}

Multithreaded programs are notoriously prone to race conditions. Prior work on dynamic race detectors includes fast but imprecise race detectors that report false alarms, as well as slow but precise race detectors that never report ...
Multithreaded test synthesis for deadlock detection
OOPSLA '14

Designing and implementing thread-safe multithreaded libraries can be a daunting task as developers of these libraries need to ensure that their implementations are free from concurrency bugs, including deadlocks. The usual practice involves employing ...
Atomizer: A dynamic atomicity checker for multithreaded programs

Ensuring the correctness of multithreaded programs is difficult, due to the potential for unexpected interactions between concurrent threads. Much previous work has focused on detecting race conditions, but the absence of race conditions does not by ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WODA '09: Proceedings of the Seventh International Workshop on Dynamic Analysis

July 2009

52 pages

ISBN:9781605586564

DOI:10.1145/2134243

Program Chairs:
Ben Liblit
University of Wisconsin-Madison
,
Andy Podgurski
Case Western Reserve University

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '09

Sponsor:

SIGSOFT

ISSTA '09: International Symposium on Software Testing and Analysis

July 20, 2009

Illinois, Chicago

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
71
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Abbaspour Asadollah SSundmark DEldh SHansson HAfzal W(2017)10 Years of research on debugging concurrent and multicore softwareSoftware Quality Journal10.1007/s11219-015-9301-725:1(49-82)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s11219-015-9301-7
Melo SSouza SSilva RSouza PVos TEldh SPrasetya W(2015)Concurrent software testing in practice: a catalog of toolsProceedings of the 6th International Workshop on Automating Test Case Design, Selection and Evaluation10.1145/2804322.2804328(31-40)Online publication date: 30-Aug-2015
https://dl.acm.org/doi/10.1145/2804322.2804328
Yi JSadowski CFlanagan CGopalakrishnan GFarchi EMercer E(2009)SideTrackProceedings of the 7th Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging10.1145/1639622.1639630(1-10)Online publication date: 19-Jul-2009
https://dl.acm.org/doi/10.1145/1639622.1639630

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents