research-article

Integrated language definition testing: enabling test-driven language development

Authors:

Lennart C.L. Kats,

Eelco VisserAuthors Info & Claims

ACM SIGPLAN Notices, Volume 46, Issue 10

Pages 139 - 154

https://doi.org/10.1145/2076021.2048080

Published: 22 October 2011 Publication History

Abstract

The reliability of compilers, interpreters, and development environments for programming languages is essential for effective software development and maintenance. They are often tested only as an afterthought. Languages with a smaller scope, such as domain-specific languages, often remain untested. General-purpose testing techniques and test case generation methods fall short in providing a low-threshold solution for test-driven language development. In this paper we introduce the notion of a language-parametric testing language (LPTL) that provides a reusable, generic basis for declaratively specifying language definition tests. We integrate the syntax, semantics, and editor services of a language under test into the LPTL for writing test inputs. This paper describes the design of an LPTL and the tool support provided for it, shows use cases using examples, and describes our implementation in the form of the Spoofax testing language.

References

[1]

K. Beck. Test-driven development: by example. Addison-Wesley Professional, 2003.

Digital Library

[2]

B. Beizer. Software testing techniques. Dreamtech Press, 2002.

[3]

A. Boujarwah and K. Saleh. Compiler test case generation methods: a survey and assessment. Information and Software Technology, 39(9):617--625, 1997.

[4]

J. Bovet and T. Parr. ANTLRWorks: an ANTLR grammar development environment. Software: Practice and Experience, 38(12):1305--1332, 2008.

Digital Library

[5]

M. Bravenboer, K. T. Kalleberg, R. Vermaas, and E. Visser. Stratego/XT 0.17. A language and toolset for program transformation. Science of Computer Programming, 72(1-2):52-- 70, 2008.

Digital Library

[6]

M. Bravenboer and E. Visser. Concrete syntax for objects: domain-specific language embedding and assimilation without restrictions. In J. M. Vlissides and D. C. Schmidt, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2004, pages 365--383. ACM, 2004.

Digital Library

[7]

P. Charles, R. M. Fuhrer, S. M. Sutton, Jr., E. Duesterwald, and J. Vinju. Accelerating the creation of customized, language-specific IDEs in Eclipse. In Proceeding of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications, OOPSLA '09, pages 191--206. ACM, 2009.

Digital Library

[8]

B. Daniel, D. Dig, K. Garcia, and D. Marinov. Automated testing of refactoring engines. In I. Crnkovic and A. Bertolino, editors, 6th joint meeting of the European Software Engineering Conference and the Int. Symposium on Foundations of Software Engineering, pages 185--194. ACM, 2007.

Digital Library

[9]

M. de Jonge, E. Nilsson-Nyman, L. C. L. Kats, and E. Visser. Natural and flexible error recovery for generated parsers. In M. van den Brand, D. Gasevic, and J. Gray, editors, Software Language Engineering, SLE 2009, volume 5969 of LNCS, pages 204--223. Springer, 2009.

Digital Library

[10]

P. Degano and C. Priami. Comparison of syntactic error handling in LR parsers. Software -- Practice and Experience, 25(6):657--679, 1995.

Digital Library

[11]

S. Efftinge and M. Völter. oAW xText - a framework for textual DSLs. In Modeling Symposium, Eclipse Summit, 2006.

[12]

M. Fowler. Refactoring: Improving the design of existing code. In D. Wells and L. A. Williams, editors, Extreme Programming and Agile Methods - XP/Agile Universe 2002, Second XP Universe and First Agile Universe Conference, volume 2418 of LNCS, page 256. Springer, 2002.

Digital Library

[13]

M. Fowler. Language workbenches: The killer-app for domain specific languages? http://martinfowler.com/articles/languageWorkbench.html, 2005.

[14]

J. D. Frens and A. Meneely. Fifteen compilers in fifteen days. In D. Baldwin, P. T. Tymann, S. M. Haller, and I. Russell, editors, 39th Technical Symposium on Computer Science Education, 2006, pages 92--96. ACM, 2006.

Digital Library

[15]

E. Gamma and K. Beck. Test infected: Programmers love writing tests. Java Report, 3(7):37--50, 1998.

[16]

J. B. Goodenough. The Ada compiler validation capability. In Proceedings of the ACM-SIGPLAN symposium on Ada programming language, SIGPLAN '80, pages 1--8. ACM, 1980.

Digital Library

[17]

gUnit - grammar unit testing. http://www.antlr.org/wiki/display/ANTLR3/gUnit+-+Grammar+ Unit+Testing.

[18]

R. Gómez, J. C. Augusto, and A. Galton. Testing an event specification language. In Software Engineering & Knowledge Engineering, SEKE 2001, pages 341--345, 2001.

[19]

P. Hamill. Unit Test Frameworks, chapter. Chapter 3: The xUnit Family of Unit Test Frameworks. O'Reilly, 2004.

[20]

Z. Hemel and E. Visser. Declaratively programming the mobile web with mobl. In Proceedings of the 26th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2011, Portland, Oregon, USA, 2011. ACM.

Digital Library

[21]

P. Hudak. Modular domain specific languages and tools. In Software Reuse, ICSR '98. Computer Society, jun 1998.

Digital Library

[22]

Jacks (Jacks is an Automated Compiler Killing Suite). http://sources.redhat.com/mauve/jacks. html.

[23]

L. C. L. Kats, M. de Jonge, E. Nilsson-Nyman, and E. Visser. Providing rapid feedback in generated modular language environments: adding error recovery to scannerless generalized- LR parsing. In S. Arora and G. T. Leavens, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2009, pages 445--464. ACM, 2009.

Digital Library

[24]

L. C. L. Kats, K. T. Kalleberg, and E. Visser. Interactive disambiguation of meta programs with concrete object syntax. In M. van den Brand, B. Malloy, and S. Staab, editors, Software Language Engineering, SLE 2010, LNCS. Springer, 2011.

Digital Library

[25]

L. C. L. Kats and E. Visser. The Spoofax language workbench: rules for declarative specification of languages and IDEs. In W. R. Cook, S. Clarke, and M. C. Rinard, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2010, pages 444--463. ACM, 2010.

Digital Library

[26]

L. C. L. Kats, E. Visser, and G. Wachsmuth. Pure and declarative syntax definition: paradise lost and regained. In W. R. Cook, S. Clarke, and M. C. Rinard, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2010, pages 918--932. ACM, 2010.

Digital Library

[27]

A. Kossatchev and M. Posypkin. Survey of compiler testing methods. Programming and Computer Software, 31(1):10-- 19, 2005.

Digital Library

[28]

H. Krahn, B. Rumpe, and S. Völkel. Monticore: Modular development of textual domain specific languages. In R. F. Paige and B. Meyer, editors, Objects, Components, Models and Patterns, TOOLS EUROPE 2008, volume 11 of Lecture Notes in Business Information Processing, pages 297--315. Springer, 2008.

[29]

R. Lämmel. Grammar testing. In H. Hußmann, editor, Fundamental Approaches to Software Engineering, FASE 2001, volume 2029 of LNCS, pages 201--216. Springer, 2001.

Digital Library

[30]

B. A. Malloy, J. F. Power, and J. T. Waldron. Applying software engineering techniques to parser design: the development of a C# parser. In SAICSIT '02: 2002 research conference of the South African institute of computer scientists and information technologists on Enablement through technology. South African Institute for Computer Scientists and Information Technologists, 2002.

Digital Library

[31]

G. Myers. The art of software testing, 2nd edition. WileyIndia, 2008.

Digital Library

[32]

R. W. Selby, editor. Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research. Wiley-Computer Society Press, 2007.

Digital Library

[33]

M. Strembeck and U. Zdun. An approach for the systematic development of domain-specific languages. Software: Practice and Experience, 39(15):1253--1292, 2009.

Digital Library

[34]

L. Tratt. Domain specific language implementation via compile-time meta-programming. Trans. Program. Lang. Syst., 30(6), 2008.

Digital Library

[35]

E. Visser. Syntax Definition for Language Prototyping. PhD thesis, University of Amsterdam, September 1997.

[36]

E. Visser. WebDSL: A case study in domain-specific language engineering. In R. Lämmel, J. Visser, and J. Saraiva, editors, Generative and Transformational Techniques in Software Engineering II, Int. Summer School, GTTSE 2007, volume 5235 of LNCS, pages 291--373. Springer, 2007.

Digital Library

[37]

M. Voelter and K. Solomatov. Language modularization and composition with projectional language workbenches illustrated with MPS. In M. van den Brand, B. Malloy, and S. Staab, editors, Software Language Engineering, SLE 2010, LNCS. Springer, 2010.

[38]

B. Wichmann and Z. Ciechanowicz. Pascal compiler validation. John Wiley & Sons, Inc. New York, NY, USA, 1983.

Digital Library

[39]

H. Wu, J. Gray, and M. Mernik. Grammar-driven generation of domain-specific language debuggers. Software: Practice and Experience, 38(10):1073--1103, 2008.

Digital Library

[40]

X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in C compilers. In Conference on Programming Language Design and Implementation, PLDI 2011. Press, June 2011.

Digital Library

Cited By

Voelter MKolb BBirken KTomassetti FAlff PWiart LWortmann ANordmann A(2019)Using language workbenches and domain-specific languages for safety-critical software developmentSoftware and Systems Modeling (SoSyM)10.1007/s10270-018-0679-018:4(2507-2530)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10270-018-0679-0
Maazouz HWachsmuth GSevenich MChiadmi DHong SChafi H(2019)A DSL-Based Framework for Performance AssessmentInnovation in Information Systems and Technologies to Support Learning Research10.1007/978-3-030-36778-7_28(260-270)Online publication date: 1-Dec-2019
https://doi.org/10.1007/978-3-030-36778-7_28
Ratiu DVoelter MPavletic D(2018)Automated testing of DSL implementations--experiences from building mbeddrSoftware Quality Journal10.1007/s11219-017-9390-626:4(1483-1518)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1007/s11219-017-9390-6
Show More Cited By

Index Terms

Integrated language definition testing: enabling test-driven language development
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools

Recommendations

The spoofax language workbench: rules for declarative specification of languages and IDEs
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applications

Spoofax is a language workbench for efficient, agile development of textual domain-specific languages with state-of-the-art IDE support. Spoofax integrates language processing techniques for parser generation, meta-programming, and IDE development into ...
Integrated language definition testing: enabling test-driven language development
OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications

The reliability of compilers, interpreters, and development environments for programming languages is essential for effective software development and maintenance. They are often tested only as an afterthought. Languages with a smaller scope, such as ...
The spoofax language workbench: rules for declarative specification of languages and IDEs
OOPSLA '10

Spoofax is a language workbench for efficient, agile development of textual domain-specific languages with state-of-the-art IDE support. Spoofax integrates language processing techniques for parser generation, meta-programming, and IDE development into ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 46, Issue 10

OOPSLA '11

October 2011

1063 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/2076021

Issue’s Table of Contents

OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications
October 2011
1104 pages
ISBN:9781450309400
DOI:10.1145/2048066
General Chair:
Cristina Videira Lopes
University of California, Irvine, USA
,
Program Chair:
Kathleen Fisher
Tufts University, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2011

Published in SIGPLAN Volume 46, Issue 10

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
605
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)2

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Voelter MKolb BBirken KTomassetti FAlff PWiart LWortmann ANordmann A(2019)Using language workbenches and domain-specific languages for safety-critical software developmentSoftware and Systems Modeling (SoSyM)10.1007/s10270-018-0679-018:4(2507-2530)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10270-018-0679-0
Maazouz HWachsmuth GSevenich MChiadmi DHong SChafi H(2019)A DSL-Based Framework for Performance AssessmentInnovation in Information Systems and Technologies to Support Learning Research10.1007/978-3-030-36778-7_28(260-270)Online publication date: 1-Dec-2019
https://doi.org/10.1007/978-3-030-36778-7_28
Ratiu DVoelter MPavletic D(2018)Automated testing of DSL implementations--experiences from building mbeddrSoftware Quality Journal10.1007/s11219-017-9390-626:4(1483-1518)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1007/s11219-017-9390-6
Ratiu DVoelter MBudnik CFraser GLonetti F(2016)Automated testing of DSL implementationsProceedings of the 11th International Workshop on Automation of Software Test10.1145/2896921.2896922(15-21)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2896921.2896922
Härtel JHärtel LLämmel R(2014)Test-Data Generation for XtextSoftware Language Engineering10.1007/978-3-319-11245-9_19(342-351)Online publication date: 2014
https://doi.org/10.1007/978-3-319-11245-9_19
Cimini M(2023)Testing Languages with a Languages-as-Databases ApproachTests and Proofs10.1007/978-3-031-38828-6_7(108-126)Online publication date: 18-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-38828-6_7
Boukham HWachsmuth GDwars MChiadmi DFischer BBurgueño LCazzola W(2022)A Multi-target, Multi-paradigm DSL Compiler for Algorithmic Graph ProcessingProceedings of the 15th ACM SIGPLAN International Conference on Software Language Engineering10.1145/3567512.3567513(2-15)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3567512.3567513
de Souza Amorim LVisser E(2020)Multi-purpose Syntax Definition with SDF3Software Engineering and Formal Methods10.1007/978-3-030-58768-0_1(1-23)Online publication date: 8-Sep-2020
https://doi.org/10.1007/978-3-030-58768-0_1
Sobernig SSobernig S(2020)DSL as Variable SoftwareVariable Domain-specific Software Languages with DjDSL10.1007/978-3-030-42152-6_1(1-31)Online publication date: 10-Jul-2020
https://doi.org/10.1007/978-3-030-42152-6_1
Rein PHirschfeld RTaeumel MVisser EMurphy-Hill ELopes C(2016)Gramada: immediacy in programming language developmentProceedings of the 2016 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/2986012.2986022(165-179)Online publication date: 20-Oct-2016
https://dl.acm.org/doi/10.1145/2986012.2986022
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents