research-article

Testing an optimising compiler by generating random lambda terms

Authors:

Michał H. Pałka,

Alejandro Russo,

John HughesAuthors Info & Claims

AST '11: Proceedings of the 6th International Workshop on Automation of Software Test

Pages 91 - 97

https://doi.org/10.1145/1982595.1982615

Published: 23 May 2011 Publication History

Abstract

This paper considers random testing of a compiler, using randomly generated programs as inputs, and comparing their behaviour with and without optimisation. Since the generated programs must compile, then we need to take into account syntax, scope rules, and type checking during our random generation. Doing so, while attaining a good distribution of test data, proves surprisingly subtle; the main contribution of this paper is a workable solution to this problem. We used it to generate typed functions on lists, which we compiled using the Glasgow Haskell compiler, a mature production quality Haskell compiler. After around 20,000 tests we triggered an optimiser failure, and automatically simplified it to a program with just a few constructs.

References

[1]

L. Augustsson. Announcing Djinn, version 2004-12-11, a coding wizard. http://permalink.gmane.org/gmane.comp.lang.haskell.general/12747, 2005.

[2]

O. Bodini, D. Gardy, and B. Gittenberger. Lambda terms of bounded unary height. In Proc. of the 8th Workshop on Analytic Algorithmics and Combinatorics, 2011.

[3]

K. Claessen and J. Hughes. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the fifth ACM SIGPLAN International Conference on Functional Programming. ACM, 2000.

Digital Library

[4]

L. Damas and R. Milner. Principal type-schemes for functional programs. In Proceedings of the 9th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. ACM, 1982.

Digital Library

[5]

B. Daniel, D. Dig, K. Garcia, and D. Marinov. Automated testing of refactoring engines. In Proc. of the 6th meeting of the European Software Engineering Conference and the ACM SIGSOFT Symp. on the Foundations of Software Engineering. ACM, 2007.

Digital Library

[6]

N. A. Danielsson and P. Jansson. Chasing bottoms: A case study in program verification in the presence of partial and infinite values. In MPC, 2004.

[7]

D. Drienyovszky, D. Horpácsi, and S. Thompson. Quickchecking refactoring tools. In Proceedings of the 9th ACM SIGPLAN workshop on Erlang. ACM, 2010.

Digital Library

[8]

R. Dyckhoff. Contraction-free sequent calculi for intuitionistic logic. Journal of Symbolic Logic, 57(3), 1992.

[9]

C. Klein, M. Flatt, and R. B. Findler. Random testing for higher-order, stateful programs. In Proc. of the ACM International Conference on Object Oriented Programming Systems Languages and Applications. ACM, 2010.

Digital Library

[10]

C. Lindig. Random testing of C calling conventions. In Proceedings of the 6th International Symposium on Automated Analysis-Driven Debugging. ACM, 2005.

Digital Library

[11]

M. Moczurad, J. Tyszkiewicz, and M. Zaionc. Statistical properties of simple types. Mathematical. Structures in Computer Science, 10, October 2000.

Digital Library

[12]

S. L. Peyton Jones. Compiling Haskell by program transformation: a report from the trenches. In Proc. of European Symp. on Programming. Springer-Verlag, 1996.

Digital Library

[13]

B. C. Pierce. Types and programming languages. MIT Press, Cambridge, MA, USA, 2002.

Digital Library

[14]

A. Rodriguez Yakushev and J. Jeuring. Enumerating well-typed terms generically. In Approaches and Applications of Inductive Programming, volume 5812 of LNCS. Springer Berlin / Heidelberg, 2010.

[15]

The GHC Team. The Glasgow Haskell Compiler. Software release. http://haskell.org/ghc/.

[16]

D. Vytiniotis and A. J. Kennedy. Functional pearl: every bit counts. In Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming. ACM, 2010.

Digital Library

[17]

J. Wang. Generating random lambda calculus terms. Technical report, Boston University, 2005.

[18]

M. Zaionc. Mechanical procedure for proof construction via closed terms in typed λ calculus. J. Autom. Reason., 4:173--190, June 1988.

Digital Library

[19]

S. Zakhour, S. Hommel, J. Royal, I. Rabinovitch, T. Risser, and M. Hoeber. The Java Tutorial: A Short Course on the Basics, 4th Edition. Prentice Hall PTR, 2006.

Cited By

Cutler JDisselkoen CEline AHe SHeadley KHicks MHietala KIoannidis EKastner JMamat AMcAdams DMcCutchen MRungta NTorlak EWells A(2024)Cedar: A New Language for Expressive, Fast, Safe, and Analyzable AuthorizationProceedings of the ACM on Programming Languages10.1145/36498358:OOPSLA1(670-697)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649835
Frank JQuiring BLampropoulos L(2024)Generating Well-Typed Terms That Are Not “Useless”Proceedings of the ACM on Programming Languages10.1145/36329198:POPL(2318-2339)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632919
Goldstein HCutler JDickstein DPierce BHead ARoychoudhury APaiva AAbreu RStorey M(2024)Property-Based Testing in PracticeProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639581(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639581
Show More Cited By

Index Terms

Testing an optimising compiler by generating random lambda terms
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. General programming languages
      1. Language types
        Functional languages

Recommendations

Comparison of adaptive random testing and random testing under various testing and debugging scenarios

Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random ...
A Comparison of Structural Testing Strategies Based on Subdomain Testing and Random Testing
DEPCOS-RELCOMEX '09: Proceedings of the 2009 Fourth International Conference on Dependability of Computer Systems

Both partition testing and random testing methods are commonly followed practice towards selection of test cases. For partition testing, the program’s input domain is divided into subsets, called subdomains, and one or more representatives from each ...
Random testing of C calling conventions
AADEBUG'05: Proceedings of the sixth international symposium on Automated analysis-driven debugging

In a C compiler, function calls are difficult to implement correctly because they must respect a platform-specific calling convention. But they are governed by a simple invariant: parameters passed to a function must be received unaltered. A violation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AST '11: Proceedings of the 6th International Workshop on Automation of Software Test

May 2011

124 pages

ISBN:9781450305921

DOI:10.1145/1982595

Program Chairs:
Antonia Bertolino
ISTI-CNR, Italy
,
Howard Foster
City University London, UK
,
J. Jenny Li
Avaya Labs Research, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 May 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE11

Sponsor:

SIGSOFT

ICSE11: International Conference on Software Engineering

May 23 - 24, 2011

HI, Waikiki, Honolulu, USA

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

75
Total Citations
View Citations
519
Total Downloads

Downloads (Last 12 months)49
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cutler JDisselkoen CEline AHe SHeadley KHicks MHietala KIoannidis EKastner JMamat AMcAdams DMcCutchen MRungta NTorlak EWells A(2024)Cedar: A New Language for Expressive, Fast, Safe, and Analyzable AuthorizationProceedings of the ACM on Programming Languages10.1145/36498358:OOPSLA1(670-697)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649835
Frank JQuiring BLampropoulos L(2024)Generating Well-Typed Terms That Are Not “Useless”Proceedings of the ACM on Programming Languages10.1145/36329198:POPL(2318-2339)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632919
Goldstein HCutler JDickstein DPierce BHead ARoychoudhury APaiva AAbreu RStorey M(2024)Property-Based Testing in PracticeProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639581(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639581
Li LLiu YPostol DLampropoulos LVan Horn DHicks M(2023)A formal model of Checked C1Journal of Computer Security10.3233/JCS-23004031:5(581-614)Online publication date: 13-Oct-2023
https://doi.org/10.3233/JCS-230040
Krook RSmallbone NSvensson BClaessen K(2023)QuickerCheck: Implementing and Evaluating a Parallel Run-Time for QuickCheckProceedings of the 35th Symposium on Implementation and Application of Functional Languages10.1145/3652561.3652570(1-12)Online publication date: 29-Aug-2023
https://dl.acm.org/doi/10.1145/3652561.3652570
Zago NBraga AMello BFeitosa S(2023)Usando Esquema GraphQL para Geração de Consultas de Forma AleatóriaProceedings of the XXVII Brazilian Symposium on Programming Languages10.1145/3624309.3624321(17-24)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3624309.3624321
Elazar Mittelman SResnick APerez IGoodloe ALampropoulos LMcDonell TVazou N(2023)Don’t Go Down the Rabbit Hole: Reprioritizing Enumeration for Property-Based TestingProceedings of the 16th ACM SIGPLAN International Haskell Symposium10.1145/3609026.3609730(59-71)Online publication date: 30-Aug-2023
https://dl.acm.org/doi/10.1145/3609026.3609730
Shi JKeles AGoldstein HPierce BLampropoulos L(2023)Etna: An Evaluation Platform for Property-Based Testing (Experience Report)Proceedings of the ACM on Programming Languages10.1145/36078607:ICFP(878-894)Online publication date: 31-Aug-2023
https://dl.acm.org/doi/10.1145/3607860
Goldstein HFrohlich SWang MPierce B(2023)Reflecting on Random GenerationProceedings of the ACM on Programming Languages10.1145/36078427:ICFP(322-355)Online publication date: 31-Aug-2023
https://dl.acm.org/doi/10.1145/3607842
Prinz JLampropoulos L(2023)Merging Inductive RelationsProceedings of the ACM on Programming Languages10.1145/35912927:PLDI(1759-1778)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591292
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents