research-article

Non-recursive make considered harmful: build systems at scale

Authors:

Simon Peyton Jones,

Simon MarlowAuthors Info & Claims

Haskell 2016: Proceedings of the 9th International Symposium on Haskell

Pages 170 - 181

https://doi.org/10.1145/2976002.2976011

Published: 08 September 2016 Publication History

Abstract

Most build systems start small and simple, but over time grow into hairy monsters that few dare to touch. As we demonstrate in this paper, there are a few issues that cause build systems major scalability challenges, and many pervasively used build systems (e.g. Make) do not scale well.

This paper presents a solution to the challenges we identify. We use functional programming to design abstractions for build systems, and implement them on top of the Shake library, which allows us to describe build rules and dependencies. To substantiate our claims, we engineer a new build system for the Glasgow Haskell Compiler. The result is more scalable, faster, and spectacularly more maintainable than its Make-based predecessor.

References

[1]

Sebastian Erdweg, Moritz Lichter, and Manuel Weiel. A sound and optimal incremental build system with dynamic dependencies. In Proc. of the ACM SIGPLAN Int’l Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 89–106. ACM, 2015.

Digital Library

[2]

Facebook. Buck. https://buckbuild.com/, 2016.

[3]

Google. Bazel. http://bazel.io/, 2016.

[4]

Berwyn Hoyt, Bryan Hoyt, and Ben Hoyt. fabricate – the better build tool. https://github.com/SimonAlfie/fabricate, 2009.

[5]

Jane Street Group. Jenga. https://github.com/janestreet/jenga, 2016.

[6]

Steven Knight. Building software with SCons. Computing in Science and Engineering, 7(1):79–88, 2005.

Digital Library

[7]

Eyal Lotem. Buildsome build system. https://github.com/ ElastiLotem/buildsome, 2016.

[8]

Douglas H. Martin and James R. Cordy. On the maintenance complexity of Makefiles. In Proceedings of the 2016 7th Interenational Workshop on Emerging Trends in Software Metrics, WETSoM ’16, 2016.

Digital Library

[9]

Douglas H. Martin, James R. Cordy, Bram Adams, and Giulio Antoniol. Make it simple: An empirical analysis of GNU Make feature use in open source projects. In Proc. of the IEEE International Conference on Program Comprehension, ICPC’15, pages 207–217. IEEE Press, 2015.

Digital Library

[10]

Evan Martin. Ninja. In Tavish Armstrong, editor, The Performance of Open Source Applications, chapter 3. 2013.

[11]

Shane McIntosh, Bram Adams, Thanh H.D. Nguyen, Yasutaka Kamei, and Ahmed E. Hassan. An empirical study of build maintenance effort. In Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pages 141–150. ACM, 2011.

Digital Library

[12]

Peter Miller. Recursive make considered harmful. Journal of AUUG Inc, 19(1):14–25, 1998.

[13]

Neil Mitchell. Shake before building - replacing Make with Haskell. In ICFP ’12: Proceedings of the 17th ACM SIGPLAN International Conference on Functional Programming. ACM, September 2012.

Digital Library

[14]

Andrey Mokhov and Victor Khomenko. Algebra of parameterised graphs. ACM Transactions on Embedded Computing Systems, 13(4s):143, 2014.

Digital Library

[15]

Avery Pennarun. redo: a top-down software build system. https:// github.com/apenwarr/redo, 2012.

[16]

Mike Shal. Build system rules and algorithms. http://gittup.org/ tup/build_system_rules_and_algorithms.pdf, 2009.

[17]

Peter Smith. Software Build Systems. Pearson Education, 2011.

Cited By

Spall SMitchell NTobin-Hochstadt S(2020)Build scripts with perfect dependenciesProceedings of the ACM on Programming Languages10.1145/34282374:OOPSLA(1-28)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3428237
MOKHOV AMITCHELL NPEYTON JONES S(2020)Build systems à la carte: Theory and practiceJournal of Functional Programming10.1017/S095679682000008830Online publication date: 21-Apr-2020
https://doi.org/10.1017/S0956796820000088
Ghaleb TDa Costa DZou Y(2019)An empirical study of the long duration of continuous integration buildsEmpirical Software Engineering10.1007/s10664-019-09695-924:4(2102-2139)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10664-019-09695-9
Show More Cited By

Index Terms

Non-recursive make considered harmful: build systems at scale
1. Software and its engineering
  1. Software notations and tools
    1. General programming languages

Recommendations

Non-recursive make considered harmful: build systems at scale
Haskell '16

Most build systems start small and simple, but over time grow into hairy monsters that few dare to touch. As we demonstrate in this paper, there are a few issues that cause build systems major scalability challenges, and many pervasively used build ...
Shake before building: replacing make with haskell
ICFP '12

Most complex software projects are compiled using a build tool (e.g. make), which runs commands in an order satisfying user-defined dependencies. Unfortunately, most build tools require all dependencies to be specified before the build starts. This ...
A sound and optimal incremental build system with dynamic dependencies
OOPSLA 2015: Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications

Build systems are used in all but the smallest software projects to invoke the right build tools on the right files in the right order. A build system must be sound (after a build, generated files consistently reflect the latest source files) and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Haskell 2016: Proceedings of the 9th International Symposium on Haskell

September 2016

192 pages

ISBN:9781450344340

DOI:10.1145/2976002

Program Chair:
Geoffrey Mainland

ACM SIGPLAN Notices Volume 51, Issue 12
Haskell '16
December 2016
192 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/3241625
Editor:
Geoffrey Mainland
Drexel University, USA
Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

ICFP'16

Sponsor:

SIGPLAN

ICFP'16: ACM SIGPLAN International Conference on Functional Programming

September 22 - 23, 2016

Nara, Japan

Acceptance Rates

Overall Acceptance Rate 57 of 143 submissions, 40%

Upcoming Conference

ICFP '25

Sponsor:
sigplan

ACM SIGPLAN International Conference on Functional Programming

October 12 - 18, 2025

Singapore , Singapore

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
532
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)4

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Spall SMitchell NTobin-Hochstadt S(2020)Build scripts with perfect dependenciesProceedings of the ACM on Programming Languages10.1145/34282374:OOPSLA(1-28)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3428237
MOKHOV AMITCHELL NPEYTON JONES S(2020)Build systems à la carte: Theory and practiceJournal of Functional Programming10.1017/S095679682000008830Online publication date: 21-Apr-2020
https://doi.org/10.1017/S0956796820000088
Ghaleb TDa Costa DZou Y(2019)An empirical study of the long duration of continuous integration buildsEmpirical Software Engineering10.1007/s10664-019-09695-924:4(2102-2139)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10664-019-09695-9
Stevens PWąsowski APaige RHaugen Ø(2018)Towards sound, optimal, and flexible building from megamodelsProceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems10.1145/3239372.3239378(301-311)Online publication date: 14-Oct-2018
https://dl.acm.org/doi/10.1145/3239372.3239378
Konat GErdweg SVisser EHuchard MKästner CFraser G(2018)Scalable incremental building with dynamic task dependenciesProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering10.1145/3238147.3238196(76-86)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1145/3238147.3238196
Mokhov AMitchell NPeyton Jones S(2018)Build systems à la carteProceedings of the ACM on Programming Languages10.1145/32367742:ICFP(1-29)Online publication date: 30-Jul-2018
https://dl.acm.org/doi/10.1145/3236774
Bisong ETran EBaysal OGonzalez-Barahona JHindle ATan L(2017)Built to last or built too fast?Proceedings of the 14th International Conference on Mining Software Repositories10.1109/MSR.2017.36(487-490)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/MSR.2017.36
Escot L(2023)Crafting Extensible Forward Incremental Parallel Embedded Build SystemsProceedings of the 35th Symposium on Implementation and Application of Functional Languages10.1145/3652561.3652568(1-11)Online publication date: 29-Aug-2023
https://dl.acm.org/doi/10.1145/3652561.3652568

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents