research-article

Public Access

Reproducible Containers

Authors:

Omar S. Navarro Leija,

Kelly Shiptoski,

Nicholas Renner,

Ryan R. Newton,

Joseph DeviettiAuthors Info & Claims

ASPLOS '20: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems

Pages 167 - 182

https://doi.org/10.1145/3373376.3378519

Published: 13 March 2020 Publication History

Abstract

We describe the design and implementation of DetTrace, a reproducible container abstraction for Linux implemented in user space. All computation that occurs inside a DetTrace container is a pure function of the initial filesystem state of the container. Reproducible containers can be used for a variety of purposes, including replication for fault-tolerance, reproducible software builds and reproducible data analytics. We use DetTrace to achieve, in an automatic fashion, reproducibility for 12,130 Debian package builds, containing over 800 million lines of code, as well as bioinformatics and machine learning workflows. We show that, while software in each of these domains is initially irreproducible, DetTrace brings reproducibility without requiring any hardware, OS or application changes. DetTrace's performance is dictated by the frequency of system calls: IO-intensive software builds have an average overhead of 3.49x, while a compute-bound bioinformatics workflow is under 2%.

References

[1]

Bazel. https://bazel.build/.

[2]

ReproducibleBuilds. https://wiki.debian.org/ReproducibleBuilds.

[3]

Pachyderm reproducible data science homepage. https://www.pachyderm.io.

[4]

Code Ocean homepage. https://codeocean.com.

[5]

John B. Carter, John K. Bennett, and Willy Zwaenepoel. Implementation and performance of munin. SIGOPS Oper. Syst. Rev., 25(5):152--164, September 1991.

Digital Library

[6]

Jonathan Goldstein, Ahmed Abdelhamid, Mike Barnett, Sebastian Burckhardt, Badrish Chandramouli, Darren Gehring, Niel Lebeck, Umar Farooq Minhas, Ryan Newton, Rahee Ghosh Peshawaria, et al. Ambrosia: Providing performant virtual resiliency for distributed applications. Technical report, Technical report, 2018. https://aka. ms/amb-tr, 2018.

[7]

Fred B. Schneider. Implementing fault-tolerant services using the state machine approach: A tutorial. ACM Comput. Surv., 22(4):299--319, December 1990.

Digital Library

[8]

Andreas Haeberlen, Petr Kouznetsov, and Peter Druschel. Peerreview: Practical accountability for distributed systems. SIGOPS Oper. Syst. Rev., 41(6):175--188, October 2007.

Digital Library

[9]

Robert Bocchino, Mohsen Vakilian, Vikram Adve, Danny Dig, Sarita Adve, Stephen Heumann, Rakesh Komuravelli, Jeffrey Overbey, Patrick Simmons, and Hyojin Sung. A type and effect system for deterministic parallel java. In Proceeding of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications - OOPSLA '09, page 97, Orlando, Florida, USA, 2009.

Digital Library

[10]

Simon Marlow, Ryan R. Newton, and Simon Peyton Jones. A monad for deterministic parallelism. In Proceedings of the 4th ACM symposium on Haskell, Haskell '11, pages 71--82. ACM, 2011.

Digital Library

[11]

Ryan G. Scott, Omar S. Navarro Leija, Joseph Devietti, and Ryan R. Newton. Monadic composition for deterministic, parallel batch processing. Proc. ACM Program. Lang., 1(OOPSLA):73:1--73:26, October 2017.

Digital Library

[12]

Amittai Aviram, Shu-Chun Weng, Sen Hu, and Bryan Ford. Efficient system-enforced deterministic parallelism. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, 2010.

Digital Library

[13]

Nicholas Hunt, Tom Bergan, Luis Ceze, and Steven D. Gribble. DDOS: taming nondeterminism in distributed systems. In Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems, volume 48, pages 499--508, March 2013.

Digital Library

[14]

Tom Bergan, Nicholas Hunt, Luis Ceze, and Steven Gribble. Deterministic process groups in dOS. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, 2010.

Digital Library

[15]

Abadi, Daniel J. and Faleiro, Jose M. An Overview of Deterministic Database Systems. Communications of the ACM, 61(9):78--88, August 2018.

Digital Library

[16]

Raymond Chen. Why are the module timestamps in Windows 10 so nonsensical? https://blogs.msdn.microsoft.com/oldnewthing/20180103-00/?p=97705.

[17]

Jared Parsons. Deterministic builds in Roslyn. http://blog.paranoidcoding.com/2016/04/05/deterministic-builds-in-roslyn.html.

[18]

tar: please add --clamp-mtime to only update mtimes after a given time. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=790415., June 2015.

[19]

Ren, Zhilei and Jiang, He and Xuan, Jifeng and Yang, Zijiang. Automated Localization for Unreproducible Builds. In Proceedings of the 40th International Conference on Software Engineering, ICSE '18, pages 71--81, New York, NY, USA, 2018. ACM.

[20]

National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. The National Academies Press, Washington, DC, 2019.

[21]

Rosemary Nan Ke and Alex Lamb and Olexa Bilaniuk and Anirudh Goyal and Yoshua Bengio. Reproducibility in Machine Learning: An ICLR 2019 Workshop. https://sites.google.com/view/icml-reproducibility-workshop/home.

[22]

Daniel Maskit. Problems Getting TensorFlow to behave Deterministically. https://github.com/tensorflow/tensorflow/issues/16889.

[23]

Jennifer Villa and Yoav Zimmerman. Reproducibility in ML: why it matters and how to achieve it. https://determined.ai/blog/reproducibility-in-ml/, May 2018.

[24]

Li Lu and Michael L. Scott. Toward a formal semantic framework for deterministic parallel programming. In Proceedings of the 25th International Conference on Distributed Computing, DISC'11, page 460--474, Berlin, Heidelberg, 2011. Springer-Verlag.

Digital Library

[25]

Lindsey Kuper, Aaron Turon, Neelakantan R. Krishnaswami, and Ryan R. Newton. Freeze after writing: quasi-deterministic parallel programming with lvars. In POPL, pages 257--270, 2014.

Digital Library

[26]

Gerald J. Popek and Robert P. Goldberg. Formal requirements for virtualizable third generation architectures. Commun. ACM, 17(7):412--421, July 1974.

Digital Library

[27]

Christopher Domas. Breaking the x86 Instruction Set. Black Hat, 2017. https://www.youtube.com/watch?v=KrksBdWcZgQ.

[28]

Intel 64 and IA-32 Architectures Software Developer Manuals. https://software.intel.com/en-us/articles/intel-sdm.

[29]

Robert O'Callahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, and Nimrod Partush. Engineering record and replay for deployability: Extended technical report, 2017.

[30]

Intel Xeon Processor E3--1200 v3 Product Family. https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e3--1200v3-spec-update.pdf, July 2018. HSW136: Software Using Intel TSX May Result in Unpredictable System Behavior.

[31]

Amittai Aviram, Sen Hu, Bryan Ford, and Ramakrishna Gummadi. Determinating timing channels in compute clouds. In Proceedings of the 2010 ACM workshop on Cloud computing security workshop, CCSW '10, page 103--108, New York, NY, USA, 2010. ACM.

Digital Library

[32]

Marek Olszewski, Jason Ansel, and Saman Amarasinghe. Kendo: Efficient deterministic multithreading in software. In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIV, pages 97--108, New York, NY, USA, 2009. ACM.

Digital Library

[33]

Alexandros Stamatakis. Raxml version 8: a tool for phylogenetic analysis and post-analysis of large phylogenies. Bioinformatics, 30(9):1312, 2014.

[34]

Ramu Chenna, Hideaki Sugawara, Tadashi Koike, Rodrigo Lopez, Toby J. Gibson, Desmond G. Higgins, and Julie D. Thompson. Multiple sequence alignment with the clustal series of programs. Nucleic Acids Research, 31(13):3497, 2003.

[35]

Sean R. Eddy. Profile hidden markov models. Bioinformatics, 14(9):755--763, 1998.

[36]

tensorflow. Models and examples built with TensorFlow. https://github.com/tensorflow/models/tree/master/tutorials/image. Commit 583408.

[37]

Reprotest GitLab page. https://salsa.debian.org/reproducible-builds/reprotest.

[38]

strip-nondeterminism Debian Package Description. https://packages.debian.org/sid/strip-nondeterminism.

[39]

Packages in Stretch/Amd64 Which Failed to Build Reproducibly. https://tests.reproducible-builds.org/debian/stretch/amd64/index_FTBR.html.

[40]

Thomas J. Leblanc and John M. Mellor-Crummey. Debugging parallel programs with instant replay. IEEE Transactions on Computers, C-36(4):471--482, April 1987.

Digital Library

[41]

Michiel Ronsse and Koen De Bosschere. RecPlay: a fully integrated practical record/replay system. ACM Transactions on Computer Systems, 17(2):133--152, May 1999.

Digital Library

[42]

David Devecsery, Michael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen. Eidetic systems. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, OSDI'14, pages 525--540, Berkeley, CA, USA, 2014. USENIX Association.

Digital Library

[43]

Dongyoon Lee, Benjamin Wester, Kaushik Veeraraghavan, Satish Narayanasamy, Peter M. Chen, and Jason Flinn. Respec: efficient online multiprocessor replayvia speculation and external determinism. In Proceedings of the fifteenth edition of ASPLOS on Architectural support for programming languages and operating systems - ASPLOS '10, page 77, Pittsburgh, Pennsylvania, USA, 2010.

Digital Library

[44]

Ali José Mashtizadeh, Tal Garfinkel, David Terei, David Mazieres, and Mendel Rosenblum. Towards practical default-on multi-core record/replay. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '17, pages 693--708, New York, NY, USA, 2017. ACM.

Digital Library

[45]

George W. Dunlap, Samuel T. King, Sukru Cinar, Murtaza A. Basrai, and Peter M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. SIGOPS Oper. Syst. Rev., 36(SI):211--224, December 2002.

[46]

George W. Dunlap, Dominic G. Lucchetti, Michael A. Fetterman, and Peter M. Chen. Execution replay of multiprocessor virtual machines. In Proceedings of the Fourth ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '08, pages 121--130, New York, NY, USA, 2008. ACM.

Digital Library

[47]

Robert OtextquoterightCallahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, and Nimrod Partush. Engineering record and replay for deployability. In 2017 USENIX Annual Technical Conference (USENIX ATC 17), pages 377--389, Santa Clara, CA, 2017. USENIX Association.

[48]

VMware: VMware workstation zealot: Enhanced execution record / replay in workstation 6.5, April 2008.

[49]

Program Record/Replay Toolkit. https://software.intel.com/en-us/articles/program-recordreplay-toolkit.

[50]

Joseph Devietti, Brandon Lucia, Luis Ceze, and Mark Oskin. DMP: deterministic shared memory multiprocessing. In Proceedings of the 14th international conference on Architectural support for programming languages and operating systems (ASPLOS '09), page 85, Washington, DC, USA, 2009.

Digital Library

[51]

Tom Bergan, Owen Anderson, Joseph Devietti, Luis Ceze, and Dan Grossman. Coredet: a compiler and runtime system for deterministic multithreaded execution. In ACM SIGARCH Computer Architecture News, volume 38, pages 53--64. ACM, 2010.

Digital Library

[52]

Derek R. Hower, Polina Dudnik, David A. Wood, and Mark D. Hill. Calvin: Deterministic or not? free will to choose. In Proceedings of the 17th International Symposium on High-Performance Computer Architecture (HPCA), 2011.

[53]

Joseph Devietti, Jacob Nelson, Tom Bergan, Luis Ceze, and Dan Grossman. RCDC: a relaxed consistency deterministic computer. In Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systems, 2011.

Digital Library

[54]

Tongping Liu, Charlie Curtsinger, and Emery D. Berger. Dthreads: efficient deterministic multithreading. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP '11, pages 327--336, New York, NY, USA, 2011.

Digital Library

[55]

Timothy Merrifield and Jakob Eriksson. Conversion: multi-version concurrency control for main memory segments. In Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys '13, pages 127--139, New York, NY, USA, 2013. ACM.

Digital Library

[56]

Kai Lu, Xu Zhou, Tom Bergan, and Xiaoping Wang. Efficient deterministic multithreading without global barriers. In Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, February 2014.

Digital Library

[57]

Timothy Merrifield, Joseph Devietti, and Jakob Eriksson. High-performance determinism with total store order consistency. In Proceedings of the Tenth European Conference on Computer Systems, EuroSys '15, pages 31:1--31:13, New York, NY, USA, 2015. ACM.

Digital Library

[58]

Timothy Merrifield, Sepideh Roghanchi, Joseph Devietti, and Jakob Eriksson. Lazy determinism for faster deterministic multithreading. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '19, pages 879--891, New York, NY, USA, 2019. ACM.

Digital Library

[59]

Manuel M. T. Chakravarty, Roman Leshchinskiy, Simon Peyton Jones, Gabriele Keller, and Simon Marlow. Data parallel haskell: A status report. In Proceedings of the 2007 Workshop on Declarative Aspects of Multicore Programming, DAMP '07, pages 10--18, New York, NY, USA, 2007. ACM.

Digital Library

[60]

Simon Marlow, Patrick Maier, Hans-Wolfgang Loidl, Mustafa K Aswad, and Phil Trinder. Seq no more: better strategies for parallel haskell. In ACM Sigplan Notices, volume 45, pages 91--102. ACM, 2010.

Digital Library

[61]

Ryan R. Newton, Ömer S. Augacan, Peter Fogg, and Sam Tobin-Hochstadt. Parallel type-checking with haskell using saturating lvars and stream generators. In Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP '16, pages 6:1--6:12, New York, NY, USA, 2016. ACM.

Digital Library

[62]

Daan Leijen, Manuel Fahndrich, and Sebastian Burckhardt. Prettier concurrency: Purely functional concurrent revisions. In Proceedings of the 4th ACM Symposium on Haskell, Haskell '11, pages 83--94. ACM, 2011.

Digital Library

[63]

Robert L. Bocchino and Vikram S. Adve. Types, regions, and effects for safe programming with object-oriented parallel frameworks. In Proceedings of the 25th European Conference on Object-oriented Programming, ECOOP'11, pages 306--332, Berlin, Heidelberg, 2011. Springer-Verlag.

Cited By

Keshani MVelican TBot GProksch S(2024)AROMA: Automatic Reproduction of Maven ArtifactsProceedings of the ACM on Software Engineering10.1145/36437641:FSE(836-858)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643764
Liu CGong SFonseca PAamodt TJerger NSwift M(2023)KIT: Testing OS-Level Virtualization for Functional Interference BugsProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3575693.3575731(427-441)Online publication date: 27-Jan-2023
https://dl.acm.org/doi/10.1145/3575693.3575731
Fourné MWermke DEnck WFahl SAcar Y(2023)It’s like flossing your teeth: On the Importance and Challenges of Reproducible Builds for Software Supply Chain Security2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179320(1527-1544)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179320
Show More Cited By

Index Terms

Reproducible Containers
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Process management
        Multiprocessing / multiprogramming / multitasking

Recommendations

Reproducible Online Search Experiments
Advances in Information Retrieval
Abstract
In the empirical sciences, the evidence is commonly manifested by experimental results. However, very often, these findings are not reproducible, hindering scientific progress. Innovations in the field of information retrieval (IR) are mainly ...
A detailed performance analysis of UDP/IP, TCP/IP, and M-VIA network protocols using Linux/SimOS

This paper presents a performance study of UDP/IP, TCP/IP, and M-VIA using Linux/SimOS. Linux/SimOS is a Linux operating system port to a complete machine simulator SimOS. A complete machine simulator includes all the system components, such as CPU, ...
A tool for converting Linux device drivers into Solaris compatible binaries
Research Articles

The Linux operating system is quickly becoming a standard, attracting a wide user community and supporting a broad variety of applications and devices. Other vendors, such as Sun, have provided Linux-compatible system call interfaces to their kernels, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASPLOS '20: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems

March 2020

1412 pages

ISBN:9781450371025

DOI:10.1145/3373376

General Chair:
James Larus
EPFL
,
Program Chairs:
Luis Ceze
University of Washington
,
Karin Strauss
Microsoft

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ASPLOS '20

Sponsor:

ASPLOS '20: Architectural Support for Programming Languages and Operating Systems

March 16 - 20, 2020

Lausanne, Switzerland

Acceptance Rates

Overall Acceptance Rate 535 of 2,713 submissions, 20%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
2,126
Total Downloads

Downloads (Last 12 months)420
Downloads (Last 6 weeks)41

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Keshani MVelican TBot GProksch S(2024)AROMA: Automatic Reproduction of Maven ArtifactsProceedings of the ACM on Software Engineering10.1145/36437641:FSE(836-858)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643764
Liu CGong SFonseca PAamodt TJerger NSwift M(2023)KIT: Testing OS-Level Virtualization for Functional Interference BugsProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3575693.3575731(427-441)Online publication date: 27-Jan-2023
https://dl.acm.org/doi/10.1145/3575693.3575731
Fourné MWermke DEnck WFahl SAcar Y(2023)It’s like flossing your teeth: On the Importance and Challenges of Reproducible Builds for Software Supply Chain Security2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179320(1527-1544)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179320
Bavay MReisecker MEgger TKorhammer D(2022)Inishell 2.0: semantically driven automatic GUI generation for scientific modelsGeoscientific Model Development10.5194/gmd-15-365-202215:2(365-378)Online publication date: 18-Jan-2022
https://doi.org/10.5194/gmd-15-365-2022
Newman ZMeyers JTorres-Arias SYin HStavrou ACremers CShi E(2022)SigstoreProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security10.1145/3548606.3560596(2353-2367)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3548606.3560596
Shi YWen MCogo FChen BJiang Z(2022)An Experience Report on Producing Verifiable Builds for Large-Scale Commercial SystemsIEEE Transactions on Software Engineering10.1109/TSE.2021.309269248:9(3361-3377)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TSE.2021.3092692
Merino XOtero C(2022)The Cost of Virtualizing Time in Linux Containers2022 IEEE Cloud Summit10.1109/CloudSummit54781.2022.00016(63-68)Online publication date: Oct-2022
https://doi.org/10.1109/CloudSummit54781.2022.00016
Butler SGamalielsson JLundell BBrax CMattsson AGustavsson TFeist JKvarnström BLönroth E(2022)On business adoption and use of reproducible builds for open and closed source softwareSoftware Quality Journal10.1007/s11219-022-09607-z31:3(687-719)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1007/s11219-022-09607-z

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents