research-article

Public Access

Testing probabilistic programming systems

Authors:

Owolabi Legunsen,

Sasa MisailovicAuthors Info & Claims

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 574 - 586

https://doi.org/10.1145/3236024.3236057

Published: 26 October 2018 Publication History

Abstract

Probabilistic programming systems (PP systems) allow developers to model stochastic phenomena and perform efficient inference on the models. The number and adoption of probabilistic programming systems is growing significantly. However, there is no prior study of bugs in these systems and no methodology for systematically testing PP systems. Yet, testing PP systems is highly non-trivial, especially when they perform approximate inference. In this paper, we characterize 118 previously reported bugs in three open-source PP systems—Edward, Pyro and Stan—and pro- pose ProbFuzz, an extensible system for testing PP systems. Prob- Fuzz allows a developer to specify templates of probabilistic models, from which it generates concrete probabilistic programs and data for testing. ProbFuzz uses language-specific translators to generate these concrete programs, which use the APIs of each PP system. ProbFuzz finds potential bugs by checking the output from running the generated programs against several oracles, including an accu- racy checker. Using ProbFuzz, we found 67 previously unknown bugs in recent versions of these PP systems. Developers already accepted 51 bug fixes that we submitted to the three PP systems, and their underlying systems, PyTorch and TensorFlow.

References

[1]

Eric Atkinson and Michael Carbin. 2016.

[2]

Towards correct-by-construction probabilistic inference. In NIPS Workshop on Machine Learning Systems.

[3]

Adrian Bachmann, Christian Bird, Foyzur Rahman, Premkumar Devanbu, and Abraham Bernstein. 2010. The missing links: bugs and bug-fix commits. In FSE.

Digital Library

[4]

Antoine Balestrat. 2018. CCG - random C Code Generator. https://github.com/ Mrktn/ccg.

[5]

Titus Barik, Justin Smith, Kevin Lubick, Elisabeth Holmes, Jing Feng, Emerson Murphy-Hill, and Chris Parnin. 2017. Do developers read compiler error messages?. In ICSE.

Digital Library

[6]

Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016.

[7]

Analyzing the state of static analysis: A large-scale evaluation in open source software. In SANER.

[8]

Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. 2010. A few billion lines of code later: using static analysis to find bugs in the real world. CACM 53, 2 (2010).

Digital Library

[9]

Mateus Borges, Antonio Filieri, Marcelo d’Amorim, and Corina S Păsăreanu. 2015. Iterative distribution-aware sampling for probabilistic symbolic execution. In FSE.

Digital Library

[10]

Johannes Borgström, Andrew D Gordon, Michael Greenberg, James Margetson, and Jurgen Van Gael. 2011. Measure transformer semantics for Bayesian machine learning. In ESOP.

[11]

Bob Carpenter. 2017.

[12]

https://github.com/standev/stan/issues/603.

[13]

Bob Carpenter, Andrew Gelman, Matt Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Michael A Brubaker, Jiqiang Guo, Peter Li, Allen Riddell, et al. 2016.

[14]

Stan: A probabilistic programming language. JSTATSOFT 20, 2 (2016).

[15]

Tsong Y Chen, Shing C Cheung, and Siu Ming Yiu. 1998.

[16]

Metamorphic testing: a new approach for generating next test cases. Technical Report. CS Department, Hong Kong University of Science and Technology.

[17]

Tsong Yueh Chen, Jianqiang Feng, and TH Tse. 2002. Metamorphic testing of programs on partial differential equations: a case study. In COMPSAC.

[18]

Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. 2013. Taming compiler fuzzers. In PLDI.

Digital Library

[19]

Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016.

[20]

Coverage-directed differential testing of JVM implementations. In PLDI.

[21]

Shauvik Roy Choudhary, Husayn Versee, and Alessandro Orso. 2010. WEBDIFF: Automated identification of cross-browser issues in web applications. In 2010 IEEE International Conference on Software Maintenance. IEEE, 1–10.

Digital Library

[22]

Guillaume Claret, Sriram K. Rajamani, Aditya V. Nori, Andrew D. Gordon, and Johannes Borgström. 2013. Bayesian Inference Using Data Flow Analysis. In FSE. ESEC/FSE ’18, November 4–9, 2018, Lake Buena Vista, FL, USA Saikat Dutta, Owolabi Legunsen, Zixin Huang and Sasa Misailovic

Digital Library

[23]

Gregory F Cooper. 1990. The computational complexity of probabilistic inference using Bayesian belief networks. Artificial intelligence 42, 2 (1990).

Digital Library

[24]

Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines. In ESEC/FSE.

Digital Library

[25]

Anthony Di Franco, Hui Guo, and Cindy Rubio-González. 2017. A comprehensive study of real-world numerical bug characteristics. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 509–519.

Digital Library

[26]

Domain Error in arguments 2018.

[27]

https://github.com/uber/pyro/issues/875.

[28]

Edward Commit 002a27e 2016.

[29]

https://github.com/bleilab/edward/commit/ 002a27e.

[30]

Edward Commit 10118db 2016.

[31]

https://github.com/bleilab/edward/commit/ 10118db.

[32]

Edward Commit 3616d41 2016.

[33]

https://github.com/bleilab/edward/commit/ 3616d41.

[34]

Edward commit 3616d41 2016.

[35]

https://github.com/bleilab/edward/commit/ 3616d41.

[36]

Edward commit 43d8a39 2016.

[37]

https://github.com/bleilab/edward/commit/ 43d8a39.

[38]

Edward Commit 79f4193 2017.

[39]

https://github.com/bleilab/edward/commit/ 79f4193.

[40]

Edward commit 79f4193 2017.

[41]

https://github.com/bleilab/edward/commit/ 79f4193.

[42]

Edward Commit 972a9d9 2017.

[43]

https://github.com/bleilab/edward/commit/ 972a9d9.

[44]

Edward Commit c9afcb1f 2017.

[45]

https://github.com/bleilab/edward/commit/ c9afcb1.

[46]

Edward Commit fe01657 2016.

[47]

https://github.com/bleilab/edward/commit/ fe01657.

[48]

EdwardWebPage 2018. Edward. http://edwardlib.org.

[49]

Antonio Filieri, Corina S Păsăreanu, and Willem Visser. 2013. Reliability analysis in symbolic pathfinder. In ICSE.

Digital Library

[50]

Timon Gehr, Sasa Misailovic, and Martin Vechev. 2016.

[51]

PSI: Exact Symbolic Inference for Probabilistic Programs. In CAV.

[52]

Jaco Geldenhuys, Matthew B Dwyer, and Willem Visser. 2012.

[53]

Probabilistic symbolic execution. In ISSTA.

[54]

Andrew Gelman, Daniel Lee, and Jiqiang Guo. 2015. Stan A Probabilistic Programming Language for Bayesian Inference and Optimization. Journal of Educational and Behavioral Statistics (2015).

[55]

Wally R Gilks, Andrew Thomas, and David J Spiegelhalter. 1994. A language and program for complex Bayesian modelling. The Statistician (1994).

[56]

Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010.

[57]

Test generation through programming in UDITA. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. ACM, 225–234.

Digital Library

[58]

Patrice Godefroid, Adam Kiezun, and Michael Y. Levin. 2008. Grammar-based Whitebox Fuzzing. In PLDI.

Digital Library

[59]

Patrice Godefroid, Michael Y Levin, David A Molnar, et al. 2008. Automated whitebox fuzz testing. In NDSS.

[60]

Noah Goodman, Vikash Mansinghka, Daniel M Roy, Keith Bonawitz, and Joshua B Tenenbaum. 2012. Church: a language for generative models. arXiv preprint arXiv:1206.3255 (2012).

Digital Library

[61]

Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. 2014. Probabilistic programming. In FoSE.

Digital Library

[62]

Arnaud Gotlieb and Bernard Botella. 2003. Automated metamorphic testing. In COMPSAC.

Digital Library

[63]

Alex Groce, Gerard Holzmann, and Rajeev Joshi. 2007. Randomized differential testing as a prelude to formal verification. In ICSE.

Digital Library

[64]

Roger B Grosse, Siddharth Ancha, and Daniel M Roy. 2016.

[65]

Measuring the reliability of MCMC inference with bidirectional Monte Carlo. In NIPS.

[66]

Farah Hariri, August Shi, Owolabi Legunsen, Milos Gligoric, Sarfraz Khurshid, and Sasa Misailovic. 2018. Approximate Transformations as Mutation Operators. In ICST.

[67]

Shawn Hershey, Jeff Bernstein, Bill Bradley, Andrew Schweitzer, Noah Stein, Theo Weber, and Ben Vigoda. 2012.

[68]

Accelerating inference: towards a full language, compiler and hardware stack. arXiv preprint arXiv:1212.2991 (2012).

[69]

Christian Holler, Kim Herzig, and Andreas Zeller. 2012.

[70]

Fuzzing with Code Fragments. In USENIX Security.

[71]

Daniel Jackson and Craig A Damon. 1996.

[72]

Elements of style: Analyzing a software design feature with a counterexample detector. TSE 22, 7 (1996).

Digital Library

[73]

Vilas Jagannath, Yun Young Lee, Brett Daniel, and Darko Marinov. 2009. Reducing the Costs of Bounded-Exhaustive Testing. In FASE.

Digital Library

[74]

Nils Jansen, Christian Dehnert, Benjamin Lucien Kaminski, Joost-Pieter Katoen, and Lukas Westhofen. 2016. Bounded Model Checking for Probabilistic Programs. ATVA (2016).

[75]

Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman, and David M. Blei. 2015.

[76]

Automatic Variational Inference in Stan. In NIPS.

[77]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014.

[78]

Compiler Validation via Equivalence Modulo Inputs. In PLDI.

[79]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In OOPSLA.

Digital Library

[80]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Randomized Stress-testing of Link-time Optimizers. In ISSTA.

Digital Library

[81]

Nuo Li, JeeHyun Hwang, and Tao Xie. 2008. Multiple-implementation Testing for XACML Implementations. In TAV-WEB.

Digital Library

[82]

Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core Compiler Fuzzing. In PLDI.

Digital Library

[83]

Daniel Liew, Daniel Schemmel, Cristian Cadar, Alastair F Donaldson, Rafael Zähl, and Klaus Wehrle. 2017. Floating-point symbolic execution: A case study in N-version programming. In ASE.

Digital Library

[84]

Huai Liu, Fei-Ching Kuo, Dave Towey, and Tsong Yueh Chen. 2014.

[85]

How effectively does metamorphic testing alleviate the oracle problem? TSE 40, 1 (2014).

Digital Library

[86]

Kasper Luckow, Corina S Păsăreanu, Matthew B Dwyer, Antonio Filieri, and Willem Visser. 2014. Exact and approximate probabilistic symbolic execution for nondeterministic programs. In ASE.

Digital Library

[87]

Rupak Majumdar and Ru-Gang Xu. 2007.

[88]

Directed Test Generation Using Symbolic Grammars. In ASE.

[89]

Vikash Mansinghka, Daniel Selsam, and Yura Perov. 2014. Venture: a higherorder probabilistic programming platform with programmable inference. arXiv preprint 1404.0099 (2014).

[90]

Piotr Mardziel, Stephen Magill, Michael Hicks, and Mudhakar Srivatsa. 2013.

[91]

Dynamic enforcement of knowledge-based security policies using probabilistic abstract interpretation. JCS 21, 4 (2013).

Digital Library

[92]

Peter M. Maurer. 1990. Generating test data with enhanced context-free grammars. Ieee Software 7, 4 (1990), 50–55.

Digital Library

[93]

William M McKeeman. 1998.

[94]

Differential testing for software. DEC Digital Technical Journal 10, 1 (1998).

[95]

Barton P. Miller, Louis Fredriksen, and Bryan So. 1990. An Empirical Study of the Reliability of UNIX Utilities. CACM 33, 12 (1990).

Digital Library

[96]

David Monniaux. 2000. Abstract interpretation of probabilistic semantics. In SAS.

Digital Library

[97]

Carroll Morgan, Annabelle McIver, and Karen Seidel. 1996. Probabilistic predicate transformers. TOPLAS 18, 3 (1996).

Digital Library

[98]

Christian Murphy and Gail E Kaiser. 2010.

[99]

Improving the dependability of machine learning applications. Technical Report. CS Department, Columbia University.

[100]

Christian Murphy, Gail E Kaiser, Lifeng Hu, and Leon Wu. 2008. Properties of Machine Learning Applications for Use in Metamorphic Testing. In SEKE.

[101]

Chandrakana Nandi, Dan Grossman, Adrian Sampson, Todd Mytkowicz, and Kathryn S McKinley. 2017. Debugging probabilistic programs. In MAPL.

Digital Library

[102]

Aditya V Nori, Chung-Kil Hur, Sriram K Rajamani, and Selva Samuel. 2014. R2: An efficient MCMC sampler for probabilistic programs. In AAAI.

Digital Library

[103]

Peter Oehlert. 2005. Violating Assumptions with Fuzzing. IEEE Security and Privacy 3, 2 (2005).

Digital Library

[104]

Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D. Ernst, Deric Pang, and Benjamin Keller. 2017. Evaluating and Improving Fault Localization. In ICSE.

Digital Library

[105]

Paul Purdom. 1972. A sentence generator for testing parsers. BIT Numerical Mathematics 12, 3 (1972).

[106]

Pyro Commit 0f1d27e 2017.

[107]

https://github.com/uber/pyro/commit/0f1d27e.

[108]

Pyro commit 3671b06 2017.

[109]

https://github.com/uber/pyro/commit/3671b06.

[110]

Pyro Commit 6e26f5e 2017.

[111]

https://github.com/uber/pyro/commit/6e26f5e.

[112]

Pyro Commit 7b6cf58 2017.

[113]

https://github.com/uber/pyro/commit/7b6cf58.

[114]

Pyro commit 7b6cf58 2017.

[115]

https://github.com/uber/pyro/commit/7b6cf58.

[116]

Pyro Commit 8c14f36 2017.

[117]

https://github.com/uber/pyro/commit/8c14f36.

[118]

Pyro Commit b94f06a 2017.

[119]

https://github.com/uber/pyro/commit/b94f06a.

[120]

Pyro Commit f5a51fe 2017.

[121]

https://github.com/uber/pyro/commit/f5a51f.

[122]

Pyro Issue 303 2017.

[123]

https://github.com/uber/pyro/issues/303.

[124]

PyroWebPage 2018. Pyro. http://pyro.ai.

[125]

PyTorchWebPage 2018. PyTorch. http://pytorch.org.

[126]

Michael Rath, Jacob Rendall, Jin LC Guo, Jane Cleland-Huang, and Patrick Mäder. 2018. Traceability in the wild: automatically augmenting incomplete trace links. In ICSE.

Digital Library

[127]

Cindy Rubio-González, Cuong Nguyen, Hong Diep Nguyen, James Demmel, William Kahan, Koushik Sen, David H Bailey, Costin Iancu, and David Hough. 2013. Precimonious: Tuning assistant for floating-point precision. In SC.

[128]

Sriram Sankaranarayanan, Aleksandar Chakarov, and Sumit Gulwani. 2013.

[129]

Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. ACM SIGPLAN Notices 48, 6 (2013).

Digital Library

[130]

Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016.

[131]

A survey on metamorphic testing. IEEE Transactions on software engineering 42, 9 (2016), 805–824.

[132]

Daniel Selsam, Percy Liang, and David L Dill. 2017.

[133]

Developing Bug-Free Machine Learning Systems With Formal Mathematics. arXiv preprint Testing Probabilistic Programming Systems ESEC/FSE ’18, November 4–9, 2018, Lake Buena Vista, FL, USA arXiv:1706.08605 (2017).

[134]

SGLD produces Nan in output 2018.

[135]

https://github.com/bleilab/edward/issues/ 859.

[136]

Guoqiang Shu, Yating Hsu, and David Lee. 2008.

[137]

Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning. In FORTE.

[138]

Emin Gün Sirer and Brian N. Bershad. 1999. Using Production Grammars in Software Testing. In DSL.

Digital Library

[139]

Siwakorn Srisakaokul, Zhengkai Wu, Angello Astorga, Oreoluwa Alebiosu, and Tao Xie. 2018. Multiple-Implementation Testing of Supervised Learning Software. In EDSMLS.

[140]

Stan 2018. http://mcstan.org.

[141]

Stan Best Practices 2018.

[142]

https://github.com/standev/stan/wiki/ Stan-Best-Practices.

[143]

Stan Commit 04fcb74 2013.

[144]

https://github.com/standev/stan/commit/04fcb74.

[145]

Stan Commit 2fc94d4 2017.

[146]

https://github.com/standev/stan/commit/2fc94d4.

[147]

Stan Commit 40c8224 2016.

[148]

https://github.com/standev/stan/commit/40c8224.

[149]

Stan commit 45a82fd 2015.

[150]

https://github.com/standev/stan/commit/45a82fd.

[151]

Stan Commit 5224850 2015.

[152]

https://github.com/standev/stan/commit/5224850.

[153]

Stan Commit 5845db97 2016.

[154]

https://github.com/standev/stan/commit/ 5845db9.

[155]

Stan Commit 7a98bd2 2016.

[156]

https://github.com/standev/stan/commit/7a98bd2.

[157]

Stan Commit 99289c85 2015.

[158]

https://github.com/standev/stan/commit/ 99289c85.

[159]

Stan Commit ae423b2 2017.

[160]

https://github.com/standev/stan/commit/ae423b2.

[161]

Stan Commit b8d5086 2013.

[162]

https://github.com/standev/stan/commit/b8d5086.

[163]

Stan Issue #2178 2017.

[164]

https://github.com/standev/stan/issues/2178.

[165]

Stan Language Manual. Appendix D. 2018. http://mcstan.org/users/ documentation/index.html.

[166]

Stan Language Manual. Chapter 3.1. 2018. http://mcstan.org/users/ documentation/index.html.

[167]

Stan Pedantic Mode 2018.

[168]

https://github.com/standev/stan/wiki/ Stan-Parser-Pedantic-Mode.

[169]

Stan throws error with empty array 2018.

[170]

https://github.com/standev/pystan/ issues/437.

[171]

Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding and analyzing compiler warning defects. In ICSE.

Digital Library

[172]

Michael Sutton, Adam Greene, and Pedram Amini. 2007.

[173]

Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley Professional.

Digital Library

[174]

Symmetric Mean Absolute Percentage Error. Armstrong 1985. http://www. forecastingprinciples.com/files/LRFCh13b.pdf.

[175]

Kunal Taneja, Nuo Li, Madhuri R. Marri, Tao Xie, and Nikolai Tillmann. 2010.

[176]

MiTV: Multiple-implementation Testing of User-input Validators for Web Applications. In ASE.

[177]

TensorFlowWebPage 2018. TensorFlow. https://www.tensorflow.org.

[178]

Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep probabilistic programming. In ICLR.

[179]

Dustin Tran, Alp Kucukelbir, Adji B. Dieng, Maja Rudolph, Dawen Liang, and David M. Blei. 2016. Edward: A library for probabilistic modeling, inference, and criticism. arXiv (2016).

[180]

Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. 2014. A new approach to probabilistic programming inference. In AISTATS.

[181]

Tao Xie, Kunal Taneja, Shreyas Kale, and Darko Marinov. 2007.

[182]

Towards a framework for differential unit testing of object-oriented programs. In AST.

[183]

Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. JSS 84, 4 (2011).

Digital Library

[184]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In PLDI.

Digital Library

[185]

Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal program enumeration for rigorous compiler testing. In PLDI.

Digital Library

Cited By

Wang JPham HLi QTan LGuo YAziz AMeijer E(2025) D 3 : Differential Testing of Distributed Deep Learning With Model Generation IEEE Transactions on Software Engineering10.1109/TSE.2024.346165751:1(38-52)Online publication date: Jan-2025
https://doi.org/10.1109/TSE.2024.3461657
Chatterjee KGoharshady ENovotný PŽikelić Đ(2024)Equivalence and Similarity Refutation for Probabilistic ProgramsProceedings of the ACM on Programming Languages10.1145/36564628:PLDI(2098-2122)Online publication date: 20-Jun-2024
https://doi.org/10.1145/3656462
Zhang LMiranskyy A(2024)Automated Flakiness Detection in Quantum Software Bug Reports2024 IEEE International Conference on Quantum Computing and Engineering (QCE)10.1109/QCE60285.2024.10274(179-181)Online publication date: 15-Sep-2024
https://doi.org/10.1109/QCE60285.2024.10274
Show More Cited By

Index Terms

Testing probabilistic programming systems
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Storm: program reduction for testing and debugging probabilistic programming systems
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Probabilistic programming languages offer an intuitive way to model uncertainty by representing complex probability models as simple probabilistic programs. Probabilistic programming systems (PP systems) hide the complexity of inference algorithms away ...
Probabilistic (logic) programming concepts

A multitude of different probabilistic programming languages exists today, all extending a traditional programming language with primitives to support modeling of complex, structured probability distributions. Each of these languages employs its own ...
Grey-box concolic testing on binary code
ICSE '19: Proceedings of the 41st International Conference on Software Engineering

We present grey-box concolic testing, a novel path-based test case generation method that combines the best of both white-box and grey-box fuzzing. At a high level, our technique systematically explores execution paths of a program under test as in white-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

October 2018

987 pages

ISBN:9781450355735

DOI:10.1145/3236024

General Chair:
Gary T. Leavens
University of Central Florida, USA
,
Program Chairs:
Alessandro Garcia
PUC-Rio, Brazil
,
Corina S. Păsăreanu
NASA Ames Research Center, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ESEC/FSE '18

Sponsor:

SIGSOFT

ESEC/FSE '18: 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 4 - 9, 2018

FL, Lake Buena Vista, USA

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
941
Total Downloads

Downloads (Last 12 months)194
Downloads (Last 6 weeks)28

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang JPham HLi QTan LGuo YAziz AMeijer E(2025) D 3 : Differential Testing of Distributed Deep Learning With Model Generation IEEE Transactions on Software Engineering10.1109/TSE.2024.346165751:1(38-52)Online publication date: Jan-2025
https://doi.org/10.1109/TSE.2024.3461657
Chatterjee KGoharshady ENovotný PŽikelić Đ(2024)Equivalence and Similarity Refutation for Probabilistic ProgramsProceedings of the ACM on Programming Languages10.1145/36564628:PLDI(2098-2122)Online publication date: 20-Jun-2024
https://doi.org/10.1145/3656462
Zhang LMiranskyy A(2024)Automated Flakiness Detection in Quantum Software Bug Reports2024 IEEE International Conference on Quantum Computing and Engineering (QCE)10.1109/QCE60285.2024.10274(179-181)Online publication date: 15-Sep-2024
https://doi.org/10.1109/QCE60285.2024.10274
Huang ZDutta SMisailovic S(2024)Debugging convergence problems in probabilistic programs via program representation learning with SixthSenseInternational Journal on Software Tools for Technology Transfer (STTT)10.1007/s10009-024-00737-226:3(249-268)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10009-024-00737-2
Tassarotti JTristan J(2023)Verified Density Compilation for a Probabilistic Programming LanguageProceedings of the ACM on Programming Languages10.1145/35912457:PLDI(615-637)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591245
Bagnall AStewart GBanerjee A(2023)Formally Verified Samplers from Probabilistic Programs with Loops and ConditioningProceedings of the ACM on Programming Languages10.1145/35912207:PLDI(1-24)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591220
Dong YLi ZTian YSun CGodfrey MNagappan M(2023)Bash in the Wild: Language Usage, Code Smells, and BugsACM Transactions on Software Engineering and Methodology10.1145/351719332:1(1-22)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3517193
Trinh VNishimura YYu H(2023)Development of a Web System for Detecting Bugs in Probabilistic Programs2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C60940.2023.00035(122-130)Online publication date: 22-Oct-2023
https://doi.org/10.1109/QRS-C60940.2023.00035
Paltenghi MPradel M(2023)MorphQ: Metamorphic Testing of the Qiskit Quantum Computing Platform2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00202(2413-2424)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00202
Xia CDutta SMisailovic SMarinov DZhang L(2023)Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00154(1801-1813)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00154
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten