research-article

Understanding Real-World Concurrency Bugs in Go

Authors:

Linhai Song, and

Yiying ZhangAuthors Info & Claims

ASPLOS '19: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems

April 2019

Pages 865 - 878

https://doi.org/10.1145/3297858.3304069

Published: 04 April 2019 Publication History

Abstract

Go is a statically-typed programming language that aims to provide a simple, efficient, and safe way to build multi-threaded software. Since its creation in 2009, Go has matured and gained significant adoption in production and open-source software. Go advocates for the usage of message passing as the means of inter-thread communication and provides several new concurrency mechanisms and libraries to ease multi-threading programming. It is important to understand the implication of these new proposals and the comparison of message passing and shared memory synchronization in terms of program errors, or bugs. Unfortunately, as far as we know, there has been no study on Go's concurrency bugs. In this paper, we perform the first systematic study on concurrency bugs in real Go programs. We studied six popular Go software including Docker, Kubernetes, and gRPC. We analyzed 171 concurrency bugs in total, with more than half of them caused by non-traditional, Go-specific problems. Apart from root causes of these bugs, we also studied their fixes, performed experiments to reproduce them, and evaluated them with two publicly-available Go bug detectors. Overall, our study provides a better understanding on Go's concurrency models and can guide future researchers and practitioners in writing better, more reliable Go software and in developing debugging and diagnosis tools for Go.

References

[1]

Principles of designing Go APIs with channels. URL: https://inconshreveable.com/07-08--2014/principles-of-designing-go-apis-with-channels/.

[2]

The Go Blog: Share Memory By Communicating. URL: https://blog.golang.org/share-memory-by-communicating.

[3]

Sameer Ajmani. Advanced Go Concurrency Patterns. URL: https://talks.golang.org/2013/advconc.slide.

[4]

Joy Arulraj, Po-Chun Chang, Guoliang Jin, and Shan Lu. Production-run software failure diagnosis via hardware performance counters. In Proceedings of the 18th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '13), Houston, Texas, USA, March 2013.

Digital Library

[5]

Joy Arulraj, Guoliang Jin, and Shan Lu. Leveraging the short-term memory of hardware to diagnose production-run software failures. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '14), Salt Lake City, Utah, USA, March 2014.

Digital Library

[6]

boltdb. An embedded key/value database for Go. URL: https://github.com/boltdb/bolt.

[7]

Yan Cai and W. K. Chan. Magiclock: Scalable detection of potential deadlocks in large-scale multithreaded programs. IEEE Transactions on Software Engineering, 40(3):266--281, 2014.

Digital Library

[8]

Lee Chew and David Lie. Kivati: Fast detection and prevention of atomicity violations. In Proceedings of the 5th European Conference on Computer systems (EuroSys '10), Paris, France, April 2010.

Digital Library

[9]

Andy Chou, Junfeng Yang, Benjamin Chelf, Seth Hallem, and Dawson Engler. An empirical study of operating systems errors. In Proceedings of the 18th ACM symposium on Operating Systems Principles (SOSP '01), Banff, Alberta, Canada, October 2001.

Digital Library

[10]

Cockroach. CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. URL: https://github.com/cockroachdb/cockroach.

[11]

Russ Cox. Bell Labs and CSP Threads. URL: http://swtch.com/ rsc/thread/.

[12]

Graphql Developers. A thread-safe way of appending errors into Result.Errors. URL: https://github.com/graphql-go/graphql/pull/434.

[13]

Docker. Docker - Build, Ship, and Run Any App, Anywhere. URL: https://www.docker.com/.

[14]

John Erickson, Madanlal Musuvathi, Sebastian Burckhardt, and Kirk Olynyk. Effective data-race detection for the kernel. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation (OSDI '10), Vancouver, BC, Canada, October 2010.

Digital Library

[15]

ETCD. A distributed, reliable key-value store for the most critical data of a distributed system. URL: https://github.com/coreos/etcd.

[16]

Cormac Flanagan and Stephen N Freund. Atomizer: A dynamic atomicity checker for multithreaded programs. In Proceedings of the 31st ACM SIGPLAN-SIGACT symposium on Principles of programming languages (POPL '04), Venice, Italy, January 2004.

Digital Library

[17]

Qi Gao, Wenbin Zhang, Zhezhe Chen, Mai Zheng, and Feng Qin. 2ndstrike: Toward manifesting hidden concurrency typestate bugs. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '11), Newport Beach, California, USA, March 2011.

Digital Library

[18]

GitHub. The fifteen most popular languages on GitHub. URL: https://octoverse.github.com/.

[19]

Google. A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. URL: https://github.com/grpc/grpc-go.

[20]

Google. Effective Go. URL: https://golang.org/doc/effective_go.html.

[21]

Google. Effective Go: Concurrency. URL: https://golang.org/doc/effective_go.html#concurrency.

[22]

Google. RPC Benchmarking. URL: https://grpc.io/docs/guides/benchmarking.html.

[23]

Google. The Go Programming Language -- Release History. URL: https://golang.org/doc/devel/release.html.

[24]

Rui Gu, Guoliang Jin, Linhai Song, Linjie Zhu, and Shan Lu. What change history tells us about thread synchronization. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, Bergamo, Italy, August 2015.

Digital Library

[25]

Haryadi S. Gunawi, Mingzhe Hao, Tanakorn Leesatapornwongsa, Tiratat Patana-anake, Thanh Do, Jeffry Adityatama, Kurnia J. Eliazar, Agung Laksono, Jeffrey F. Lukman, Vincentius Martin, and Anang D. Satria. What bugs live in the cloud? a study of 3000

[26]

issues in cloud systems. In Proceedings of the ACM Symposium on Cloud Computing (SOCC' 14), Seattle, Washington, USA, November 2014.

[27]

Hectane. Lightweight SMTP client written in Go. URL: https://github.com/hectane.

[28]

C. A. R. Hoare. Communicating Sequential Processes. Communications of the ACM, 21(8):666--677, 1978.

Digital Library

[29]

Omar Inverso, Truc L. Nguyen, Bernd Fischer, Salvatore La Torre, and Gennaro Parlato. Lazy-cseq: A context-bounded model checking tool for multi-threaded c-programs. In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15), Lincoln, Nebraska, USA, November 2015.

Digital Library

[30]

Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. Understanding and detecting real-world performance bugs. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI' 12), Beijing, China, June 2012.

Digital Library

[31]

Guoliang Jin, Linhai Song, Wei Zhang, Shan Lu, and Ben Liblit. Automated atomicity-violation fixing. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI' 11), San Jose, California, USA, June 2011.

Digital Library

[32]

Guoliang Jin, Aditya V. Thakur, Ben Liblit, and Shan Lu. Instrumentation and sampling strategies for cooperative concurrency bug isolation. In Proceedings of the ACM International Conference on Object oriented programming systems languages and applications (OOPSLA '10), Reno/Tahoe, Nevada, USA, October 2010.

Digital Library

[33]

Guoliang Jin, Wei Zhang, Dongdong Deng, Ben Liblit, and Shan Lu. Automated concurrency-bug fixing. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI'12), Hollywood, California, USA, October 2012.

Digital Library

[34]

Pallavi Joshi, Chang-Seo Park, Koushik Sen, and Mayur Naik. A randomized dynamic program analysis technique for detecting real deadlocks. In Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '09), Dublin, Ireland, June 2009.

Digital Library

[35]

Horatiu Jula, Daniel Tralamazza, Cristian Zamfir, and George Candea. Deadlock immunity: Enabling systems to defend against deadlocks. In Proceedings of the 8th USENIX Conference on Operating systems design and implementation (OSDI '08), San Diego, California, USA, December 2008.

Digital Library

[36]

Daniel Kroening, Daniel Poetzl, Peter Schrammel, and Björn Wachter. Sound static deadlock analysis for c/pthreads. In 31st IEEE/ACM International Conference on Automated Software Engineering (ASE '16), Singapore, Singapore, September 2016.

Digital Library

[37]

Kubernetes. Production-Grade Container Orchestration. URL: https://kubernetes.io/.

[38]

Leslie Lamport. Concurrent Reading and Writing. Communications of the ACM, 20(11):806--811, 1977.

Digital Library

[39]

Julien Lange, Nicholas Ng, Bernardo Toninho, and Nobuko Yoshida. Fencing off go: Liveness and safety for channel-based programming. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL '17), Paris, France, January 2017.

Digital Library

[40]

Julien Lange, Nicholas Ng, Bernardo Toninho, and Nobuko Yoshida. A static verification framework for message passing in go using behavioural types. In IEEE/ACM 40th International Conference on Software Engineering (ICSE '18), Gothenburg, Sweden, June 2018.

Digital Library

[41]

Tanakorn Leesatapornwongsa, Jeffrey F. Lukman, Shan Lu, and Haryadi S. Gunawi. Taxdc: A taxonomy of non-deterministic concurrency bugs in datacenter distributed systems. In Proceedings of the 21th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '16), Atlanta, Georgia, USA, April 2016.

Digital Library

[42]

Zhenmin Li, Lin Tan, Xuanhui Wang, Shan Lu, Yuanyuan Zhou, and Chengxiang Zhai. Have things changed now?: An empirical study of bug characteristics in modern open source software. In Proceedings of the 1st workshop on Architectural and system support for improving software dependability (ASID '06), San Jose, California, USA, October 2006.

Digital Library

[43]

Ziyi Lin, Darko Marinov, Hao Zhong, Yuting Chen, and Jianjun Zhao. Jacontebe: A benchmark suite of real-world java concurrency bugs. In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15), Lincoln, Nebraska, USA, November 2015.

Digital Library

[44]

Haopeng Liu, Yuxi Chen, and Shan Lu. Understanding and generating high quality patches for concurrency bugs. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '16), Seattle, Washington, USA, November 2016.

Digital Library

[45]

Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Shan Lu. A study of linux file system evolution. In Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST '13), San Jose, California, USA, February 2013.

Digital Library

[46]

Shan Lu, Soyeon Park, Eunsoo Seo, and Yuanyuan Zhou. Learning from mistakes -- a comprehensive study of real world concurrency bug characteristics. In Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '08), Seattle, Washington, USA, March 2008.

Digital Library

[47]

Shan Lu, Joseph Tucek, Feng Qin, and Yuanyuan Zhou. Avio: Detecting atomicity violations via access interleaving invariants. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '06), San Jose, California, USA, October 2006.

Digital Library

[48]

Brandon Lucia and Luis Ceze. Finding concurrency bugs with context-aware communication graphs. In Proceedings of the 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO '09), New York, USA, December 2009.

Digital Library

[49]

Kedar S. Namjoshi. Are concurrent programs that are easier to write also easier to check? In Workshop on Exploiting Concurrency Efficiently and Correctly, 2008.

[50]

Nicholas Ng and Nobuko Yoshida. Static deadlock detection for concurrent go by global session graph synthesis. In Proceedings of the 25th International Conference on Compiler Construction (CC '16), Barcelona, Spain, March 2016.

Digital Library

[51]

Rob Pike. Go Concurrency Patterns. URL: https://talks.golang.org/2012/concurrency.slide.

[52]

Dawson R. Engler and Ken Ashcraft. Racerx: Effective, static detection of race conditions and deadlocks. In Proceedings of the 19th ACM symposium on Operating systems principles (SOSP '03), Bolton Landing, New York, USA, October 2003.

Digital Library

[53]

Stefan Savage, Michael Burrows, Greg Nelson, Patrick Sobalvarro, and Thomas Anderson. Eraser: A dynamic data race detector for multithreaded programs. ACM Transactions on Computer Systems, 15(4):391--411, 1997.

Digital Library

[54]

Konstantin Serebryany and Timur Iskhodzhanov. Threadsanitizer: Data race detection in practice. In Proceedings of the Workshop on Binary Instrumentation and Applications (WBIA '09), New York, USA, December 2009.

Digital Library

[55]

Vivek K Shanbhag. Deadlock-detection in java-library using static-analysis. In 15th Asia-Pacific Software Engineering Conference (APSEC '08), Beijing, China, December 2008.

Digital Library

[56]

Francesco Sorrentino. Picklock: A deadlock prediction approach under nested locking. In Proceedings of the 22nd International Symposium on Model Checking Software (SPIN '15), Stellenbosch, South Africa, August 2015.

Digital Library

[57]

Kai Stadtmüller, Martin Sulzmann, and Peter" Thiemann. Static trace-based deadlock analysis for synchronous mini-go. In 14th Asian Symposium on Programming Languages and Systems (APLAS '16), Hanoi, Vietnam, November 2016.

[58]

Jie Wang, Wensheng Dou, Yu Gao, Chushu Gao, Feng Qin, Kang Yin, and Jun Wei. A comprehensive study on real world concurrency bugs in node.js. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE '17), Urbana-Champaign, Illinois, USA, October 2017.

Digital Library

[59]

Yin Wang, Terence Kelly, Manjunath Kudlur, Stéphane Lafortune, and Scott A Mahlke. Gadara: Dynamic deadlock avoidance for multithreaded programs. In Proceedings of the 8th USENIX Conference on Operating systems design and implementation (OSDI '08), San Diego, California, USA, December 2008.

Digital Library

[60]

Yin Wang, Stéphane Lafortune, Terence Kelly, Manjunath Kudlur, and Scott A. Mahlke. The theory of deadlock avoidance via discrete control. In Proceedings of the 36th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages (POPL '09), Savannah, Georgia, USA, January 2009.

Digital Library

[61]

Wikipedia. Go (programming language). URL: https://en.wikipedia.org/wiki/Go_(programming_language).

[62]

Weiwei Xiong, Soyeon Park, Jiaqi Zhang, Yuanyuan Zhou, and Zhiqiang Ma. Ad hoc synchronization considered harmful. In Proceedings of the 9th USENIX Conference on Operating systems design and implementation (OSDI '10), Vancouver, British Columbia, Canada, October 2010.

Digital Library

[63]

Jie Yu and Satish Narayanasamy. A case for an interleaving constrained shared-memory multi-processor. In Proceedings of the 36th annual International symposium on Computer architecture (ISCA '09), Austin, Texas, USA, June 2009.

Digital Library

[64]

Yuan Yu, Tom Rodeheffer, and Wei Chen. Racetrack: Efficient detection of data race conditions via adaptive tracking. In Proceedings of the 20th ACM symposium on Operating systems principles (SOSP '05), Brighton, United Kingdom, October 2005.

Digital Library

[65]

Wei Zhang, Chong Sun, and Shan Lu. Conmem: detecting severe concurrency bugs through an effect-oriented approach. In Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '10), Pittsburgh, Pennsylvania, USA, March 2010.

Digital Library

Cited By

Qin BChen YLiu HZhang HWen QSong LZhang Y(2024)Understanding and Detecting Real-World Safety Issues in RustIEEE Transactions on Software Engineering10.1109/TSE.2024.338039350:6(1306-1324)Online publication date: Jun-2024
https://doi.org/10.1109/TSE.2024.3380393
Saioc GShirchenko DChabbi M(2024)Unveiling and Vanquishing Goroutine Leaks in Enterprise Microservices: A Dynamic Analysis Approach2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)10.1109/CGO57630.2024.10444835(411-422)Online publication date: 2-Mar-2024
https://doi.org/10.1109/CGO57630.2024.10444835
Chopra AChristie V SAgmon NAn BRicci AYeoh W(2023)Communication Meaning: Foundations and Directions for Systems ResearchProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598840(1786-1791)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.5555/3545946.3598840
Show More Cited By

Index Terms

Understanding Real-World Concurrency Bugs in Go
1. Computing methodologies
  1. Concurrent computing methodologies
    1. Concurrent programming languages
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Automatically detecting and fixing concurrency bugs in go software systems
ASPLOS '21: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

Go is a statically typed programming language designed for efficient and reliable concurrent programming. For this purpose, Go provides lightweight goroutines and recommends passing messages using channels as a less error-prone means of thread ...
Read More
Who goes first? detecting go concurrency bugs via message reordering
ASPLOS '22: Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

Go is a young programming language invented to build safe and efficient concurrent programs. It provides goroutines as lightweight threads and channels for inter-goroutine communication. Programmers are encouraged to explicitly pass messages through ...
Read More
Understanding memory and thread safety practices and issues in real-world Rust programs
PLDI 2020: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation

Rust is a young programming language designed for systems software development. It aims to provide safety guarantees like high-level languages and performance efficiency like low-level languages. The core design of Rust is a set of strict safety rules ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASPLOS '19: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems

April 2019

1126 pages

ISBN:9781450362405

DOI:10.1145/3297858

General Chairs:
Iris Bahar
Brown University
,
Maurice Herlihy
Brown University
,
Program Chairs:
Emmett Witchel
University of Texas, Austin
,
Alvin Lebeck
Duke University

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASPLOS '19

Sponsor:

ASPLOS '19: Architectural Support for Programming Languages and Operating Systems

April 13 - 17, 2019

RI, Providence, USA

Acceptance Rates

ASPLOS '19 Paper Acceptance Rate 74 of 351 submissions, 21%;

Overall Acceptance Rate 535 of 2,713 submissions, 20%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
1,159
Total Downloads

Downloads (Last 12 months)167
Downloads (Last 6 weeks)11

Other Metrics

View Author Metrics

Citations

Cited By

Qin BChen YLiu HZhang HWen QSong LZhang Y(2024)Understanding and Detecting Real-World Safety Issues in RustIEEE Transactions on Software Engineering10.1109/TSE.2024.338039350:6(1306-1324)Online publication date: Jun-2024
https://doi.org/10.1109/TSE.2024.3380393
Saioc GShirchenko DChabbi M(2024)Unveiling and Vanquishing Goroutine Leaks in Enterprise Microservices: A Dynamic Analysis Approach2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)10.1109/CGO57630.2024.10444835(411-422)Online publication date: 2-Mar-2024
https://doi.org/10.1109/CGO57630.2024.10444835
Chopra AChristie V SAgmon NAn BRicci AYeoh W(2023)Communication Meaning: Foundations and Directions for Systems ResearchProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598840(1786-1791)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.5555/3545946.3598840
Li CChen RWang BWang ZYu TJiang YGu BYang MJust RFraser G(2023)An Empirical Study on Concurrency Bugs in Interrupt-Driven Embedded SoftwareProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598140(1345-1356)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598140
Tunç HMathur UPavlogiannis AViswanathan M(2023)Sound Dynamic Deadlock Prediction in Linear TimeProceedings of the ACM on Programming Languages10.1145/35912917:PLDI(1733-1758)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591291
Jiang ZWen MYang YPeng CYang PJin H(2023)Effective Concurrency Testing for Go via Directional Primitive-Constrained Interleaving Exploration2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00086(1364-1376)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00086
Zheng WDeng PGui KWu X(2023)An Abstract Syntax Tree based static fuzzing mutation for vulnerability evolution analysisInformation and Software Technology10.1016/j.infsof.2023.107194158:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.infsof.2023.107194
Ghosh DSingh JSingh J(2023)Concurrent fault localization using ANNInternational Journal of System Assurance Engineering and Management10.1007/s13198-023-02081-114:6(2345-2353)Online publication date: 19-Sep-2023
https://doi.org/10.1007/s13198-023-02081-1
Dilley NLange J(2023)Automated verification of concurrent go programs via bounded model checkingAutomated Software Engineering10.1007/s10515-023-00391-z30:2Online publication date: 26-Aug-2023
https://dl.acm.org/doi/10.1007/s10515-023-00391-z
Kirszenberg AMartin AMoreau HRenault E(2023)Go2Pins: a framework for the LTL verification of Go programs (extended version)International Journal on Software Tools for Technology Transfer10.1007/s10009-022-00692-w25:1(77-94)Online publication date: 6-Jan-2023
https://doi.org/10.1007/s10009-022-00692-w
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents