Consensus in Data Management: With Use Cases in Edge-Cloud and Blockchain Systems
Pages 4233 - 4236
Abstract
Consensus is a fundamental problem in distributed systems, involving the challenge of achieving agreement among distributed nodes. It plays a critical role in various distributed data management problems. This tutorial aims to provide a comprehensive primer for data management researchers on the topic of consensus and its fundamental and modern applications in data management. We begin by exploring the basic principles of consensus, including the problem statement, system models, failure scenarios, and various consensus algorithms such as Paxos and its variants. The tutorial then delves into the applications of consensus in distributed data management, focusing on distributed atomic commitment and data replication. We explain how consensus is integral to these areas and present examples of research and industry work that apply consensus to data management. The tutorial extends to modern use cases of consensus in the evolving landscapes of edge-cloud systems and blockchain technology. We discuss how consensus mechanisms are being adapted and applied in these areas, highlighting their growing importance in emerging areas of data management. By exploring these cutting-edge applications, participants will gain insights into how consensus is shaping ongoing and future research on distributed data management.
The tutorial builds on the authors' recent book "Consensus in Data Management: from Distributed Commit to Blockchain". The book will serve as the foundation and reading material for the tutorial. This tutorial targets data management researchers and practitioners to equip them with the knowledge and perspective needed to innovate in these emerging fields. This includes graduate students and junior researchers starting their careers in the area of distributed data management. Also, it includes researchers in other areas of data management who wish to explore the area of distributed data management with the goal of utilizing it in their own fields.
References
[1]
F. Nawab and M. Sadoghi, "Consensus in data management: From distributed commit to blockchain," Foundations and Trends® in Databases, vol. 12, no. 4, pp. 221--364, 2023.
[2]
P. A. Alsberg and J. D. Day, "A principle for resilient sharing of distributed resources," in Proceedings of the 2nd International Conference on Software Engineering (ICSE), 1976, pp. 562--570. (1976).
[3]
B. Lampson and H. E. Sturgis, "Crash recovery in a distributed data storage system," in Computer Science Lab, Xerox Parc, Palo Alto, CA, Technical Report. (1976), 1976.
[4]
J. N. Gray, "Notes on data base operating systems," in Operating Systems, An Advanced Course, ser. Lecture Notes in Computer Science. Springer, 1978, vol. 60, pp. 393--481. (1978).
[5]
J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild et al., "Spanner: Google's globally distributed database," ACM Transactions on Computer Systems (TOCS), vol. 31, no. 3, pp. 1--22. (2013), 2013.
[6]
F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber, "Bigtable: A distributed storage system for structured data," ACM Transactions on Computer Systems (TOCS), vol. 26, no. 2, pp. 1--26. (2008), 2008.
[7]
J. Baker, C. Bond, J. C. Corbett, J. J. Furman, A. Khorlin, J. Larson, J. Leon, Y. Li, A. Lloyd, and V. Yushprakh, "Megastore: Providing scalable, highly available storage for interactive services," in Innovative Data Systems Research (CIDR), pp. 223--234. (2011).
[8]
F. Nawab, D. Agrawal, and A. El Abbadi, "Dpaxos: Managing data closer to users for low-latency and mobile applications," in ACM International Conference on Management of Data (SIGMOD), 2018, pp. 1221--1236. (2018).
[9]
D. Abadi, O. Arden, F. Nawab, and M. Shadmon, "Anylog: a grand unification of the internet of things," in Conference on Innovative Data Systems Research (CIDR '20), 2020.
[10]
F. Nawab and M. Shadmon, "The tipping point of edge-cloud data management." in Conference on Innovative Data Systems Research (CIDR '24), 2024.
[11]
S. Gupta, S. Rahnama, E. Linsenmayer, F. Nawab, and M. Sadoghi, "Reliable transactions in serverless-edge architecture," in 2023 IEEE 39th International Conference on Data Engineering (ICDE). IEEE, 2023, pp. 301--314.
[12]
A. Khan, S. Mehrotra, and F. Nawab, "Transedge: Supporting efficient read queries across untrusted edge nodes." 2023.
[13]
N. Mittal and F. Nawab, "Coolsm: Distributed and cooperative indexing across edge and cloud machines," in 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 2021, pp. 420--431.
[14]
F. Nawab, D. Agrawal, and A. El Abbadi, "Nomadic datacenters at the network edge: Data management challenges for the cloud with mobile infrastructure." in EDBT, 2018, pp. 497--500.
[15]
J. Hellings and M. Sadoghi, "Byshard: Sharding in a byzantine environment," Proceedings of the VLDB Endowment, vol. 14, no. 11, pp. 2230--2243. (2021), 2021.
[16]
M. El-Hindi, C. Binnig, A. Arasu, D. Kossmann, and R. Ramamurthy, "BlockchainDB: A shared database on blockchains," Proc. VLDB Endow., vol. 12, no. 11, pp. 1597--1609, 2019.
[17]
T. McConaghy, R. Marques, A. Müller, D. De Jonghe, T. McConaghy, G. McMullen, R. Henderson, S. Bellemare, and A. Granzotto, "Bigchaindb: a scalable blockchain database," white paper, BigChainDB, pp. 53--72, 2016.
[18]
J. Fang, F. Habibi, K. Bruhwiler, F. Alshammari, A. Singh, Y. Zhou, and F. Nawab, "Pelopartition: Improving blockchain resilience to network partitioning," in 2022 IEEE International Conference on Blockchain (Blockchain). IEEE, 2022, pp. 274--281.
[19]
S. Gupta, J. Hellings, and M. Sadoghi, Fault-Tolerant Distributed Transactions on Blockchain, ser. Synthesis Lectures on Data Management. Morgan & Claypool Publishers. (2021), 2021.
[20]
S. Gupta, M. J. Amiri, and M. Sadoghi, "Chemistry behind agreement," in 13th Conference on Innovative Data Systems Research, CIDR 2023, Amsterdam, The Netherlands, January 8--11, 2023. www.cidrdb.org, 2023.
[21]
M. J. Amiri, C. Wu, D. Agrawal, A. E. Abbadi, B. T. Loo, and M. Sadoghi, "The bedrock of byzantine fault tolerance: A unified platform for BFT protocols analysis, implementation, and experimentation," in 21st USENIX Symposium on Networked Systems Design and Implementation, NSDI 2024, Santa Clara, CA, April 15--17, 2024, L. Vanbever and I. Zhang, Eds. USENIX Association, 2024, pp. 371--400.
[22]
S. Gupta, J. Hellings, and M. Sadoghi, "RCC: resilient concurrent consensus for high-throughput secure transaction processing," in IEEE International Conference on Data Engineering (ICDE). IEEE, 2021, pp. 1392--1403. (2021).
[23]
S. Gupta, J. Hellings, S. Rahnama, and M. Sadoghi, "Proof-of-Execution: Reaching consensus through fault-tolerant speculation," in International Conference on Extending Database Technology (EDBT). OpenProceedings.org, 2021, pp. 301--312. (2021).
[24]
J. Chen, S. Gupta, S. Rahnama, and M. Sadoghi, "Power-of-Collaboration: A sustainable resilient ledger built democratically," IEEE Data Eng. Bull., vol. 45, no. 2, pp. 25--36, 2022.
[25]
J. Hellings and M. Sadoghi, "ByShard: sharding in a byzantine environment," VLDB J., vol. 32, no. 6, pp. 1343--1367, 2023.
[26]
S. Gupta, S. Rahnama, J. Hellings, and M. Sadoghi, "ResilientDB: Global scale resilient blockchain fabric," Proceedings of the VLDB Endowment, vol. 13, no. 6, pp. 868--883. (2020), 2020.
[27]
D. Kang, S. Rahnama, J. Hellings, and M. Sadoghi, "SpotLess: Concurrent rotational consensus made practical through rapid view synchronization," in 40th IEEE International Conference on Data Engineering, ICDE 2024, Utrecht, Netherlands, May 13--17, 2024. IEEE, 2024.
[28]
J. Hellings, D. P. Hughes, J. Primero, and M. Sadoghi, "[problem] cerberus: Minimalistic multi-shard byzantine-resilient transaction processing," Journal of Systems Research, vol. 3, no. 1, 2023.
[29]
S. Rahnama, S. Gupta, R. Sogani, D. Krishnan, and M. Sadoghi, "RingBFT: Resilient consensus over sharded ring topology," in International Conference on Extending Database Technology (EDBT). OpenProceedings.org, 2022, pp. 2:298--2:311. (2022).
[30]
S. Gupta, S. Rahnama, S. Pandey, N. Crooks, and M. Sadoghi, "Dissecting BFT consensus: In trusted components we trust!" in Proceedings of the Eighteenth European Conference on Computer Systems, EuroSys 2023, Rome, Italy, May 8--12, 2023, G. A. D. Luna, L. Querzoni, A. Fedorova, and D. Narayanan, Eds. ACM, 2023, pp. 521--539.
[31]
F. Nawab and M. Sadoghi, "Blockplane: A global-scale Byzantizing middleware," in 35th IEEE International Conference on Data Engineering, ICDE 2019, Macao, China, April 8--11, 2019. IEEE, 2019, pp. 124--135.
[32]
J. Hellings and M. Sadoghi, "[Solution] Byzantine cluster-sending in expected constant cost and constant time," J. Syst. Res., vol. 3, no. 1, 2023.
[33]
J. Hellings and M. Sadoghi, "The fault-tolerant cluster-sending problem," in Foundations of Information and Knowledge Systems - 12th International Symposium, FoIKS 2022, Helsinki, Finland, June 20--23, 2022, Proceedings, ser. Lecture Notes in Computer Science, I. Varzinczak, Ed., vol. 13388. Springer, 2022, pp. 168--186.
[34]
L. Lamport, "Paxos made simple," ACM Sigact News, vol. 32, no. 4, pp. 18--25. (2001), 2001.
[35]
L. Lamport, "Fast paxos," Distributed Computing, vol. 19, no. 2, pp. 79--103. (2006), 2006.
[36]
L. Lamport, "Generalized consensus and paxos," Technical Report MSR-TR-2005-33, Microsoft Research. (2005), 2005.
[37]
L. Lamport, D. Malkhi, and L. Zhou, "Stoppable paxos," TechReport, Microsoft Research. (2008), 2008.
[38]
L. Lamport, D. Malkhi, and L. Zhou, "Vertical paxos and primary-backup replication," in ACM symposium on Principles of Distributed Computing (PODC), 2009, pp. 312--313. (2009).
[39]
L. Lamport, D. Malkhi, and L. Zhou, "Reconfiguring a state machine," ACM SIGACT News, vol. 41, no. 1, pp. 63--73. (2010), 2010.
[40]
V. Gramoli, L. Bass, A. Fekete, and D. W. Sun, "Rollup: Non-disruptive rolling upgrade with fast consensus-based dynamic reconfigurations," IEEE Transactions on Parallel and Distributed Systems, vol. 27, no. 9, pp. 2711--2724. (2015), 2015.
[41]
L. Lamport and M. Massa, "Cheap paxos," in International Conference on Dependable Systems and Networks, 2004, pp. 307--314. (2004).
[42]
J. R. Lorch, A. Adya, W. J. Bolosky, R. Chaiken, J. R. Douceur, and J. Howell, "The smart way to migrate replicated stateful services," in ACM SIGOPS European Conference on Computer Systems (EuroSys), 2006, pp. 103--115. (2006).
[43]
M. Biely, Z. Milosevic, N. Santos, and A. Schiper, "S-paxos: Offloading the leader for high throughput state machine replication," in IEEE Symposium on Reliable Distributed Systems (SRDS), 2012, pp. 111--120. (2012).
[44]
A. Ailijiang, A. Charapko, M. Demirbas, and T. Kosar, "Wpaxos: Wide area network flexible consensus," IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 1, pp. 211--223. (2019), 2019.
[45]
T. Kraska, G. Pang, M. J. Franklin, S. Madden, and A. Fekete, "Mdcc: Multi-data center consistency," in ACM European Conference on Computer Systems (EuroSys), 2013, pp. 113--126. (2013).
[46]
Y. Mao, F. P. Junqueira, and K. Marzullo, "Mencius: Building efficient replicated state machine for wans," in USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 369--384. (2008).
[47]
I. Moraru, D. G. Andersen, and M. Kaminsky, "There is more consensus in egalitarian parliaments," in ACM Symposium on Operating Systems Principles (SOSP), 2013, pp. 358--372. (2013).
[48]
B. M. Oki and B. H. Liskov, "Viewstamped replication: A new primary copy method to support highly-available distributed systems," in ACM Symposium on Principles of Distributed Computing (PODC), 1988, pp. 8--17. (1988).
[49]
D. Ongaro and J. Ousterhout, "In search of an understandable consensus algorithm," in USENIX Annual Technical Conference (ATC), 2014, pp. 305--319. (2014).
[50]
F. P. Junqueira, B. C. Reed, and M. Serafini, "Zab: High-performance broadcast for primary-backup systems," in IEEE/IFIP International Conference on Dependable Systems & Networks (DSN), 2011, pp. 245--256. (2011).
[51]
R. Van Renesse, N. Schiper, and F. B. Schneider, "Vive la différence: Paxos vs. viewstamped replication vs. zab," IEEE Transactions on Dependable and Secure Computing, vol. 12, no. 4, pp. 472--484. (2014), 2014.
Index Terms
- Consensus in Data Management: With Use Cases in Edge-Cloud and Blockchain Systems
Index terms have been assigned to the content through auto-classification.
Recommendations
A Blockchain-based platform for data management and sharing
AbstractData management is the collection, processing, storing, and sharing of data. In today's dispensation, data sharing and collaborative data processing is a necessity for multi-partner organizations as it can lead to the discovery of new insight. ...
Application of Blockchain Technology in Data Management: Advantages and Solutions
Big Scientific Data ManagementAbstractBlockchain is considered to be the key technology to lead the transformation of Information Internet to Value Internet. More and more organizations are exploring the industry applications of blockchain, and how to apply blockchain in data ...
The k-simultaneous consensus problem
This paper introduces and investigates the k-simultaneous consensus task: each process participates at the same time in k independent consensus instances until it decides in any one of them. It is shown that the k-simultaneous consensus task is ...
Comments
Information & Contributors
Information
Published In
Publisher
VLDB Endowment
Publication History
Published: 01 August 2024
Published in PVLDB Volume 17, Issue 12
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 29Total Downloads
- Downloads (Last 12 months)29
- Downloads (Last 6 weeks)17
Reflects downloads up to 01 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in