Online Marketplace: A Benchmark for Data Management in Microservices
Article No.: 3, Pages 1 - 26
Abstract
Microservice architectures have become a popular approach for designing scalable distributed applications. Despite their extensive use in industrial settings for over a decade, there is limited understanding of the data management challenges that arise in these applications. Consequently, it has been difficult to advance data system technologies that effectively support microservice applications. To fill this gap, we present Online Marketplace, a microservice benchmark that highlights core data management challenges that existing benchmarks fail to address. These challenges include transaction processing, query processing, event processing, constraint enforcement, and data replication. We have defined criteria for various data management issues to enable proper comparison across data systems and platforms. Through case studies with state-of-the-art data platforms, we discuss the issues encountered while implementing and meeting Online Marketplace's criteria. By capturing the overhead of meeting the key data management requirements that are overlooked by existing benchmarks, we gain actionable insights into the experimental platforms. This highlights the significance of Online Marketplace in advancing future data systems to meet the needs of microservice practitioners.
References
[1]
Materialize [n.d.]. Materialize: The Operational Data Warehouse. Materialize. Retrieved December 12, 2023 from https://materialize.com
[2]
2018. InconsistentStateException with ADO.NET SQL Server grain storage. Retrieved November 25, 2023 from https: //github.com/dotnet/orleans/issues/4697#issuecomment-398556401
[3]
2023. Cluster management in Orleans. Retrieved November 25, 2023 from https://learn.microsoft.com/en-us/dotnet/ orleans/implementation/cluster-management
[4]
2023. Grain persistence. Retrieved November 25, 2023 from https://learn.microsoft.com/en-us/dotnet/orleans/grains/ grain-persistence/?pivots=orleans-7-0
[5]
Olist 2024. Brazilian E-Commerce Public Dataset by Olist. Olist. Retrieved September 5, 2024 from https://www.kaggle. com/datasets/olistbr/brazilian-ecommerce
[6]
DuckDB 2024. DuckDB. DuckDB. Retrieved April 12, 2024 from https://github.com/duckdb
[7]
Microsoft 2024. Microservice Applications. Microsoft. Retrieved September 16, 2024 from https://azure.microsoft.com/enus/ solutions/microservice-applications
[8]
Amazon AWS 2024. Microservices. Amazon AWS. Retrieved September 16, 2024 from https://aws.amazon.com/ microservices
[9]
Alibaba Cloud 2024. Microservices Engine. Alibaba Cloud. Retrieved September 16, 2024 from https://www.alibabacloud. com/en/product/microservices-engine
[10]
Google Cloud 2024. What is Microservices Architecture? Google Cloud. Retrieved September 16, 2024 from https: //cloud.google.com/learn/what-is-microservices-architecture
[11]
Andreas Andreakis, Falguni Jhaveri, Ioannis Papapanagiotou, Mark Cho, Poorna Reddy, and Tongliang Liu. 2019. Delta: A Data Synchronization and Enrichment Platform. Netflix. Retrieved April 8, 2024 from https://netflixtechblog.com/deltaa-data-synchronization-and-enrichment-platform-e82c36a79aee
[12]
Anonymous. 2024. A Benchmark for Data Management Challenges in Microservices. https://zenodo.org/records/12731873 Version with Appendix.
[13]
Arvind Arasu, Mitch Cherniack, Eduardo Galvez, David Maier, Anurag S. Maskey, Esther Ryvkina, Michael Stonebraker, and Richard Tibbetts. 2004. Linear Road: A Stream Data Management Benchmark. In Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30 (Toronto, Canada) (VLDB '04). VLDB Endowment, 480--491.
[14]
BroadleafCommerce. [n.d.]. Broadleaf Microservices Architecture. Broadleaf Commerce. Retrieved March 24, 2024 from https://www.broadleafcommerce.com/product/microservice-architecture
[15]
Sebastian Burckhardt, Badrish Chandramouli, Chris Gillum, David Justo, Konstantinos Kallas, Connor McMahon, Christopher S. Meiklejohn, and Xiangfeng Zhu. 2022. Netherite: efficient execution of serverless workflows. Proc. VLDB Endow. 15, 8 (April 2022), 1591--1604. https://doi.org/10.14778/3529337.3529344
[16]
Sergey Bykov, Alan Geller, Gabriel Kliot, Jim Larus, Ravi Pandya, and Jorgen Thelin. 2010. Orleans: A Framework for Cloud Computing. Technical Report MSR-TR-2010--159. https://www.microsoft.com/en-us/research/publication/ orleans-a-framework-for-cloud-computing/
[17]
Marcus Cavalcanti. 2020. Lessons learned about running Microservices. B2W. Retrieved April 8, 2024 from https://medium.com/b2w-engineering-en/less-running-microservices-a51d952cb50b
[18]
Brian Chavez. 2024. Bogus. Microsoft. Retrieved April 8, 2024 from https://github.com/bchavez/Bogus Bogus is fundamentally a C port of faker.js and inspired by Fluent Validation's syntax sugar.
[19]
Jon Chew. 2019. Avoiding Double Payments in a Distributed Payments System. AirBnb. Retrieved April 8, 2024 from https://medium.com/airbnb-engineering/avoiding-double-payments-in-a-distributed-payments-system-2981f6b070bb
[20]
Dmitry Chornyi. 2018. Engineering Uber's Next-Gen Payments Platform. Uber Technologies Inc. Retrieved April 8, 2024 from https://eng.uber.com/payments-platform
[21]
Richard Cole, Florian Funke, Leo Giakoumakis, Wey Guy, Alfons Kemper, Stefan Krompass, Harumi Kuno, Raghunath Nambiar, Thomas Neumann, Meikel Poess, Kai-Uwe Sattler, Michael Seibold, Eric Simon, and Florian Waas. 2011. The mixed workload CH-benCHmark. In Proceedings of the Fourth International Workshop on Testing Database Systems (Athens, Greece) (DBTest '11). Association for Computing Machinery, New York, NY, USA, Article 8, 6 pages. https: //doi.org/10.1145/1988842.1988850
[22]
Dan Conger. 2021. How Apache Kafka Enables Podium to "Ship It and See What Happens". Confluent. Retrieved April 8, 2024 from https://www.confluent.io/blog/how-apache-kafka-enables-podium-to-ship-it-and-see-what-happens
[23]
Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking Cloud Serving Systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing (Indianapolis, Indiana, USA) (SoCC '10). Association for Computing Machinery, New York, NY, USA, 143--154. https://doi.org/10.1145/1807128. 1807152
[24]
TPC Council. 2005. TPC-W: Benchmarking An Ecommerce Solution. https://www.tpc.org/tpcw/tpc-w_wh.pdf Revision 1.2.
[25]
TPC Council. 2010. TPC Benchmark C revision. https://www.tpc.org/tpcc Revision 5.11.
[26]
Jérôme Darmont and Michel Schneider. 2002. Object-Oriented Database Benchmarks. In Advanced Topics in Database Research, Vol. 1, Keng Siau (Ed.). Idea Group, 34--57.
[27]
Datadog. [n.d.]. What Is Audit Logging? Retrieved November 20, 2023 from https://www.datadoghq.com/knowledgecenter/ audit-logging/#audit-logs-vs-regular-system-logs
[28]
Stephan Ewen. 2020. Stateful Functions 2.0 - An Event-driven Database on Apache Flink. Retrieved November 25, 2023 from https://flink.apache.org/2020/04/07/stateful-functions-2.0-an-event-driven-database-on-apache-flink
[29]
Rafael Ferreira. 2019. Microservices at Nubank, an overview. Nubank. Retrieved April 8, 2024 from https://building. nubank.com.br/microservices-at-nubank-an-overview
[30]
Rafael Ferreira and Edward Wible. 2017. Architecting a Modern Financial Institution. Nubank. Retrieved April 8, 2024 from https://www.infoq.com/presentations/nubank-architecture
[31]
Apache Flink. 2024. Flink Architecture. Apache Flink. Retrieved September 23, 2024 from https://nightlies.apache.org/ flink/flink-docs-master/docs/concepts/flink-architecture
[32]
Martin Fowler. 2017. What do you mean by "Event-Driven"? Retrieved November, 1st 2023 from https://martinfowler. com/articles/201701-event-driven.html
[33]
Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, et al. 2019. An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems. 3--18.
[34]
Seth Gilbert and Nancy Lynch. 2002. Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. SIGACT News 33, 2 (jun 2002), 51--59. https://doi.org/10.1145/564585.564601
[35]
Adam Gluck. 2022. Introducing Domain-Oriented Microservice Architecture. Uber Technologies Inc. Retrieved April 8, 2024 from https://www.uber.com/en-DK/blog/microservice-architecture
[36]
IATA. [n.d.]. Airline Retailing. IATA. Retrieved March 24, 2024 from https://www.iata.org/en/programs/airlinedistribution/ retailing
[37]
imbursepayments. [n.d.]. What is a PSP and how does it work? Retrieved November, 4 2022 from https: //imbursepayments.com/what-is-a-psp
[38]
Ninad Khisti. 2018. Measuring Transactional Integrity in Airbnb's Distributed Payment Ecosystem. AirBnb. Retrieved April 8, 2024 from https://medium.com/airbnb-engineering/measuring-transactional-integrity-in-airbnbs-distributedpayment-ecosystem-a670d6926d22
[39]
Martin Kleppmann, Alastair R. Beresford, and Boerge Svingen. 2019. Online Event Processing. Commun. ACM 62, 5 (apr 2019), 43--49. https://doi.org/10.1145/3312527
[40]
Rodrigo Laigner. 2023. EventBenchmark. Retrieved April 4, 2024 from https://github.com/diku-dk/EventBenchmark/releases/tag/v1.0
[41]
Rodrigo Laigner, Yongluan Zhou, Marcos Antonio Vaz Salles, Yijian Liu, and Marcos Kalinowski. 2021. Data Management in Microservices: State of the Practice, Challenges, and Research Directions. Proc. VLDB Endow. 14, 13 (sep 2021), 3348--3361. https://doi.org/10.14778/3484224.3484232
[42]
Yijian Liu, Rodrigo Laigner, and Yongluan Zhou. 2024. Rethinking State Management in Actor Systems for Cloud-Native Applications. In Proceedings of the 2024 ACM Symposium on Cloud Computing (Redmond, WA, USA) (SoCC '24). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3698038.3698540
[43]
Yijian Liu, Li Su, Vivek Shah, Yongluan Zhou, and Marcos Antonio Vaz Salles. 2022. Hybrid Deterministic and Nondeterministic Execution of Transactions in Actor Systems. In Proceedings of the 2022 International Conference on Management of Data (Philadelphia, PA, USA) (SIGMOD '22). Association for Computing Machinery, New York, NY, USA, 65--78. https://doi.org/10.1145/3514221.3526172
[44]
Materialize. [n.d.]. How Ecommerce Company Drizly uses Materialize for Real-Time Notifications, Alerting, and Personalization. Retrieved November, 4 2022 from https://materialize.com/customer-stories/drizly
[45]
Math.NET. 2023. Math.NET Numerics. Retrieved November 20, 2023 from https://numerics.mathdotnet.com/
[46]
Ricardo Mayerhofer. 2018. restQL: Tackling microservice query complexity. B2W. Retrieved April 8, 2024 from https://medium.com/b2w-engineering-en/restql-tackling-microservice-query-complexity-27def5d09b40
[47]
Osvaldo Santana Neto. 2018. Olist Architecture From Monolith to microservices. Olist. Retrieved March 18, 2024 from https://www.slideshare.net/osantana/olist-architecture-v20
[48]
Matheus Oliveira. 2017. Gerenciamento de dados com micro-serviços e a experiência do iFood. iFood. Retrieved April 8, 2024 from https://www.infoq.com/br/presentations/gerenciamento-de-dados-com-micro-servicos-e-a-experienciado-ifood
[49]
Microsoft Orleans. 2023. Request scheduling. Retrieved November 25, 2023 from https://learn.microsoft.com/enus/ dotnet/orleans/grains/request-scheduling
[50]
Microsoft Orleans. 2024. Grain placement. Microsoft. Retrieved September 23, 2024 from https://learn.microsoft.com/enus/ dotnet/orleans/grains/grain-placement#random-placement
[51]
Microsoft Orleans. 2024. Orleans streams implementation details. Microsoft. Retrieved April 8, 2024 from https: //learn.microsoft.com/en-us/dotnet/orleans/implementation/streams-implementation/
[52]
Microsoft Orleans. 2024. Streaming with Orleans. Microsoft. Retrieved March 24, 2024 from https://learn.microsoft. com/en-us/dotnet/orleans/streaming/?pivots=orleans-7-0
[53]
Dan Pritchett. 2008. Base: An Acid Alternative. In File Systems and Storage, Vol. 6. ACM Queue. Issue 3.
[54]
Nitin Sarma. 2020. Event Driven Architectures Netflix Content Finance Engineering. Netflix. Retrieved April 8, 2024 from https://www.linkedin.com/pulse/event-driven-architectures-netflix-content-finance-engineering-s/
[55]
Vivek Shah and Marcos Antonio Vaz Salles. 2018. Reactors: A Case for Predictable, Virtualized Actor Database Systems. In Proceedings of the 2018 International Conference on Management of Data (Houston, TX, USA) (SIGMOD '18). Association for Computing Machinery, New York, NY, USA, 259--274. https://doi.org/10.1145/3183713.3183752
[56]
Natan Silnitsky. 2022. Event Driven Architecture - 5 Pitfalls to Avoid. Wix. Retrieved July 1, 2023 from https: //medium.com/wix-engineering/event-driven-architecture-5-pitfalls-to-avoid-b3ebf885bdb1
[57]
Natan Silnitsky. 2022. Troubleshooting Kafka for 2000 Microservices at Wix. Wix. Retrieved July 1, 2023 from https://medium.com/wix-engineering/troubleshooting-kafka-for-2000-microservices-at-wix-986ee382fd1e
[58]
Aakriti Singla and Simon Wu. 2020. Revolutionizing Money Movements at Scale with Strong Data Consistency. Uber Technologies Inc. Retrieved April 8, 2024 from https://eng.uber.com/money-scale-strong-data
[59]
Apache Flink Statefun. 2023. Co-located Functions. Retrieved November 25, 2023 from https://nightlies.apache.org/ flink/flink-statefun-docs-master/docs/concepts/distributed_architecture/#co-located-functions
[60]
Apache Flink Statefun. 2023. Distributed Architecture. Retrieved November 25, 2023 from https://nightlies.apache.org/ flink/flink-statefun-docs-master/docs/concepts/distributed_architecture
[61]
Apache Flink Statefun. 2023. Shopping Cart Example with Docker Compose. Retrieved November 25, 2023 from https://github.com/apache/flink-statefun-playground/tree/main/java/shopping-cart
[62]
Apache Flink Statefun. 2023. Stateful Functions: A Platform-Independent Stateful Serverless Stack. Retrieved November 11, 2023 from https://nightlies.apache.org/flink/flink-statefun-docs-master/
[63]
Statista. 2024. eCommerce - Worldwide. Retrieved March 18, 2024 from https://www.statista.com/outlook/emo/ ecommerce/worldwide
[64]
Jan Stenberg. 2019. Experiences Moving from Microservices to Workflows at Jet.com. Jet.com. Retrieved March 21, 2024 from https://www.infoq.com/news/2019/02/migrate-microservices-workflows
[65]
Rebecca Taft, Nosayba El-Sayed, Marco Serafini, Yu Lu, Ashraf Aboulnaga, Michael Stonebraker, Ricardo Mayerhofer, and Francisco Andrade. 2018. P-Store: An Elastic Database System with Predictive Provisioning. In Proceedings of the 2018 International Conference on Management of Data (Houston, TX, USA) (SIGMOD '18). Association for Computing Machinery, New York, NY, USA, 205--219. https://doi.org/10.1145/3183713.3190650
[66]
UCloud. [n.d.]. Products - UCloud. Retrieved November 28, 2023 from https://docs.cloud.sdu.dk/guide/resourcesproducts. html#compute
[67]
UCloud. [n.d.]. UCloud User Guide. Retrieved November 28, 2023 from https://docs.cloud.sdu.dk
[68]
Nicolas Viennot, Mathias Lécuyer, Jonathan Bell, Roxana Geambasu, and Jason Nieh. 2015. Synapse: A Microservices Architecture for Heterogeneous-Database Web Applications. In Proceedings of the Tenth European Conference on Computer Systems (Bordeaux, France) (EuroSys '15). Association for Computing Machinery, New York, NY, USA, Article 21, 16 pages. https://doi.org/10.1145/2741948.2741975
[69]
VTEX. [n.d.]. Commerce Microservices. VTEX. Retrieved March 18, 2024 from https://vtex.com/us-en/commercemicroservices/
[70]
Yiwen Wang, Júlio Cesar dos Reis, Kasper Myrtue Borggren, Marcos Antonio Vaz Salles, Claudia Bauzer Medeiros, and Yongluan Zhou. 2019. Modeling and Building IoT Data Platforms with Actor-Oriented Databases. In Advances in Database Technology - 22nd International Conference on Extending Database Technology, EDBT 2019, Lisbon, Portugal, March 26--29, 2019. OpenProceedings.org, 512--523. https://doi.org/10.5441/002/edbt.2019.47
[71]
Yang Yang, Zhifeng Chen, Qichao Chu, Haitao Zhang, and George Teo. 2021. Enabling Seamless Kafka Async Queuing with Consumer Proxy. Uber Technologies Inc. Retrieved April 8, 2024 from https://www.uber.com/en-SE/blog/kafkaasync-queuing-with-consumer-proxy/
[72]
Chao Zhang, Guoliang Li, and Tao Lv. 2024. HyBench: A New Benchmark for HTAP Databases. Proc. VLDB Endow. 17, 5 (May 2024), 939--951. https://doi.org/10.14778/3641204.3641206
[73]
Chao Zhang, Jiaheng Lu, Pengfei Xu, and Yuxing Chen. 2019. UniBench: A Benchmark for Multi-model Database Management Systems: Recognizing Outstanding Ph.D. Research. 7--23. https://doi.org/10.1007/978--3-030--11404--6_2
[74]
Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. 2021. Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study. IEEE Trans. Softw. Eng. 47, 2 (feb 2021), 243--260. https://doi.org/10.1109/TSE.2018.2887384
[75]
Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chenjie Xu, Chao Ji, and Wenyun Zhao. 2018. Benchmarking microservice systems for software engineering research. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 323--324. https://doi.org/10.1145/3183440.3194991
Index Terms
- Online Marketplace: A Benchmark for Data Management in Microservices
Recommendations
A choice of selling format in the online marketplace with cross-sales supply chain: Platform selling or traditional reselling?
AbstractIn the online retail market, how to work with upstream suppliers is a key issue for downstream online retailers (e-retailers). Online retailers can choose between functioning either as the “two-sided platforms” (e.g., Taobao.com or eBay.com) ...
An empirical test to measure the effectiveness of online advertising in online marketplaces using a hierarchical Bayes model
Online marketplace, taken the form of ''open market'' where a very large number of buyers and sellers participate, has occupied a rapid increasing position in e-commerce, which resulting in sellers' increasing investment on online advertising. Hence, ...
How to measure the effectiveness of online advertising in online marketplaces
The online marketplace, in the form of an ''open market'' where a very large number of buyers and sellers participate, has occupied a rapidly increasing position in e-commerce, resulting in sellers' increasing investment in online advertising. Hence, ...
Comments
Information & Contributors
Information
Published In
February 2025
2261 pages
Copyright © 2025.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 February 2025
Published in PACMMOD Volume 3, Issue 1
Author Tags
Qualifiers
- Research-article
Funding Sources
- Independent Research Fund Denmark
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 2Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)2
Reflects downloads up to 11 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in