research-article

Synapse: a microservices architecture for heterogeneous-database web applications

Authors:

Nicolas Viennot,

Mathias Lécuyer,

Roxana Geambasu,

Jason NiehAuthors Info & Claims

EuroSys '15: Proceedings of the Tenth European Conference on Computer Systems

Article No.: 21, Pages 1 - 16

https://doi.org/10.1145/2741948.2741975

Published: 17 April 2015 Publication History

Abstract

The growing demand for data-driven features in today's Web applications -- such as targeting, recommendations, or predictions -- has transformed those applications into complex conglomerates of services operating on each others' data without a coherent, manageable architecture. We present Synapse, an easy-to-use, strong-semantic system for large-scale, data-driven Web service integration. Synapse lets independent services cleanly share data with each other in an isolated and scalable way. The services run on top of their own databases, whose layouts and engines can be completely different, and incorporate read-only views of each others' shared data. Synapse synchronizes these views in real-time using a new scalable, consistent replication mechanism that leverages the high-level data models in popular MVC-based Web applications to replicate data across heterogeneous databases. We have developed Synapse on top of the popular Web framework Ruby-on-Rails. It supports data replication among a wide variety of SQL and NoSQL databases, including MySQL, Oracle, PostgreSQL, MongoDB, Cassandra, Neo4j, and Elasticsearch. We and others have built over a dozen microservices using Synapse with great ease, some of which are running in production with over 450,000 users.

Supplementary Material

MP4 File (a21-sidebyside.mp4)

Download
895.00 MB

References

[1]

A. Adya, G. Cooper, D. Myers, and M. Piatek. Thialfi: A Client Notification Service for Internet-Scale Applications. In Proc. 23rd ACM Symposium on Operating Systems Principles (SOSP), Oct. 2011.

Digital Library

[2]

P. Bailis, A. Fekete, A. Ghodsi, J. M. Hellerstein, and I. Stoica. The Potential Dangers of Causal Consistency and an Explicit Solution. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC), Oct. 2012.

Digital Library

[3]

P. Bailis, A. Ghodsi, J. M. Hellerstein, and I. Stoica. Bolton Causal Consistency. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data (SIGMOD), June 2013.

Digital Library

[4]

H. Balsters and B. Haarsma. An ORM-Driven Implementation Framework for Database Federations. In R. Meersman, P. Herrero, and T. S. Dillon, editors, OTM Workshops, volume 5872 of Lecture Notes in Computer Science, pages 659--670. Springer, Nov. 2009.

Digital Library

[5]

R. Barcia, G. Hambrick, K. Brown, R. Peterson, and K. Bhogal. Persistence in the Enterprise: A Guide to Persistence Technologies. IBM Press, first edition, May 2008.

Digital Library

[6]

K. Birman, A. Schiper, and P. Stephenson. Lightweight Causal and Atomic Group Multicast. ACM Trans. Comput. Syst., 9(3): 272--314, Aug. 1991.

Digital Library

[7]

M. Bowers. Database Revolution: Old SQL, New SQL, NoSQL... Huh? https://www.marklogic.com/resources/database-revolution-old-sql-new-sql-nosql-huh/, Apr. 2013.

[8]

M. Burrows. The Chubby Lock Service for Loosely-Coupled Distributed Systems. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI), Oct. 2006.

Digital Library

[9]

A. Carzaniga, D. S. Rosenblum, and A. L. Wolf. Design and Evaluation of a Wide-Area Event Notification Service. ACM Trans. Comput. Syst., 19(3), Aug. 2001.

Digital Library

[10]

E. Cecchet, G. Candea, and A. Ailamaki. Middleware-based Database Replication: The Gaps Between Theory and Practice. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data (SIGMOD), June 2008.

Digital Library

[11]

G. K. Y. Chan, Q. Li, and L. Feng. Design and Selection of Materialized Views in a Data Warehousing Environment: A Case Study. In Proceedings of the 2nd ACM International Workshop on Data Warehousing and OLAP (DOLAP), Nov. 1999.

Digital Library

[12]

S. Chaudhuri and U. Dayal. An Overview of Data Warehousing and OLAP Technology. SIGMOD Rec., 26(1): 65--74, Mar. 1997.

Digital Library

[13]

B. F. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein, P. Bohannon, H. arno Jacobsen, N. Puz, D. Weaver, and R. Yerneni. PNUTS: Yahoo!'s hosted data serving platform. In Proc. of the International Conference on Very Large Data Bases (VLDB), Aug. 2008.

Digital Library

[14]

S. Das, C. Botev, and K. Surlaker, et.al. All Aboard the Databus! LinkedIn's Scalable Consistent Change Data Capture Platform. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC), Oct. 2012.

Digital Library

[15]

G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon's Highly Available Key-Value Store. In Proceedings of the 21st ACM Symposium on Operating Systems Principles (SOSP), Oct. 2007.

Digital Library

[16]

A. Doan, A. Y. Halevy, and Z. G. Ives. Principles of Data Integration. Morgan Kaufmann, July 2012.

Digital Library

[17]

J. Ferris and J. Clayton. Factory Girl. A library for setting up Ruby objects as test data. https://github.com/thoughtbot/factory_girl.

[18]

M. Fowler. Patterns of Enterprise Application Architecture. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, Nov. 2002.

Digital Library

[19]

A. Gordeyev. Camintejs. http://www.camintejs.com.

[20]

H. Gupta and I. S. Mumick. Selection of Views to Materialize in a Data Warehouse. IEEE Transactions on Knowledge and Data Engineering, 17(1): 24--43, Jan. 2005.

Digital Library

[21]

Java Message Service. A Java message oriented middleware API for sending messages between two or more clients. http://en.wikipedia.org/wiki/Java_Message_Service.

[22]

Kafka. A High-Throughput Distributed Messaging System. http://kafka.apache.org.

[23]

D. Kubb. Datamapper. http://datamapper.org.

[24]

O. Laadan, N. Viennot, and J. Nieh. Transparent, Lightweight Application Execution Replay on Commodity Multiprocessor Operating Systems. In Proceedings of the ACM International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS), June 2010.

Digital Library

[25]

W. Lloyd, M. J. Freedman, M. Kaminsky, and D. G. Andersen. Don't Settle for Eventual: Scalable Causal Consistency for Wide-area Storage with COPS. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP), Oct. 2011.

Digital Library

[26]

W. Lloyd, M. J. Freedman, M. Kaminsky, and D. G. Andersen. Stronger Semantics for Low-latency Geo-replicated Storage. In Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation (NSDI), Apr. 2013.

Digital Library

[27]

R. Louapre. A MongoDB to Elasticsearch Translator. https://github.com/richardwilly98/elasticsearch-river-mongodb/.

[28]

MoSQL. A MongoDB to SQL Streaming Translator. https://github.com/stripe/mosql.

[29]

S. Newman. Building Microservices -- Designing Fine-Grained Systems. O'Reilly Media, Feb. 2015.

[30]

Oracle Corporation. MySQL 5.5 Reference Manual -- Replication. http://dev.mysql.com/doc/refman/5.5/en/replication.html.

[31]

M. Patiño Martinez, R. Jiménez-Peris, B. Kemme, and G. Alonso. MIDDLE-R: Consistent Database Replication at the Middleware Level. ACM Trans. Comput. Syst., 23(4), Nov. 2005.

Digital Library

[32]

M. Perham. Sidekiq. Simple, efficient background processing for Ruby. http://sidekiq.org.

[33]

P. R. Pietzuch and J. Bacon. Hermes: A Distributed Event-Based Middleware Architecture. In Proceedings of the 22nd International Conference on Distributed Computing Systems Workshops (ICDCSW), July 2002.

Digital Library

[34]

Postgres-R. A Database Replication System for PostgreSQL. http://www.postgres-r.org.

[35]

RabbitMQ. Messaging That Just Works. http://www.rabbitmq.com.

[36]

R. Ramakrishnan and J. Gehrke. Database Management Systems, 3rd Edition. McGraw-Hill Education, Aug. 2002.

Digital Library

[37]

Red Hat Community. Hibernate. http://hibernate.org.

[38]

A. I. T. Rowstron, A.-M. Kermarrec, M. Castro, and P. Druschel. SCRIBE: The Design of a Large-Scale Event Notification Infrastructure. In Proceedings of the International COST264 Workshop on Networked Group Communication, Nov. 2001.

Digital Library

[39]

S. Sanfilippo. Redis. An Advanced Key-Value Store. http://redis.io.

[40]

Spree. Ecommerce Platform. http://spreecommerce.com.

[41]

R. Storm, G. Banavar, T. Chandra, M. Kaplan, K. Miller, B. Mukherjee, D. Sturman, and M. Ward. Gryphon: An Information Flow Based Approach to Message Brokering. In In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE), Oct. 1998.

[42]

Textalytics. Web Services for Text Analysis and Mining. https://textalytics.com/.

[43]

N. Viennot. Synapse Sources. https://github.com/nviennot/synapse.

[44]

N. Viennot, S. Nair, and J. Nieh. Transparent Mutable Replay for Multicore Debugging and Patch Validation. In Proceedings of the 18th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Mar. 2013.

Digital Library

[45]

K. Wang. A DynamoDB to Elasticsearch Translator. https://github.com/kzwang/elasticsearch-river-dynamodb.

[46]

J. Yang, K. Karlapalem, and Q. Li. Algorithms for Materialized View Design in Data Warehousing Environment. In Proceedings of the 23rd International Conference on Very Large Data Bases (VLDB), Aug. 1997.

Digital Library

[47]

ZooKeeper. Centralized Service for Maintaining Configuration Information, Naming, and Providing Distributed Synchronization. http://hadoop.apache.org/zookeeper/.

Cited By

Laigner RZhang ZLiu YGomes LZhou Y(2025)Online Marketplace: A Benchmark for Data Management in MicroservicesProceedings of the ACM on Management of Data10.1145/37096533:1(1-26)Online publication date: 11-Feb-2025
https://dl.acm.org/doi/10.1145/3709653
Nikolic LDimitrieski VCelikovic M(2024)An approach for supporting transparent acid transactions over heterogeneous data stores in microservice architecturesComputer Science and Information Systems10.2298/CSIS221210006N21:1(167-202)Online publication date: 2024
https://doi.org/10.2298/CSIS221210006N
Kazanavičius J(2024)Research on legacy monolith applications decomposition into microservice architectureundefined10.20334/2024-018-MOnline publication date: 2024
https://doi.org/10.20334/2024-018-M
Show More Cited By

Index Terms

Synapse: a microservices architecture for heterogeneous-database web applications

Recommendations

Incorporating synaptic time-dependent plasticity and dynamic synapse into a computational model of wind-up

"Wind-up", a condition related to chronic pain, is a form of plasticity in spinal dorsal horn that can be observed during electrical stimulation of pain receptors at low frequencies (0.3-3 Hz). In this paper, we present a computational model to explain ...
Spatiotemporal model of tripartite synapse with perinodal astrocytic process
Abstract
Information transfer may not be limited only to synapses. Therefore, the processes and dynamics of biological neuron-astrocyte coupling and intercellular interaction within this domain are worth investigating. Existing models of tripartite synapse ...
A synapse which can switch from inhibitory to excitatory and back

Co-release of transmitters has recently been observed at synapse terminals and can even be a combination such as glutamate and GABA. A second recent experimental finding is a short-term synaptic plasticity, which depends on postsynaptic depolarization ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EuroSys '15: Proceedings of the Tenth European Conference on Computer Systems

April 2015

503 pages

ISBN:9781450332385

DOI:10.1145/2741948

General Chair:
Laurent Réveillère
LaBRI, University of Bordeaux, France
,
Program Chairs:
Tim Harris
Oracle Labs, UK
,
Maurice Herlihy
Brown University

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

EuroSys '15

Sponsor:

SIGOPS

EuroSys '15: Tenth EuroSys Conference 2015

April 21 - 24, 2015

Bordeaux, France

Acceptance Rates

Overall Acceptance Rate 241 of 1,308 submissions, 18%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

35
Total Citations
View Citations
1,648
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)3

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Laigner RZhang ZLiu YGomes LZhou Y(2025)Online Marketplace: A Benchmark for Data Management in MicroservicesProceedings of the ACM on Management of Data10.1145/37096533:1(1-26)Online publication date: 11-Feb-2025
https://dl.acm.org/doi/10.1145/3709653
Nikolic LDimitrieski VCelikovic M(2024)An approach for supporting transparent acid transactions over heterogeneous data stores in microservice architecturesComputer Science and Information Systems10.2298/CSIS221210006N21:1(167-202)Online publication date: 2024
https://doi.org/10.2298/CSIS221210006N
Kazanavičius J(2024)Research on legacy monolith applications decomposition into microservice architectureundefined10.20334/2024-018-MOnline publication date: 2024
https://doi.org/10.20334/2024-018-M
Liu YLaigner RZhou Y(2024)Rethinking State Management in Actor Systems for Cloud-Native ApplicationsProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698540(898-914)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698540
Somashekar GDutt AAdak MLorido Botran TGandhi AChua TNgo CKa-Wei Lee RKumar RLauw H(2024)GAMMA: Graph Neural Network-Based Multi-Bottleneck Localization for Microservices ApplicationsProceedings of the ACM Web Conference 202410.1145/3589334.3645665(3085-3095)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645665
Saxena DBhowmik B(2024)Analysis of Selected Load Balancing Algorithms in Containerized Cloud Environment for Microservices2024 IEEE 4th International Conference on VLSI Systems, Architecture, Technology and Applications (VLSI SATA)10.1109/VLSISATA61709.2024.10560139(1-6)Online publication date: 17-May-2024
https://doi.org/10.1109/VLSISATA61709.2024.10560139
Karathanasis CMaikantis TNikolaidis NAmpatzoglou AChatzigeorgiou AMittas N(2024)A Semi-Automated Approach for Resolving Data-Driven Architecture Mismatches2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C63560.2024.00007(1-7)Online publication date: 4-Jun-2024
https://doi.org/10.1109/ICSA-C63560.2024.00007
Liu YYu ZYuan XKe WFang ZDu THan C(2022)Assessing Database Contribution via Distributed Tracing for Microservice SystemsApplied Sciences10.3390/app12221148812:22(11488)Online publication date: 12-Nov-2022
https://doi.org/10.3390/app122211488
Kazanavičius JMažeika DKalibatienė D(2022)An Approach to Migrate a Monolith Database into Multi-Model Polyglot Persistence Based on Microservice Architecture: A Case Study for Mainframe DatabaseApplied Sciences10.3390/app1212618912:12(6189)Online publication date: 17-Jun-2022
https://doi.org/10.3390/app12126189
Rabiu SHuah Yong CMashita Syed Mohamad S(2022)A Cloud-Based Container Microservices: A Review on Load-Balancing and Auto-Scaling IssuesInternational Journal of Data Science10.18517/ijods.3.2.80-92.20223:2(80-92)Online publication date: 7-Sep-2022
https://doi.org/10.18517/ijods.3.2.80-92.2022
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten