research-article

DualDB: An Efficient LSM-based Publish/Subscribe Storage System

Authors:

Mohiuddin Abdul Qader,

Vagelis HristidisAuthors Info & Claims

SSDBM '17: Proceedings of the 29th International Conference on Scientific and Statistical Database Management

Article No.: 24, Pages 1 - 6

https://doi.org/10.1145/3085504.3085528

Published: 27 June 2017 Publication History

Get Access

Abstract

Publish/Subscribe systems allow subscribers to monitor for events of interest generated by publishers. Current publish/subscribe query systems are efficient when the subscriptions (queries) are relatively static -- for instance, the set of followers in Twitter -- or can fit in memory. However, an increasing number of applications in this era of Big Data and Internet of Things (IoT) are based on a highly dynamic query paradigm, where continuous queries are in the millions and are created and expire in a rate comparable, or even higher, to that of the data (event) entries. For instance moving objects like airplanes, cars or sensors may continuously generate measurement data like air pressure or traffic, which are consumed by other moving objects.

In this paper we propose and compare a novel publish/subscribe storage architecture, DualDB, based on the popular NoSQL Log-Structured Merge Tree (LSM) storage paradigm, to support high-throughput and dynamic publish/subscribe systems. Our method naturally supports queries on both past and future data, and generate instant notifications, which are desirable properties missing from many previous systems. We implemented and experimentally evaluated our methods on the popular LSM-based LevelDB system, using real datasets. Our results show that we can achieve significantly higher throughput compared to state-of-the-art baselines.

References

[1]

Sattam Alsubaiee, Alexander Behm, Vinayak Borkar, Zachary Heilbron, Young-Seok Kim, Michael J Carey, Markus Dreseler, and Chen Li. 2014. Storage Management in AsterixDB. Proceedings of the VLDB Endowment 7, 10 (2014).

Digital Library

Google Scholar

[2]

Michael J Carey, Steven Jacobs, and Vassilis J Tsotras. 2016. Breaking BAD: a data serving vision for big active data. In Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems. ACM, 181--186.

Digital Library

Google Scholar

[3]

Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C Hsieh, Deborah A Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E Gruber. 2008. Bigtable: A distributed storage system for structured data. TOCS 26, 2 (2008), 4.

Digital Library

Google Scholar

[4]

Jianjun Chen, David J DeWitt, Feng Tian, and Yuan Wang. 2000. NiagaraCQ: A scalable continuous query system for internet databases. In ACM SIGMOD Record, Vol. 29. ACM, 379--390.

Digital Library

Google Scholar

[5]

Patrick Th Eugster, Pascal A Felber, Rachid Guerraoui, and Anne-Marie Kermarrec. 2003. The many faces of publish/subscribe. ACM Computing Surveys (CSUR) 35, 2 (2003), 114--131.

Digital Library

Google Scholar

[6]

Eli Fidler, Hans-Arno Jacobsen, Guoli Li, and Serge Mankovski. 2005. The PADRES Distributed Publish/Subscribe System. In FIW. 12--30.

Google Scholar

[7]

Google Inc. 2017. LevelDB. http://leveldb.org/. (Feb 2017).

Google Scholar

[8]

Hans-Arno Jacobsen, Vinod Muthusamy, and Guoli Li. 2009. The PADRES Event Processing Network: Uniform Querying of Past and Future EventsDas PADRES Ereignisverarbeitungsnetzwerk: Einheitliche Anfragen auf Ereignisse der Vergangenheit und Zukunft. it-Information Technology Methoden und innovative Anwendungen der Informatik und Informationstechnik 51, 5 (2009), 250--260.

Google Scholar

[9]

Avinash Lakshman and Prashant Malik. 2010. Cassandra: A Decentralized Structured Storage System. SIGOPS Oper. Syst. Rev. 44, 2 (apr 2010), 35--40.

Digital Library

Google Scholar

[10]

Jennifer Widom and Sheldon J Finkelstein. 1990. Set-oriented production rules in relational database systems. In ACM SIGMOD Record, Vol. 19. ACM, 259--270.

Digital Library

Google Scholar

Cited By

View all

Silva DHolanda M(2021)Applications of geospatial big data in the Internet of ThingsTransactions in GIS10.1111/tgis.1284626:1(41-71)Online publication date: 24-Sep-2021
https://doi.org/10.1111/tgis.12846
Li FLu YYang ZShu J(2020)SineKV: Decoupled Secondary Indexing for LSM-based Key-Value Stores2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS47774.2020.00071(1112-1122)Online publication date: Nov-2020
https://doi.org/10.1109/ICDCS47774.2020.00071
Jacobs SWang XCarey MTsotras VUddin M(2020)BAD to the bone: Big Active Data at its coreThe VLDB Journal10.1007/s00778-020-00616-7Online publication date: 23-May-2020
https://doi.org/10.1007/s00778-020-00616-7
Show More Cited By

Index Terms

DualDB: An Efficient LSM-based Publish/Subscribe Storage System
1. Information systems
  1. Data management systems
    1. Database management system engines
  2. Information systems applications
    1. Spatial-temporal systems
      1. Location based services

Recommendations

High-throughput publish/subscribe on top of LSM-based storage

State-of-the-art publish/subscribe systems are efficient when the subscriptions are relatively static--for instance, the set of followers in Twitter--or can fit in memory. However, now-a-days, many big data and IoT based applications follow a highly ...
WiscKey: Separating Keys from Values in SSD-Conscious Storage
Special Issue on USENIX FAST 2016 and Regular Papers

We present WiscKey, a persistent LSM-tree-based key-value store with a performance-oriented data layout that separates keys from values to minimize I/O amplification. The design of WiscKey is highly SSD optimized, leveraging both the sequential and ...
An efficient design and implementation of LSM-tree based key-value store on open-channel SSD
EuroSys '14: Proceedings of the Ninth European Conference on Computer Systems

Various key-value (KV) stores are widely employed for data management to support Internet services as they offer higher efficiency, scalability, and availability than relational database systems. The log-structured merge tree (LSM-tree) based KV stores ...

Comments

Information & Contributors

Information

Published In

SSDBM '17: Proceedings of the 29th International Conference on Scientific and Statistical Database Management

June 2017

373 pages

ISBN:9781450352826

DOI:10.1145/3085504

General Chair:
Alok Choudhary,
Program Chair:
Kesheng Wu,
Publications Chair:
Bin Dong

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

Northwestern University: Northwestern University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SSDBM '17

SSDBM '17: 29th International Conference on Scientific and Statistical Database Management

June 27 - 29, 2017

IL, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 56 of 146 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
104
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Silva DHolanda M(2021)Applications of geospatial big data in the Internet of ThingsTransactions in GIS10.1111/tgis.1284626:1(41-71)Online publication date: 24-Sep-2021
https://doi.org/10.1111/tgis.12846
Li FLu YYang ZShu J(2020)SineKV: Decoupled Secondary Indexing for LSM-based Key-Value Stores2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS47774.2020.00071(1112-1122)Online publication date: Nov-2020
https://doi.org/10.1109/ICDCS47774.2020.00071
Jacobs SWang XCarey MTsotras VUddin M(2020)BAD to the bone: Big Active Data at its coreThe VLDB Journal10.1007/s00778-020-00616-7Online publication date: 23-May-2020
https://doi.org/10.1007/s00778-020-00616-7
Soares NVairiinhos MBeça PRibeiro T(2019)Real-Time NotificationsSmart Marketing With the Internet of Things10.4018/978-1-5225-5763-0.ch006(114-122)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-5763-0.ch006
Qader MHristidis V(2019)High-throughput publish/subscribe on top of LSM-based storageDistributed and Parallel Databases10.1007/s10619-018-7236-237:1(101-132)Online publication date: 1-Mar-2019
https://dl.acm.org/doi/10.1007/s10619-018-7236-2
Qader MCheng SHristidis VDas GJermaine CBernstein P(2018)A Comparative Study of Secondary Indexing Techniques in LSM-based NoSQL DatabasesProceedings of the 2018 International Conference on Management of Data10.1145/3183713.3196900(551-566)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3183713.3196900

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

High-throughput publish/subscribe on top of LSM-based storage

WiscKey: Separating Keys from Values in SSD-Conscious Storage

An efficient design and implementation of LSM-tree based key-value store on open-channel SSD

Comments

Published In

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

High-throughput publish/subscribe on top of LSM-based storage

WiscKey: Separating Keys from Values in SSD-Conscious Storage

An efficient design and implementation of LSM-tree based key-value store on open-channel SSD

Comments

Information

Published In

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations