research-article

Bluecache: a scalable distributed flash-based key-value store

Authors:

ArvindAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 10, Issue 4

Pages 301 - 312

https://doi.org/10.14778/3025111.3025113

Published: 01 November 2016 Publication History

Abstract

A key-value store (KVS), such as memcached and Redis, is widely used as a caching layer to augment the slower persistent backend storage in data centers. DRAM-based KVS provides fast key-value access, but its scalability is limited by the cost, power and space needed by the machine cluster to support a large amount of DRAM. This paper offers a 10X to 100X cheaper solution based on flash storage and hardware accelerators. In BlueCache key-value pairs are stored in flash storage and all KVS operations, including the flash controller are directly implemented in hardware. Furthermore, BlueCache includes a fast interconnect between flash controllers to provide a scalable solution. We show that BlueCache has 4.18X higher throughput and consumes 25X less power than a flash-backed KVS software implementation on x86 servers. We further show that BlueCache can outperform DRAM-based KVS when the latter has more than 7.4% misses for a read-intensive aplication. BlueCache is an attractive solution for both rack-level appliances and data-center-scale key-value cache.

References

[1]

A Hash Function for Hash Table Lookup. http://goo.gl/VDzzLb.

[2]

Bluespec Inc. http://www.bluespec.com.

[3]

Netflix EVCache. http://goo.gl/9zoxJ6.

[4]

Nvidia GeForce GTX 780 Specifications. http://goo.gl/6Yhlv6.

[5]

Redis. http://redis.io.

[6]

Samsung 850 PRO. http://goo.gl/vjPj7V.

[7]

Samsung 950 PRO. http://goo.gl/DCwQpd.

[8]

Samsung M393A2G40DB0-CPB. http://goo.gl/BOL4ye.

[9]

Samsung DDR4 SDRAM. http://goo.gl/LO1ExG, June 2013.

[10]

D. G. Andersen, J. Franklin, M. Kaminsky, A. Phanishayee, L. Tan, and V. Vasudevan. FAWN: A Fast Array of Wimpy Nodes. In SOSP, pages 1--14, 2009.

Digital Library

[11]

B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, and M. Paleczny. Workload analysis of a large-scale key-value store. In SIGMETRICS, pages 53--64, 2012.

Digital Library

[12]

A. Badam and V. S. Pai. SSDAlloc: Hybrid SSD/RAM Memory Management Made Easy. In NSDI, pages 211--224, 2011.

Digital Library

[13]

S. Barahmand and S. Ghandeharizadeh. BG: A Benchmark to Evaluate Interactive Social Networking Actions. In CIDR, 2013.

[14]

M. A. Bender, M. Farach-Colton, R. Johnson, R. Kraner, B. C. Kuszmaul, D. Medjedovic, P. Montes, P. Shetty, R. P. Spillane, and E. Zadok. Don'T Thrash: How to Cache Your Hash on Flash. Proc. VLDB Endow., pages 1627--1637, 2012.

Digital Library

[15]

M. Berezecki, E. Frachtenberg, M. Paleczny, and K. Steele. Many-core Key-value Store. In IGCC, pages 1--8, 2011.

Digital Library

[16]

M. Blott, K. Karras, L. Liu, K. Vissers, J. Bär, and Z. István. Achieving 10Gbps Line-rate Key-value Stores with FPGAs. In Presented as part of the 5th USENIX Workshop on Hot Topics in Cloud Computing, 2013.

[17]

M. Blott, L. Liu, K. Karras, and K. Vissers. Scaling Out to a Single-Node 80Gbps Memcached Server with 40Terabytes of Memory. In HotStorage, 2015.

Digital Library

[18]

N. Bronson, Z. Amsden, G. Cabrera, P. Chakka, P. Dimov, H. Ding, J. Ferris, A. Giardullo, S. Kulkarni, H. Li, M. Marchukov, D. Petrov, L. Puzar, Y. J. Song, and V. Venkataramani. TAO: Facebook's Distributed Data Store for the Social Graph. In USENIX ATC, pages 49--60, 2013.

Digital Library

[19]

S. R. Chalamalasetti, K. Lim, M. Wright, A. AuYoung, P. Ranganathan, and M. Margala. An FPGA Memcached Appliance. In FPGA, pages 245--254, 2013.

Digital Library

[20]

F. Chen, R. Lee, and X. Zhang. Essential roles of exploiting internal parallelism of flash memory based solid state drives in high-speed data processing. In HPCA, pages 266--277, 2011.

Digital Library

[21]

T.-S. Chung, D.-J. Park, S. Park, D.-H. Lee, S.-W. Lee, and H.-J. Song. System Software for Flash Memory: A Survey. In EUC, pages 394--404, 2006.

Digital Library

[22]

B. K. Debnath, S. Sengupta, and J. Li. FlashStore: High Throughput Persistent Key-Value Store. Proc. VLDB Endow., pages 1414--1425, 2010.

Digital Library

[23]

A. Dragojević, D. Narayanan, O. Hodson, and M. Castro. FaRM: Fast Remote Memory. In NSDI, pages 401--414, 2014.

Digital Library

[24]

E. S. Fukuda, H. Inoue, T. Takenaka, D. Kim, T. Sadahisa, T. Asai, and M. Motomura. Caching memcached at reconfigurable network interface. In FPL, pages 1--6, 2014.

[25]

Fusion IO. using membrain as a flash-based cache. http://goo.gl/Khecz6, December 2011.

[26]

S. Gunelius. The Data Explosion in 2014 Minute by Minute Infographic. http://goo.gl/9CqKj5, July 2014.

[27]

T. H. Hetherington, M. O'Connor, and T. M. Aamodt. MemcachedGPU: Scaling-up Scale-out Key-value Stores. In SoCC, pages 43--57, 2015.

Digital Library

[28]

T. H. Hetherington, T. G. Rogers, L. Hsu, M. O'Connor, and T. M. Aamodt. Characterizing and Evaluating a Key-value Store Application on Heterogeneous CPU-GPU Systems. In ISPASS, pages 88--98, 2012.

Digital Library

[29]

Intel Inc. Intel Data Direct I/O Technology. http://goo.gl/2puCwN.

[30]

Intel Inc. Intel Data Plane Development Kit(Intel DPDK) Overview - Packet Processing on Intel Architecture. http://goo.gl/W5oBBV, December 2012.

[31]

J. Jose, H. Subramoni, K. Kandalla, M. Wasi-ur Rahman, H. Wang, S. Narravula, and D. K. Panda. Scalable Memcached Design for InfiniBand Clusters Using Hybrid Transports. In CCGRID, pages 236--243, 2012.

Digital Library

[32]

J. Jose, H. Subramoni, M. Luo, M. Zhang, J. Huang, M. Wasi-ur Rahman, N. S. Islam, X. Ouyang, H. Wang, S. Sur, and D. K. Panda. Memcached Design on High Performance RDMA Capable Interconnects. In ICPP, pages 743--752, 2011.

Digital Library

[33]

S.-W. Jun, M. Liu, K. E. Fleming, and Arvind. Scalable Multi-access Flash Store for Big Data Analytics. In FPGA, pages 55--64, 2014.

Digital Library

[34]

S.-W. Jun, M. Liu, S. Lee, J. Hicks, J. Ankcorn, M. King, S. Xu, and Arvind. BlueDBM: An Appliance for Big Data Analytics. In ISCA, pages 1--13, 2015.

Digital Library

[35]

S.-W. Jun, M. Liu, S. Xu, and Arvind. A transport-layer network for distributed fpga platforms. In FPL, pages 1--4, 2015.

[36]

A. Kalia, M. Kaminsky, and D. G. Andersen. Using RDMA Efficiently for Key-value Services. In SIGCOMM, pages 295--306, 2014.

Digital Library

[37]

D. Karger, E. Lehman, T. Leighton, R. Panigraphy, M. Levine, and D. Lewin. Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web. In STOC, pages 654--663, 1997.

Digital Library

[38]

M. King, J. Hicks, and J. Ankcorn. Software-Driven Hardware Development. In FPGA, pages 13--22, 2015.

Digital Library

[39]

S. Ko, S. Jun, Y. Ryu, O. Kwon, and K. Koh. A New Linux Swap System for Flash Memory Storage Devices. In ICCSA, pages 151--156, 2008.

Digital Library

[40]

M. Lavasani, H. Angepat, and D. Chiou. An FPGA-based In-Line Accelerator for Memcached. Computer Architecture Letters, pages 57--60, 2014.

Digital Library

[41]

C. Lee, D. Sim, J. Hwang, and S. Cho. F2FS: A New File System for Flash Storage. In FAST, pages 273--286, 2015.

Digital Library

[42]

S. Lee, J. Kim, and Arvind. Refactored Design of I/O Architecture for Flash Storage. Computer Architecture Letters, pages 70--74, 2015.

[43]

S. Lee, M. Liu, S. Jun, S. Xu, J. Kim, and Arvind. Application-Managed Flash. In FAST, pages 339--353, 2016.

Digital Library

[44]

S.-W. Lee, B. Moon, C. Park, J.-M. Kim, and S.-W. Kim. A case for flash memory ssd in enterprise database applications. In SIGMOD, pages 1075--1086, 2008.

Digital Library

[45]

S. Li, H. Lim, V. W. Lee, J. H. Ahn, A. Kalia, M. Kaminsky, D. G. Andersen, O. Seongil, S. Lee, and P. Dubey. Architecting to Achieve a Billion Requests Per Second Throughput on a Single Key-value Store Server Platform. In ISCA, pages 476--488, 2015.

Digital Library

[46]

H. Lim, B. Fan, D. G. Andersen, and M. Kaminsky. Silt: A memory-efficient, high-performance key-value store. In SOSP, pages 1--13, 2011.

Digital Library

[47]

H. Lim, D. Han, D. G. Andersen, and M. Kaminsky. MICA: A Holistic Approach to Fast In-memory Key-value Storage. In NSDI, pages 429--444, 2014.

Digital Library

[48]

K. Lim, D. Meisner, A. G. Saidi, P. Ranganathan, and T. F. Wenisch. Thin servers with smart pipes: Designing soc accelerators for memcached. In ISCA, pages 36--47, 2013.

Digital Library

[49]

M. Liu, S.-W. Jun, S. Lee, J. Hicks, and Arvind. minflash: A minimalistic clustered flash array. In DATE, pages 1255--1260, 2016.

Digital Library

[50]

X. Liu and K. Salem. Hybrid Storage Management for Database Systems. Proc. VLDB Endow., pages 541--552, 2013.

Digital Library

[51]

C. Mitchell, Y. Geng, and J. Li. Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store. In USENIX ATC, pages 103--114, 2013.

Digital Library

[52]

R. Nishtala, H. Fugal, S. Grimm, M. Kwiatkowski, H. Lee, H. C. Li, R. McElroy, M. Paleczny, D. Peek, P. Saab, D. Stafford, T. Tung, and V. Venkataramani. Scaling Memcache at Facebook. In NSDI, pages 385--398, 2013.

Digital Library

[53]

J. Ousterhout. RAMCloud and the Low-Latency Datacenter. http://goo.gl/uWsPnu, 2014.

[54]

J. Ousterhout, P. Agrawal, D. Erickson, C. Kozyrakis, J. Leverich, D. Mazières, S. Mitra, A. Narayanan, G. Parulkar, M. Rosenblum, S. M. Rumble, E. Stratmann, and R. Stutsman. The Case for RAMClouds: Scalable High-performance Storage Entirely in DRAM. ACM SIGOPS Operating Systems Review, pages 92--105, 2010.

Digital Library

[55]

J. Ouyang, S. Lin, S. Jiang, Z. Hou, Y. Wang, and Y. Wang. SDF: Software-defined Flash for Web-scale Internet Storage Systems. In ASPLOS, pages 471--484, 2014.

Digital Library

[56]

X. Ouyang, N. Islam, R. Rajachandrasekar, J. Jose, M. Luo, H. Wang, and D. Panda. SSD-Assisted Hybrid Memory to Accelerate Memcached over High Performance Networks. In ICPP, pages 470--479, 2012.

Digital Library

[57]

I. Petrov, G. Almeida, A. Buchmann, and U. Gräf. Building Large Storage Based On Flash Disks. In ADMS@ VLDB, 2010.

[58]

M. Rosenblum and A. N. Mario Flajslik. Low Latency RPC in RAMCloud. http://goo.gl/3FwCnU, 2011.

[59]

SanDisk. Fusion ioMemory PX600 PCIe Application Accelerators. http://goo.gl/rqePxN.

[60]

M. Saxena and M. M. Swift. FlashVM: Virtual Memory Management on Flash. In USENIX ATC, pages 14--14, 2010.

Digital Library

[61]

R. Stoica and A. Ailamaki. Improving Flash Write Performance by Using Update Frequency. Proc. VLDB Endow., pages 733--744, 2013.

Digital Library

[62]

P. Stuedi, A. Trivedi, and B. Metzler. Wimpy Nodes with 10GbE: Leveraging One-Sided Operations in Soft-RDMA to Boost Memcached. In USENIX ATC, pages 347--353, 2012.

Digital Library

[63]

Twitter Inc. Fatcache: memcache on SSD. https://github.com/twitter/fatcache.

[64]

K. Zhang, K. Wang, Y. Yuan, L. Guo, R. Lee, and X. Zhang. Mega-KV: A Case for GPUs to Maximize the Throughput of In-memory Key-value Stores. Proc. VLDB Endow., pages 1226--1237, 2015.

Digital Library

Cited By

Wong LZhang JLi J(2024)DONGLE 2.0: Direct FPGA-Orchestrated NVMe Storage for HLSACM Transactions on Reconfigurable Technology and Systems10.1145/365003817:3(1-32)Online publication date: 17-Sep-2024
https://dl.acm.org/doi/10.1145/3650038
Wang KChen F(2023)Catalyst: Optimizing Cache Management for Large In-memory Key-value SystemsProceedings of the VLDB Endowment10.14778/3625054.362506816:13(4339-4352)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.14778/3625054.3625068
Lee KJo IAhn JLee HLee HSul WJung H(2023)Deploying Computational Storage for HTAP DBMSs Takes More Than Just Computation OffloadingProceedings of the VLDB Endowment10.14778/3583140.358316116:6(1480-1493)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.14778/3583140.3583161
Show More Cited By

Recommendations

HPDA: A hybrid parity-based disk array for enhanced performance and reliability

Flash-based Solid State Drive (SSD) has been productively shipped and deployed in large scale storage systems. However, a single flash-based SSD cannot satisfy the capacity, performance and reliability requirements of the modern storage systems that ...
A multiple-file write scheme for improving write performance of small files in Fast File System

Fast File System (FFS) stores files to disk in separate disk writes, each of which incurs a disk positioning (seek + rotation) limiting the write performance for small files. We propose a new scheme called co-writing to accelerate small file writes in ...
Flash-Aware RAID Techniques for Dependable and High-Performance Flash Memory SSD

Solid-state disks (SSDs), which are composed of multiple NAND flash chips, are replacing hard disk drives (HDDs) in the mass storage market. The performances of SSDs are increasing due to the exploitation of parallel I/O architectures. However, ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 10, Issue 4

November 2016

180 pages

ISSN:2150-8097

Editors:
Peter Boncz
CWI
,
Ken Salem
University of Waterloo

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 November 2016

Published in PVLDB Volume 10, Issue 4

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
357
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wong LZhang JLi J(2024)DONGLE 2.0: Direct FPGA-Orchestrated NVMe Storage for HLSACM Transactions on Reconfigurable Technology and Systems10.1145/365003817:3(1-32)Online publication date: 17-Sep-2024
https://dl.acm.org/doi/10.1145/3650038
Wang KChen F(2023)Catalyst: Optimizing Cache Management for Large In-memory Key-value SystemsProceedings of the VLDB Endowment10.14778/3625054.362506816:13(4339-4352)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.14778/3625054.3625068
Lee KJo IAhn JLee HLee HSul WJung H(2023)Deploying Computational Storage for HTAP DBMSs Takes More Than Just Computation OffloadingProceedings of the VLDB Endowment10.14778/3583140.358316116:6(1480-1493)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.14778/3583140.3583161
Trivedi ABrunella MBaumann ACrooks NSchwarzkopf M(2023)CPU-free Computing: A Vision with a BlueprintProceedings of the 19th Workshop on Hot Topics in Operating Systems10.1145/3593856.3595906(1-14)Online publication date: 22-Jun-2023
https://dl.acm.org/doi/10.1145/3593856.3595906
Bergman SCassel NBjørling MSilberstein M(2023)ZNSwap: un-Block your SwapACM Transactions on Storage10.1145/358243419:2(1-25)Online publication date: 6-Mar-2023
https://dl.acm.org/doi/10.1145/3582434
Dann JRitter DFröning H(2023)Non-relational Databases on FPGAs: Survey, Design Decisions, ChallengesACM Computing Surveys10.1145/356899055:11(1-37)Online publication date: 9-Feb-2023
https://dl.acm.org/doi/10.1145/3568990
Koo JBae JYuk MOh SKim JPark JLee EKim BLee SFedorova ANarayanan DDi Luna GQuerzoni L(2023)All-Flash Array Key-Value Cache for Large ObjectsProceedings of the Eighteenth European Conference on Computer Systems10.1145/3552326.3567509(784-799)Online publication date: 8-May-2023
https://dl.acm.org/doi/10.1145/3552326.3567509
Hu JChen JZhu YYang QPeng ZYu Y(2023)PMEH: A Parallel and Write-Optimized Extendible Hashing for Persistent MemoryIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.327157942:11(3801-3814)Online publication date: 28-Apr-2023
https://dl.acm.org/doi/10.1109/TCAD.2023.3271579
Yu LLu YMandava MRichter EMailthody VMin SHwu WChen D(2023)FSSD: FPGA-Based Emulator for SSDs2023 33rd International Conference on Field-Programmable Logic and Applications (FPL)10.1109/FPL60245.2023.00022(101-108)Online publication date: 4-Sep-2023
https://doi.org/10.1109/FPL60245.2023.00022
Fakhry DAbdelsalam MEl-Kharashi MSafar M(2023)A review on computational storage devices and near memory computing for high performance applicationsMemories - Materials, Devices, Circuits and Systems10.1016/j.memori.2023.1000514(100051)Online publication date: Jul-2023
https://doi.org/10.1016/j.memori.2023.100051
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents