Article

Position: short object lifetimes require a delete-optimized storage system

Authors:

Elizabeth S. Richards,

William H. Tetzlaff,

John M. Tracey,

Jian YinAuthors Info & Claims

EW 11: Proceedings of the 11th workshop on ACM SIGOPS European workshop

Pages 6 - es

https://doi.org/10.1145/1133572.1133593

Published: 19 September 2004 Publication History

Abstract

Early file systems were designed with the expectation that data would typically be read from disk many times before being deleted; on-disk structures were therefore optimized for reading. As main memory sizes increased, more read requests could be satisfied from data cached in memory, motivating file system designs that optimize write performance. Here, we describe how one might build a storage system that optimizes not only reading and writing, but creation and deletion as well. Efficiency is achieved, in part, by automating deletion based on relative retention values rather than requiring data be deleted explicitly by an application. This approach is well suited to an emerging class of applications that process data at consistently high rates of ingest. This paper explores trade-offs in clustering data by retention value and age and examines the effects of allowing the retention values to change under application control.

References

[1]

T. Blackwell, J. Harris, and M. Seltzer. Heuristic Cleaning Algorithms in Log-Structured File Systems. In Proceedings of the Winter 1995 USENIX Conference, January 1995.]]

Digital Library

[2]

S. Chandrasekaran et al. TelegraphCQ: Continuous dataflow processing for an uncertain world. In Proceedings of the First Biennial Conference on Innovative Data Systems Research (CIDR), 2003.]]

[3]

A. Gärtner, A. Kemper, D. Kossmann, and B. Zeller. Efficient bulk deletes in relational databases. In Proceedings of the 17th International Conference on Data Engineering, Heidelberg, Germany, April 2001. IEEE.]]

Digital Library

[4]

Bezalel Gavish and Olivia R. Liu Sheng. Dynamic file migration in distributed computer systems. Communications of the ACM, 33(2):177--189, 1990.]]

Digital Library

[5]

Raxco Software, Inc. A tutorial on disk defragmentation for windows nt/2000/xp and windows server 2003. http://www.raxco.com/products/perfectdisk2k/whitepapers/defrag_tutorial%.pdf, May 2003.]]

[6]

Sitaram Iyer and Peter Druschel. Anticipatory scheduling: A disk scheduling framework to overcome deceptive idleness in synchronous I/O. In Symposium on Operating Systems Principles, pages 117--130, 2001.]]

Digital Library

[7]

Marshall K. McKusick et al. A fast file system for Unix. ACM Transactions on Computer Systems, 2(3):181--197, August 1984.]]

Digital Library

[8]

J. Menon and L. Stockmeyer. An age-threshold algorithm for garbage collection in log-structured arrays and file systems. IBM Research Report RJ 10120, 1998.]]

[9]

J. Ousterhout and F. Douglis. Beating the I/O bottleneck: A case for log-structured file systems. ACM Operating Systems Review, 23(1):11--28, January 1989.]]

Digital Library

[10]

R. Patterson, G. Gibson, E. Ginting, D. Stodolsky, and J. Zelenka. Informed Prefetching and Caching. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, pages 79--95, December 1995.]]

Digital Library

[11]

M. Rosenblum and J. Ousterhout. The design and implementation of a log-structured file system. ACM Transactions on Computer Systems, 10(1):26--52, February 1992.]]

Digital Library

[12]

Douglas S. Santry, Michael J. Feeley, Norman C. Hutchinson, Alistair C. Veitch, Ross W. Carton, and Jacob Ofir. Deciding when to forget in the elephant file system. In Symposium on Operating Systems Principles, pages 110--123, 1999.]]

Digital Library

[13]

M. Seltzer, K. Bostic, M. McKusick, and C. Staelin. An Implementation of a Log-Structured File System for UNIX. In Proceedings of the Winter 1993 USENIX Conference, pages 307--326, January 1993.]]

Digital Library

[14]

Jun Wang and Yiming Hu. Wolf -- a novel reordering write buffer to boost the performance of log-structured file systems. In Proceedings of the First USENIX Conference on File and Storage Technologies, Monterey, CA, 2002.]]

Digital Library

[15]

J. Wilkes, R. Golding, C. Staelin, and T. Sullivan. The HP autoraid hierarchical storage system. In Proceedings of the 15th Symposium on Operating Systems Principles, December 1995.]]

Digital Library

[16]

Stan Zdonik et al. The aurora and medusa projects. Bulletin of the Technical Committee on Data Engineering, pages 3--10, March 2003.]]

Cited By

Agrawal NVulimiri A(2017)Low-Latency Analytics on Colossal Data Streams with SummaryStoreProceedings of the 26th Symposium on Operating Systems Principles10.1145/3132747.3132758(647-664)Online publication date: 14-Oct-2017
https://dl.acm.org/doi/10.1145/3132747.3132758
Khandekar RHildrum KParekh SRajan DWolf JWu KAndrade HGedik B(2009)COLAProceedings of the ACM/IFIP/USENIX 10th international conference on Middleware10.5555/1813355.1813377(308-327)Online publication date: 30-Nov-2009
https://dl.acm.org/doi/10.5555/1813355.1813377
Khandekar RHildrum KParekh SRajan DWolf JWu KAndrade HGedik BCampbell RBacon JCooper B(2009)COLAProceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware10.5555/1656980.1657002(1-20)Online publication date: 30-Nov-2009
https://dl.acm.org/doi/10.5555/1656980.1657002
Show More Cited By

Position: short object lifetimes require a delete-optimized storage system
1. Information systems
  1. Information retrieval
    1. Specialized information retrieval
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems

Recommendations

Flash-Aware RAID Techniques for Dependable and High-Performance Flash Memory SSD

Solid-state disks (SSDs), which are composed of multiple NAND flash chips, are replacing hard disk drives (HDDs) in the mass storage market. The performances of SSDs are increasing due to the exploitation of parallel I/O architectures. However, ...
Ziggurat: a tiered file system for non-volatile main memories and disks
FAST'19: Proceedings of the 17th USENIX Conference on File and Storage Technologies

Emerging fast, byte-addressable Non-Volatile Main Memory (NVMM) provides huge increases in storage performance compared to traditional disks. We present Ziggurat, a tiered file system that combines NVMM and slow disks to create a storage system with ...
HPDA: A hybrid parity-based disk array for enhanced performance and reliability

Flash-based Solid State Drive (SSD) has been productively shipped and deployed in large scale storage systems. However, a single flash-based SSD cannot satisfy the capacity, performance and reliability requirements of the modern storage systems that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EW 11: Proceedings of the 11th workshop on ACM SIGOPS European workshop

September 2004

214 pages

ISBN:9781450378079

DOI:10.1145/1133572

General Chair:
Yolande Berbers
K. U. Leuven, Leuven (Heverlee), Belgium
,
Program Chair:
Miguel Castro
Microsoft Research Ltd., Cambridge, UK

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

EW04

Sponsor:

SIGOPS

EW04: ACM SIGOPS European Workshop 2004

September 19 - 22, 2004

Leuven, Belgium

Acceptance Rates

EW 11 Paper Acceptance Rate 37 of 37 submissions, 100%;

Overall Acceptance Rate 37 of 37 submissions, 100%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
318
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Agrawal NVulimiri A(2017)Low-Latency Analytics on Colossal Data Streams with SummaryStoreProceedings of the 26th Symposium on Operating Systems Principles10.1145/3132747.3132758(647-664)Online publication date: 14-Oct-2017
https://dl.acm.org/doi/10.1145/3132747.3132758
Khandekar RHildrum KParekh SRajan DWolf JWu KAndrade HGedik B(2009)COLAProceedings of the ACM/IFIP/USENIX 10th international conference on Middleware10.5555/1813355.1813377(308-327)Online publication date: 30-Nov-2009
https://dl.acm.org/doi/10.5555/1813355.1813377
Khandekar RHildrum KParekh SRajan DWolf JWu KAndrade HGedik BCampbell RBacon JCooper B(2009)COLAProceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware10.5555/1656980.1657002(1-20)Online publication date: 30-Nov-2009
https://dl.acm.org/doi/10.5555/1656980.1657002
Khandekar RHildrum KParekh SRajan DWolf JWu KAndrade HGedik B(2009)COLA: Optimizing Stream Processing Applications via Graph PartitioningMiddleware 200910.1007/978-3-642-10445-9_16(308-327)Online publication date: 2009
https://doi.org/10.1007/978-3-642-10445-9_16
Wolf JBansal NHildrum KParekh SRajan DWagle RWu KFleischer LJoosen WBerbers Y(2008)SODAProceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware10.5555/1496950.1496970(306-325)Online publication date: 2-Dec-2008
https://dl.acm.org/doi/10.5555/1496950.1496970
Hildrum KDouglis FWolf JYu PFleischer LKatta A(2008)Storage optimization for large-scale distributed stream-processing systemsACM Transactions on Storage10.1145/1326542.13265473:4(1-28)Online publication date: 25-Feb-2008
https://dl.acm.org/doi/10.1145/1326542.1326547
Palpanas TChaudhry JAndritsos PVelegrakis Y(2008)Entity Data Management in OKKAMProceedings of the 2008 19th International Conference on Database and Expert Systems Application10.1109/DEXA.2008.13(729-733)Online publication date: 1-Sep-2008
https://dl.acm.org/doi/10.1109/DEXA.2008.13
Wolf JBansal NHildrum KParekh SRajan DWagle RWu KFleischer L(2008)SODAProceedings of the ACM/IFIP/USENIX 9th International Middleware Conference10.1007/978-3-540-89856-6_16(306-325)Online publication date: 1-Dec-2008
https://dl.acm.org/doi/10.1007/978-3-540-89856-6_16
Chandra SGehani AYu X(2007)Diamonds are forever, files are notProceedings of the 5th USENIX conference on File and Storage Technologies10.5555/1267903.1267913(10-10)Online publication date: 13-Feb-2007
https://dl.acm.org/doi/10.5555/1267903.1267913
Hildrum KDouglis FWolf JYu PFleischer LKatta A(2007)Storage Optimization for Large-Scale Distributed Stream Processing Systems2007 IEEE International Parallel and Distributed Processing Symposium10.1109/IPDPS.2007.370633(1-8)Online publication date: Mar-2007
https://doi.org/10.1109/IPDPS.2007.370633
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten