Article

Buttress: a toolkit for flexible and high fidelity I/O benchmarking

Authors:

Mahesh Kallahalla,

Ram SwaminathanAuthors Info & Claims

FAST'04: Proceedings of the 3rd USENIX conference on File and storage technologies

Page 4

Published: 31 March 2004 Publication History

Abstract

In benchmarking I/O systems, it is important to generate, accurately, the I/O access pattern that one is intending to generate. However, timing accuracy ( issuing I/Os at the desired time) at high I/O rates is difficult to achieve on stock operating systems. We currently lack tools to easily and accurately generate complex I/O workloads on modern storage systems. As a result, we may be introducing substantial errors in observed system metrics when we benchmark I/O systems using inaccurate tools for replaying traces or for producing synthetic workloads with known inter-arrival times.

In this paper, we demonstrate the need for timing accuracy for I/O benchmarking in the context of replaying I/O traces. We also quantitatively characterize the impact of error in issuing I/Os on measured system parameters. For instance, we show that the error in perceived I/O response times can be as much as+350% or-15% by using naive benchmarking tools that have timing inaccuracies. To address this problem, we present Buttress, a portable and flexible toolkit that can generate I/O workloads with microsecond accuracy at the I/O throughputs of high-end enterprise storage arrays. In particular, Buttress can issue I/O requests within 100µs of the desired issue time even at rates of 10000 I/Os per second (IOPS).

References

[1]

HP Storage Works disk array xp1024. http://www. hp.com/products1/storage/products/disk_arrays/ highend/xp1024/.

[2]

HP StorageWorks disk array xp512. http://www. hp.com/products1/storage/products/disk_arrays/ highend/xp512/.

[3]

HP StorageWorks virtual array 7400. http://www. hp.com/products1/storage/products/disk_arrays /midrange/va7400/.

[4]

A. Adya, J. Howell, M. Theimer, W.J. Bolosky, and J.R. Douceur. Cooperative task management without manual stack management or, event-driven programming is not the opposite of threaded programming. In Proceedings of the USENIX 2002 Annual Technical Conference, June 2002.

Digital Library

[5]

D. Anderson and J. Chase. Fstress: a flexible network file system benchmark. Technical Report CS-2002-01, Duke University, January 2002.

[6]

T. Bray. Bonnie benchmark. http://www.textuality.com/ bonnie, 1988.

[7]

P. Chen and D. Patterson. A new approach to I/O performance evaluation - self-scaling I/O benchmarks, predicted I/O performance. In Proc. of the ACM SIGMETRICS Conf. on Measurement and Modeling of Computer Systems, pages 1-12, May 1993.

Digital Library

[8]

Standard Performance Evaluation Corporation. SPEC SFS release 3.0 run and report rules, 2001.

[9]

Storage Performance Council. SPC-1 benchmark. http://www.storageperformance.org, 2002.

[10]

Hewlett-Packard. HP OpenMail. http://www. openmail.com/cyc/om/50/index.html.

[11]

Hewlett-Packard Company. HP SureStore E Disk Array FC60 - Advanced User's Guide, December 2000.

[12]

J.H. Howard, M.L. Kazar, S.G. Menees, D.A. Nichols, M. Satyanarayanan, R.N. Sidebotham, and M.J. West. Scale and performance in a distributed file system. ACM Trans. on Computer Systems, 6(1):51-81, February 1988.

Digital Library

[13]

IOmeter performance analysis tool. http://developer .intel.com/design/servers/devtools/iometer/.

[14]

IOzone file system benchmark. www.iozone.org, 1998.

[15]

J. Katcher. Postmark: a new file system benchmark. Technical Report TR-3022, Network Appliance, Oct 1997.

[16]

G.S. Manku, S. Rajagopalan, and B.G. Lindsay. Approximate medians and other quantiles in one pass and with limited memory. In Proc. of the 1998 ACM SIGMOD Intl. Conf. on Management of data, pages 426-435, 1998.

Digital Library

[17]

L. McVoy and C. Staelin. lmbench: portable tools for performance analysis. In Proc. Winter 1996 USENIX Technical Conference, pages 279-84, January 1996.

Digital Library

[18]

J. Ousterhout. Why Threads Are A Bad Idea (for most purposes). Invited Talk at the 1996 USENIX Technical Conference, January 1996. http://home.pacbell.net/ouster/threads.ppt.

[19]

A. Park and J.C. Becker. IOStone: a synthetic file system benchmark. Computer Architecture News, 18(2):45-52, June 1990.

Digital Library

[20]

A.D. Robison. The Abstraction Penalty for Small Objects in C++. In Parallel Object-Oriented Methods and Applications '96, Santa Fe, New Mexico, February 1996.

[21]

The transaction processing performance council. TPC Benchmark B. http://www.tpc.org/tpcb/spec/ tpcb_current.pdf, June 1994.

[22]

Tpc - transaction processing performance council. www.tpc.org, Nov 2002.

[23]

R. von Behren, J. Condit, and E. Brewer. Why events are a bad idea (for high-concurrency servers). In Proc. of the 9th Wkshp. on Hot Topics in Operating Systems (HotOS IX), pages 19-24, 2003.

Digital Library

[24]

J. Wilkes, R. Golding, C. Staelin, and T. Sullivan. The hp autoraid hierarchical storage system. In Proc 15th ACM Symposium on Operating Systems Principles (SOSP), pages 96-108, 1995.

Digital Library

[25]

B.L. Wolman and T.M. Olson. IOBENCH: a system independent IO benchmark. Computer Architecture News, 17(5):55-70, September 1989.

Digital Library

Cited By

Wortman PChandy J(2020)A trace-based study of SMB network file system workloads in an academic enterpriseProceedings of the 2020 Summer Simulation Conference10.5555/3427510.3427552(1-8)Online publication date: 20-Jul-2020
https://dl.acm.org/doi/10.5555/3427510.3427552
Haghdoost AHe WFredin JDu DKuenning GWaldspurger C(2017)On the accuracy and scalability of intensive I/O workload replayProceedings of the 15th Usenix Conference on File and Storage Technologies10.5555/3129633.3129662(315-327)Online publication date: 27-Feb-2017
https://dl.acm.org/doi/10.5555/3129633.3129662
Haghdoost AHe WFredin JDu D(2017)hfplayerACM Transactions on Storage10.1145/314939213:4(1-25)Online publication date: 15-Dec-2017
https://dl.acm.org/doi/10.1145/3149392
Show More Cited By

Recommendations

Buttress: A Toolkit for Flexible and High Fidelity I/O Benchmarking
FAST '04: Proceedings of the 3rd USENIX Conference on File and Storage Technologies

In benchmarking I/O systems, it is important to generate, accurately, the I/O access pattern that one is intending to generate. However, timing accuracy ( issuing I/Os at the desired time) at high I/O rates is difficult to achieve on stock operating ...
Mark-copy: fast copying GC with less space overhead
OOPSLA '03: Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications

Copying garbage collectors have a number of advantages over non-copying collectors, including cheap allocation and avoiding fragmentation. However, in order to provide completeness (the guarantee to reclaim each garbage object eventually), standard ...
Mark-copy: fast copying GC with less space overhead
Special Issue: Proceedings of the OOPSLA '03 conference

Copying garbage collectors have a number of advantages over non-copying collectors, including cheap allocation and avoiding fragmentation. However, in order to provide completeness (the guarantee to reclaim each garbage object eventually), standard ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

FAST'04: Proceedings of the 3rd USENIX conference on File and storage technologies

March 2004

18 pages

Sponsors

VERITAS: VERITAS
NetApp
Google Inc.
Intel: Intel

Publisher

USENIX Association

United States

Publication History

Published: 31 March 2004

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wortman PChandy J(2020)A trace-based study of SMB network file system workloads in an academic enterpriseProceedings of the 2020 Summer Simulation Conference10.5555/3427510.3427552(1-8)Online publication date: 20-Jul-2020
Haghdoost AHe WFredin JDu DKuenning GWaldspurger C(2017)On the accuracy and scalability of intensive I/O workload replayProceedings of the 15th Usenix Conference on File and Storage Technologies10.5555/3129633.3129662(315-327)Online publication date: 27-Feb-2017
Haghdoost AHe WFredin JDu D(2017)hfplayerACM Transactions on Storage10.1145/314939213:4(1-25)Online publication date: 15-Dec-2017
Chahal DVirk RNambiar MAvritzer AIosup AZhu XBecker S(2016)Performance Extrapolation of IO Intensive WorkloadsProceedings of the 7th ACM/SPEC on International Conference on Performance Engineering10.1145/2851553.2858665(105-108)Online publication date: 12-Mar-2016
Gracia-Tinedo RHarnik DNaor DSotnikov DToledo SZuck ASchindler JZadok E(2015)SDGenProceedings of the 13th USENIX Conference on File and Storage Technologies10.5555/2750482.2750506(317-330)Online publication date: 16-Feb-2015
Tarihi MAsadi HSarbazi-Azad H(2015)DiskAccelACM SIGMETRICS Performance Evaluation Review10.1145/2796314.274585643:1(297-308)Online publication date: 15-Jun-2015
Tarihi MAsadi HSarbazi-Azad HLin BXu JSengupta SShah D(2015)DiskAccelProceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems10.1145/2745844.2745856(297-308)Online publication date: 15-Jun-2015
Weiss ZHarter TArpaci-Dusseau AArpaci-Dusseau RKaminsky MDahlin M(2013)ROOTProceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles10.1145/2517349.2522734(373-387)Online publication date: 3-Nov-2013
Zheng QChen HWang YZhang JDuan JTůma PCasale GField TAmaral J(2013)COSBenchProceedings of the 4th ACM/SPEC International Conference on Performance Engineering10.1145/2479871.2479900(199-210)Online publication date: 21-Apr-2013
Kaminsky MDahlin M(2013)Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems PrinciplesundefinedOnline publication date: 3-Nov-2013
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents