Article

Free access

Analysis of file I/O traces in commercial computing environments

Authors:

K. K. Ramakrishnan,

Prabuddha Biswas,

Ramakrishna KaredlaAuthors Info & Claims

SIGMETRICS '92/PERFORMANCE '92: Proceedings of the 1992 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems

Pages 78 - 90

https://doi.org/10.1145/133057.133090

Published: 01 June 1992 Publication History

Abstract

Improving the performance of the file system is becoming increasingly important to alleviate the effect of I/O bottlenecks in computer systems. To design changes to an existing file system or to architect a new file system it is important to understand current usage patterns. In this paper we analyze file I/O traces of several existing production computer sytems to understand file access behavior.

Our analysis suggests that a relatively small percentage of the files are active. The amount of total data active is also quite small for interactive environments. An average file encounters a relatively small number of file opens while receiving an order of magnitude larger number of reads to it. An average process opens quite a large number of files over a typical prime time period. What is more significant is that the effect of outliers on many of the characteristics we studied is dominant. A relatively small number of processes dominate the activity, and a very small number of files receive most of these operations.

In addition, we provide a comprehensive analysis of the dynamic sharing of files in each of these enviroments, addressing both the simultaneous and sequential sharing aspects, and the activity to these shared files. We observe that although only a third of the active files are sequentially shared, they receive a very large proportion of the total operations. We analyze the traces from a given environment across different lengths of time, such as one hour, three hour and whole work-day intervals and do this for 3 different environments. This gives us an idea of the shortest length of the trace needed to have confidence in the estimation of the parameters.

References

[1]

Biswas, P., Ramakdshnan, K.K., "File Access Characterization of VAX/VMS Environments," Proceedings of the 10th International Conference on Distributed Computing Systems, Paris, France, May 1990.

[2]

Bodnarchuk, R., Bunt, R., "A Synthetic Workload Model for a Distributed Systems File Server," Proceedings of the 1991 ACM Sigmetrics Conference on Measurement and Modeling of Computer Systems, Vol. 19, No. 1, May 1991.

Digital Library

[3]

Floyd, R., "Short-Term File Reference Patterns in a Unix Environment," Technical Report TR-177, Department of Computer Science, University of Rochester, 1986.

[4]

Gray, C.G., Cheriton, D.R., "Leases: An Efficient Fault- Tolerant Mechanism for Distributed File Cache Consistency," Proceedings of the 12th ACM Symposium on Operating Systems Principles, Litchfield Park, December 1989.

Digital Library

[5]

Howard, J.H., Kazar, M.L., Menees, S.G., Nichols, D.A., Satyanarayanan, M., Sidebotham, R.N., West, M.J., "Scale and Performance in a Distributed File System," ACM Transactions on Computer Systems 6(1), February 1988.

Digital Library

[6]

Jain, R., Turner, R., "Workload Characterization using Image Accounting," Proceedings of the Computer Performance Evaluation Users Group 18th Meeting, Washington D.C., October 1982.

[7]

Kronenberg, N., Levy, H., Strecker, W., "VAXclusters: A Closely-coupled Distributed System," ACM Transactions on Computer Systems, Vol. 4, No. 2, May 1986.

Digital Library

[8]

Kure, O., "Optimization of File Migration in Distributed Systems," Ph.D. thesis, Computer Science Division, University of California, Berkeley, CA, Report No. UCB/CSD/88/413, April 1988.

Digital Library

[9]

Lazowska, E., Zahorjan, J., Cheriton, D., Zwaenepoel, W., "File Access Performance of Diskless Workstations," ACM Transactions on Computer Systems 4(2), August 1986.

Digital Library

[10]

Majumdar, S., Bunt, R., "Measurement and Analysis of Locality Phases in File Referencing Behaviour", Proceedings of the Performance '86 and ACM Sigmetrics '86, Joint Conference on Computer Performance Modeling, Measurement and Evaluation, Raleigh, N.C., May 1986.

Digital Library

[11]

McCoy, K., "VMS File Systems Internals", Digital Press, Bedford, Mass., t 990.

Digital Library

[12]

McKusick, M.K., Joy, W.M., Leffier, S.J., Fabry, R.S., "A Fast File System for Unix," ACM Transactions on Computer Systems 2(3), 1984.

Digital Library

[13]

Morris, J.H., Satyanarayanan, M., Conner, M.H., Howard, J.H., Rosenthal, D.S., Smith, F.D., "Andrew: A Distributed Personal Computing Environment," Communications of the ACM 29(3), March 1986.

Digital Library

[14]

Murthy, V.S. Devarakonda, "File Usage Analysis and Resource Usage Prediction: A Measurement-Based Study", Ph.D. thesis, Department of Computer Science, University of Illinois at Urbana-Champaign, 1988.

Digital Library

[15]

Nelson, M.N., Welch, B.B., Ousterhout, J.K., "Caching in the Sprite Network File System," ACM Transactions on Computer Systems 6(1), February 1988.

Digital Library

[16]

Nichols, W.G., Emer, J.S., "Design and Implementation of the VAX Distributed File Service," Digital Technical Journal 9, June 1989.

[17]

Ousterhout, J.K., Da Costa, H., Harrison, D., Kunze, J., Kupfer, M., Thompson, J., "A Trace-Driven Analysis of the Unix 4.2 BSD File System," Proceeding of the 10th ACM Symposium on Operating System Principles, Orcas island, December 1985.

Digital Library

[18]

Ousterhout, J.K., Cherenson, A.R., Douglis, F., Nelson, M.N., Welch, B.B., "The Sprite Network Operating System," Computer 21 (2), February 1988.

Digital Library

[19]

Ousterhout, J.K., Douglis, F., "Beating the I/0 Bottleneck: A Case for Log Structured File Systems", Operating Systems Review, ACM, Vol. 23, No.i, January 1989.

Digital Library

[20]

Ramakrishnan, K.K., Emer, J.S., "Performance Analysis of Mass Storage Service Alternatives for Distributed Systems," IEEE Transactions on Software Engineering 15(2), February 1989.

Digital Library

[21]

Satyanarayanan, M., "A Survey of Distributed File Systems," Technical Report CMU-CS-89-116, Department of Computer Science, Carnegie Mellon University, February 1989.

[22]

Satyanarayanan, M., ",4 Study of File Size and Functional Lifetimes," Proceedings of the 8th ACM Symposium on Operating System Principles, Asilomar, December 1981.

Digital Library

[23]

Smith, A.J., "Analysis of Long-Term File Reference Patterns for Application to File Migration Algorithms," IEEE Transactions on Software Engineering 7(4), July 1981.

Digital Library

[24]

SPEC, "SPEC Benchmark Suite Release 1.0," SPEC Newsletter 2(2), Available from Waterside Associates, Fremont, CA., 1990.

Cited By

Sim HVazhkudai S(2019)Profiling the Usage of an Extreme-Scale Archival Storage System2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)10.1109/MASCOTS.2019.00050(410-422)Online publication date: Oct-2019
https://doi.org/10.1109/MASCOTS.2019.00050
Thomasian A(2014)Disk arrays with multiple RAID levelsACM SIGARCH Computer Architecture News10.1145/2641361.264136441:5(6-24)Online publication date: 18-Jun-2014
https://dl.acm.org/doi/10.1145/2641361.2641364
Thomasian A(2014)Analysis of Fork/Join and Related Queueing SystemsACM Computing Surveys10.1145/262891347:2(1-71)Online publication date: 25-Aug-2014
https://dl.acm.org/doi/10.1145/2628913
Show More Cited By

Index Terms

Analysis of file I/O traces in commercial computing environments

Recommendations

Analysis of file I/O traces in commercial computing environments

Improving the performance of the file system is becoming increasingly important to alleviate the effect of I/O bottlenecks in computer systems. To design changes to an existing file system or to architect a new file system it is important to understand ...
Multistructured file system design for high-performance computing
VM aware journaling: improving journaling file system performance in virtualization environments

Journaling file systems, which are widely used in modern operating systems, guarantee file system consistency and data integrity by logging file system updates to a journal, which is a reserved space on the storage, before the updates are written to the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMETRICS '92/PERFORMANCE '92: Proceedings of the 1992 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems

June 1992

267 pages

ISBN:0897915070

DOI:10.1145/133057

Chairman:
Michael K. Molloy
Hewlett Packard, Ft. Collins, CO
,
Editor:
Blaine D. Gaither
Amdahl Corp., Rexburg, ID

ACM SIGMETRICS Performance Evaluation Review Volume 20, Issue 1
June 1992
260 pages
ISSN:0163-5999
DOI:10.1145/149439
Editor:
Blaine D. Gaither
Ahmdahl Corp., Rexburg, IN
Issue’s Table of Contents

Copyright © 1992 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation
IFIP WG 7.3: IFIP WG 7.3

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 1992

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGMETRICS92

Sponsor:

SIGMETRICS
IFIP WG 7.3

SIGMETRICS92: ACM SIGMETRICS Conference on Measurement & Modeling of Computer Systems

June 1 - 5, 1992

Rhode Island, Newport, USA

Acceptance Rates

Overall Acceptance Rate 459 of 2,691 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

76
Total Citations
View Citations
801
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)10

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sim HVazhkudai S(2019)Profiling the Usage of an Extreme-Scale Archival Storage System2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)10.1109/MASCOTS.2019.00050(410-422)Online publication date: Oct-2019
https://doi.org/10.1109/MASCOTS.2019.00050
Thomasian A(2014)Disk arrays with multiple RAID levelsACM SIGARCH Computer Architecture News10.1145/2641361.264136441:5(6-24)Online publication date: 18-Jun-2014
https://dl.acm.org/doi/10.1145/2641361.2641364
Thomasian A(2014)Analysis of Fork/Join and Related Queueing SystemsACM Computing Surveys10.1145/262891347:2(1-71)Online publication date: 25-Aug-2014
https://dl.acm.org/doi/10.1145/2628913
Meister DKaiser JBrinkmann ACortes TKuhn MKunkel JHollingsworth J(2012)A study on data deduplication in HPC storage systemsProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/2388996.2389006(1-11)Online publication date: 10-Nov-2012
https://dl.acm.org/doi/10.5555/2388996.2389006
Thomasian A(2012)Rebuild processing in RAID5 with emphasis on the supplementary parity augmentation method[37]ACM SIGARCH Computer Architecture News10.1145/2234336.223434040:2(18-27)Online publication date: 31-May-2012
https://dl.acm.org/doi/10.1145/2234336.2234340
Meister DKaiser JBrinkmann ACortes TKuhn MKunkel J(2012)A study on data deduplication in HPC storage systemsProceedings of the 2012 International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2012.14(1-11)Online publication date: 10-Nov-2012
https://dl.acm.org/doi/10.1109/SC.2012.14
Miranda ACortes T(2012)Analyzing Long-Term Access Locality to Find Ways to Improve Distributed Storage SystemsProceedings of the 2012 20th Euromicro International Conference on Parallel, Distributed and Network-based Processing10.1109/PDP.2012.15(544-553)Online publication date: 15-Feb-2012
https://dl.acm.org/doi/10.1109/PDP.2012.15
Chen YSrinivasan KGoodson GKatz RWobber TDruschel P(2011)Design implications for enterprise storage systems via multi-dimensional trace analysisProceedings of the Twenty-Third ACM Symposium on Operating Systems Principles10.1145/2043556.2043562(43-56)Online publication date: 23-Oct-2011
https://dl.acm.org/doi/10.1145/2043556.2043562
Thomasian A(2011)Survey and analysis of disk scheduling methodsACM SIGARCH Computer Architecture News10.1145/2024716.202471939:2(8-25)Online publication date: 31-Aug-2011
https://dl.acm.org/doi/10.1145/2024716.2024719
Kim YGermain-Renaud C(2011)Characterizing E-Science File Access Behavior via Latent Dirichlet AllocationProceedings of the 2011 Fourth IEEE International Conference on Utility and Cloud Computing10.1109/UCC.2011.31(162-169)Online publication date: 5-Dec-2011
https://dl.acm.org/doi/10.1109/UCC.2011.31
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents