Article

Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices

Authors:

Qi (George) Zhao,

Abhishek Kumar,

Jun (Jim) XuAuthors Info & Claims

SIGMETRICS '05: Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems

Pages 350 - 361

https://doi.org/10.1145/1064212.1064258

Published: 06 June 2005 Publication History

Abstract

The traffic volume between origin/destination (OD) pairs in a network, known as traffic matrix, is essential for efficient network provisioning and traffic engineering. Existing approaches of estimating the traffic matrix, based on statistical inference and/or packet sampling, usually cannot achieve very high estimation accuracy. In this work, we take a brand new approach in attacking this problem. We propose a novel data streaming algorithm that can process traffic stream at very high speed (e.g., 40 Gbps) and produce traffic digests that are orders of magnitude smaller than the traffic stream. By correlating the digests collected at any OD pair using Bayesian statistics, the volume of traffic flowing between the OD pair can be accurately determined. We also establish principles and techniques for optimally combining this streaming method with sampling, when sampling is necessary due to stringent resource constraints. In addition, we propose another data streaming algorithm that estimates flow matrix, a finer-grained characterization than traffic matrix. Flow matrix is concerned with not only the total traffic between an OD pair (traffic matrix), but also how it splits into flows of various sizes. Through rigorous theoretical analysis and extensive synthetic experiments on real Internet traffic, we demonstrate that these two algorithms can produce very accurate estimation of traffic matrix and flow matrix respectively.

References

[1]

S. Bhattacharyya, C. Diot, J. Jetcheva, and N. Taft. Geographical and temporal characteristics of inter-pop flows: View from a single pop. European Transactions on Telecommunications, 2000.

[2]

J. Cao, D. Davis, S. Vander Wiel, and B. Yu. Time-varing network tomography:router link data. Journal of American Statistics Association, pages 1063--1075, 2000.

[3]

J. Carter and M. Wegman. Universal classes of hash functions. Journal of Computer and System Sciences, pages 143--154, 1979.

[4]

N. Duffield and M. Grossglauser. Trajectory sampling for direct traffic observation. IEEE transaction of Networking, pages 280--292, June 2001.

Digital Library

[5]

N. Duffield, C. Lund, and M. Thorup. Properties and prediction of flow statistics from sampled packet streams. In Proc. of ACM/SIGCOMM IMW, August 2002.

Digital Library

[6]

N. Duffield, C. Lund, and M. Thorup. Estimating flow distribution from sampled flow statistics. In Proc. of ACM SIGCOMM, August 2003.

Digital Library

[7]

C. Estan and G. Varghese. New directions in traffic measurement and a ccounting. In Proc. of ACM SIGCOMM, August 2002.

Digital Library

[8]

C. Estan and G. Varghese. Bitmap algorithms for counting active flows on high speed links. In Proc. of ACM/SIGCOMM IMC, October 2003.

Digital Library

[9]

A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, and F.True. Deriving traffic demand for operational IP networks: Methodology and experience. In Proc. of ACM SIGCOMM, August 2000.

Digital Library

[10]

D.A. Huffman. A method for the construction of minimum-redundancy codes. Proc. of I.R.E, pages 1098--1102, 1952.

[11]

S. Jaiswal, G. Iannaccone, C. Diot, J. Kurose, and D. Towsley. Measurement and classification of out-of-sequence packets in a tier-1 IP backbone. In Proc. of IEEE INFOCOM, March 2003.

[12]

A. Kumar, M. Sung, J. Xu, and J. Wang. Data streaming algorithms for efficient and accurate estimation of flow size distribution. In Proc. of ACM SIGMETRICS, 2004.

Digital Library

[13]

A. Kumar, J. Xu, J. Wang, O. Spatschek, and L. Li. Space-code bloom filter for efficient per-flow traffic measurement. In Proc. of IEEE INFOCOM, March 2004.

[14]

A. Medina, N. Taft, K. Salamatian, S. Bhattacharyya, and C. Diot. Traffic matrix estimation:existing techniques and new directions. In Proc. of ACM SIGCOMM, August 2002.

Digital Library

[15]

http://pma.nlanr.net.

[16]

A. Nucci, R. Cruz, N. Taft, and C. Diot. Design of IGP link weight changes for estimation of traffic matrices. In Proc. of IEEE INFOCOM, March 2004.

[17]

A. Pasztor and D. Veitch. PC based precision timing without GPS. In Proc. of ACM SIGMETRICS, June 2002.

Digital Library

[18]

S. Ramabhadran and G. Varghese. Efficient implementation of a statistics counter architecture. In Proc. of ACM SIGMETRICS, June 2003.

Digital Library

[19]

M. Ramakrishna, E. Fu, and E. Bahcekapili. Efficient hardware hashing functions for high performance computers. IEEE Transactions on Computers, pages 1378--1381, 1997.

Digital Library

[20]

http://www.ripe.net.

[21]

A. Snoeren, C. Partridge, L. Sanchez, C. Jones, F. Tchakountio, S. Kent, and W. Strayer. Hash-based IP traceback. In Proc. of ACM SIGCOMM, August 2001.

Digital Library

[22]

A. Soule, A. Nucci, R. Cruz, E. Leonardi, and N. Taft. How to identify and estimate the largest traffic matrix elements in a dynamic environment. In Proc. of ACM SIGMETRICS, June 2004.

Digital Library

[23]

C. Tebaldi and M. West. Bayesian inference on network traffic using link count data. Journal of American Statistics Association, pages 557--576, 1998.

[24]

Y. Vardi. Internet tomography: estimating source-destination traffic intensities from link data. Journal of American Statistics Association, pages 365--377, 1996.

[25]

K.Y. Whang, B.T. Vander-zanden, and H.M. Taylor. A linear-time probabilistic counting algorithm for database applications. IEEE transaction of Database Systems, pages 208--229, June 1990.

Digital Library

[26]

J. Xu, M. Singhal, and Joanne Degroat. A novel cache architecture to support layer-four packet classification at memory access speeds. In Proc. of IEEE INFOCOM, March 2000.

[27]

Y. Zhang, M. Roughan, N. Duffield, and A. Greenberg. Fast accurate computation of large-scale IP traffic matrices from link loads. In Proc. of ACM SIGMETRICS, June 2003.

Digital Library

[28]

Y. Zhang, M. Roughan, C. Lund, and D. Donoho. An information-theoretic approach to traffic matrix estimation. In Proc. of ACM SIGCOMM, August 2003.

Digital Library

Cited By

Queiroz WCapretz MDantas M(2020)A MapReduce Approach for Traffic Matrix Estimation in SDNIEEE Access10.1109/ACCESS.2020.30162498(149065-149076)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3016249
Martins Rda Silva Villaça RVerdi F(2020)BitMatrix: A Multipurpose Sketch for Monitoring of Multi-tenant NetworksJournal of Network and Systems Management10.1007/s10922-020-09556-7Online publication date: 30-Jul-2020
https://doi.org/10.1007/s10922-020-09556-7
Martins RVerdi FVillaca RGarcia L(2018)Using Probabilistic Data Structures for Monitoring of Multi-tenant P4-based Networks2018 IEEE Symposium on Computers and Communications (ISCC)10.1109/ISCC.2018.8538352(00204-00207)Online publication date: Jun-2018
https://doi.org/10.1109/ISCC.2018.8538352
Show More Cited By

Index Terms

Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices

Recommendations

Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices
Performance evaluation review

The traffic volume between origin/destination (OD) pairs in a network, known as traffic matrix, is essential for efficient network provisioning and traffic engineering. Existing approaches of estimating the traffic matrix, based on statistical inference ...
Robust traffic matrix estimation with imperfect information: making use of multiple data sources
SIGMETRICS '06/Performance '06: Proceedings of the joint international conference on Measurement and modeling of computer systems

Estimation of traffic matrices, which provide critical input for network capacity planning and traffic engineering, has recently been recognized as an important research problem. Most of the previous approaches infer traffic matrix from either SNMP link ...
Data streaming algorithms for efficient and accurate estimation of flow size distribution
SIGMETRICS '04/Performance '04: Proceedings of the joint international conference on Measurement and modeling of computer systems

Knowing the distribution of the sizes of traffic flows passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and accommodate new traffic demands through better ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMETRICS '05: Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems

June 2005

428 pages

ISBN:1595930221

DOI:10.1145/1064212

General Chairs:
Derek Eager
University of Saskatchewan, Canada
,
Carey Williamson
University of Calgary, Canada
,
Program Chairs:
Sem Borst
Bell Labs, USA & CWI, The Netherlands
,
John C. S. Lui
Chinese University of Hong Kong, China

ACM SIGMETRICS Performance Evaluation Review Volume 33, Issue 1
Performance evaluation review
June 2005
417 pages
ISSN:0163-5999
DOI:10.1145/1071690
Issue’s Table of Contents

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGMETRICS05

Sponsor:

SIGMETRICS05: 2005 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

June 6 - 10, 2005

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 459 of 2,691 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
653
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Queiroz WCapretz MDantas M(2020)A MapReduce Approach for Traffic Matrix Estimation in SDNIEEE Access10.1109/ACCESS.2020.30162498(149065-149076)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3016249
Martins Rda Silva Villaça RVerdi F(2020)BitMatrix: A Multipurpose Sketch for Monitoring of Multi-tenant NetworksJournal of Network and Systems Management10.1007/s10922-020-09556-7Online publication date: 30-Jul-2020
https://doi.org/10.1007/s10922-020-09556-7
Martins RVerdi FVillaca RGarcia L(2018)Using Probabilistic Data Structures for Monitoring of Multi-tenant P4-based Networks2018 IEEE Symposium on Computers and Communications (ISCC)10.1109/ISCC.2018.8538352(00204-00207)Online publication date: Jun-2018
https://doi.org/10.1109/ISCC.2018.8538352
(2018)Bibliography5G Networks10.1016/B978-0-12-812707-0.00020-6(387-393)Online publication date: 2018
https://doi.org/10.1016/B978-0-12-812707-0.00020-6
Larsson C(2018)Methods of Big Data Analytics5G Networks10.1016/B978-0-12-812707-0.00017-6(285-322)Online publication date: 2018
https://doi.org/10.1016/B978-0-12-812707-0.00017-6
Zhou YZhou YChen MChen S(2017)Persistent Spread Measurement for Big Network Data Based on Register IntersectionACM SIGMETRICS Performance Evaluation Review10.1145/3143314.307859345:1(67-67)Online publication date: 5-Jun-2017
https://dl.acm.org/doi/10.1145/3143314.3078593
Lafferty JZhai C(2017)Document Language Models, Query Models, and Risk Minimization for Information RetrievalACM SIGIR Forum10.1145/3130348.313037551:2(251-259)Online publication date: 2-Aug-2017
https://dl.acm.org/doi/10.1145/3130348.3130375
Järvelin KKekäläinen J(2017)IR evaluation methods for retrieving highly relevant documentsACM SIGIR Forum10.1145/3130348.313037451:2(243-250)Online publication date: 2-Aug-2017
https://dl.acm.org/doi/10.1145/3130348.3130374
Turtle HCroft W(2017)Inference Networks for Document RetrievalACM SIGIR Forum10.1145/3130348.313036151:2(124-147)Online publication date: 2-Aug-2017
https://dl.acm.org/doi/10.1145/3130348.3130361
Fagan J(2017)Automatic P h r a s e Indexing for Document RetrievalACM SIGIR Forum10.1145/3130348.313035551:2(51-61)Online publication date: 2-Aug-2017
https://dl.acm.org/doi/10.1145/3130348.3130355
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten