research-article

Revisiting the case for a minimalist approach for network flow monitoring

Authors:

Michael K. Reiter,

Hui ZhangAuthors Info & Claims

IMC '10: Proceedings of the 10th ACM SIGCOMM conference on Internet measurement

Pages 328 - 341

https://doi.org/10.1145/1879141.1879186

Published: 01 November 2010 Publication History

Abstract

Network management applications require accurate estimates of a wide range of flow-level traffic metrics. Given the inadequacy of current packet-sampling-based solutions, several application-specific monitoring algorithms have emerged. While these provide better accuracy for the specific applications they target, they increase router complexity and require vendors to commit to hardware primitives without knowing how useful they will be to meet the needs of future applications. In this paper, we show using trace-driven evaluations that such complexity and early commitment may not be necessary. We revisit the case for a "minimalist" approach in which a small number of simple yet generic router primitives collect flow-level data from which different traffic metrics can be estimated. We demonstrate the feasibility and promise of such a minimalist approach using flow sampling and sample-and-hold as sampling primitives and configuring these in a network-wide coordinated fashion using cSamp. We show that this proposal yields better accuracy across a collection of application-level metrics than dividing the same memory resources across metric-specific algorithms. Moreover, because a minimalist approach enables late binding to what application level metrics are important, it better insulates router implementations and deployments from changing monitoring needs.

References

[1]

Flexible Netflow. http: //www.cisco.com/en/US/products/ps6965/ products_ios_protocol_option_home.html,.

[2]

Juniper cflowd. http://www.juniper.net/ techpubs/software/junos/junos91/ swconfig-policy/cflowd.html.

[3]

Narus Intercept Solution. http://www.narus.com/ index.php/solutions/intercept.

[4]

NetFlow Input Filters. http://www.cisco.com/en/US/docs/ios/12_ 3t/12_3t4/feature/guide/gtnfinpf.html.

[5]

Packet Sampling, IETFWorking Group Charter. http://www.ietf.org/html.charters/psamp/ charter/.

[6]

K. Argyraki, P. Maniatis, O. Irzak, S. Ashish, and S. Shenker. Loss and Delay Accountability for the Internet. In Proc. ICNP, 2007.

[7]

D. Brauckhoff, B. Tellenbach, A. Wagner, A. Lakhina, and M. May. Impact of Traffic Sampling on Anomaly Detection Metrics. In Proc. IMC, 2006.

Digital Library

[8]

C. Cranor, T. Johnson, O. Spatscheck, and V. Shkapenyuk. Gigascope: A stream database for network applications. In Proc. ACM SIGMOD, 2003.

Digital Library

[9]

G. R. Cantieni, G. Iannaccone, C. Barakat, C. Diot, and P. Thiran. Reformulating the Monitor Placement problem: Optimal Network-Wide Sampling. In Proc. CoNeXT, 2006.

Digital Library

[10]

L. D. Carli, Y. Pan, A. Kumar, C. Estan, and K. Sankaralingam. PLUG: Flexible Lookup Modules for Rapid Deployment of New Protocols in High-speed Routers. In Proc. SIGCOMM, 2010.

Digital Library

[11]

B. Claise. Cisco Systems NetFlow Services Export Version 9. RFC 3954.

[12]

M. P. Collins and M. K. Reiter. Finding Peer-to-Peer File-sharing using Coarse Network Behaviors. In Proc. ESORICS, 2006.

Digital Library

[13]

G. Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms, 55, 2005.

Digital Library

[14]

N. Duffield, C. Lund, and M. Thorup. Charging from sampled network usage. In Proc. IMW, 2001.

Digital Library

[15]

N. Duffield, C. Lund, and M. Thorup. Estimating Flow Distributions from Sampled Flow Statistics. In Proc. of ACM SIGCOMM, 2003.

Digital Library

[16]

C. Estan, K. Keys, D. Moore, and G. Varghese. Building a Better NetFlow. In Proc. ACM SIGCOMM, 2004.

Digital Library

[17]

C. Estan and G. Varghese. New Directions in Traffic Measurement and Accounting. In Proc. ACM SIGCOMM, 2002.

Digital Library

[18]

A. Feldmann, A. G. Greenberg, C. Lund, N. Reingold, J. Rexford, and F. True. Deriving Traffic Demands for Operational IP Networks: Methodology and Experience. In Proc. ACM SIGCOMM, 2000.

Digital Library

[19]

Google sparse hash project. http: //code.google.com/p/google-sparsehash/.

[20]

H. Song, S. Dharmapurikar, J. Turner, and J. Lockwood. Fast Hash Table Lookup Using Extended Bloom Filter: An Aid to Network Processing. In Proc. ACM SIGCOMM, 2005.

Digital Library

[21]

N. Hohn and D. Veitch. Inverting Sampled Traffic. In Proc. IMC, 2003.

Digital Library

[22]

T. Karagiannis, D. Papagiannaki, and M. Faloutsos. BLINC: Multilevel Traffic Classification in the Dark. In Proc. ACM SIGCOMM, 2005.

Digital Library

[23]

K. Keys, D. Moore, and C. Estan. A Robust System for Accurate Real-time Summaries of Internet Traffic. In Proc. SIGMETRICS, 2005.

Digital Library

[24]

R. Kompella and C. Estan. The Power of Slicing in Internet Flow Measurement. In Proc. IMC, 2005.

Digital Library

[25]

B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen. Sketch-based change detection: Methods, evaluation, and applications. In Proc. ACM IMC, 2003.

Digital Library

[26]

A. Kumar, M. Sung, J. Xu, and J. Wang. Data Streaming Algorithms for Efficient and Accurate Estimation of Flow Distribution. In Proc. ACM SIGMETRICS, 2004.

Digital Library

[27]

A. Kumar and J. Xu. Sketch Guided Sampling -- Using On-Line Estimates of Flow Size for Adaptive Data Collection. In Proc. IEEE Infocom, 2006.

[28]

A. Lakhina, M. Crovella, and C. Diot. Diagnosing Network-Wide Traffic Anomalies. In Proc. ACM SIGCOMM, 2004.

Digital Library

[29]

A. Lakhina, M. Crovella, and C. Diot. Mining anomalies using traffic feature distributions. In Proc. ACM SIGCOMM, 2005.

Digital Library

[30]

A. Lall, V. Sekar, J. Xu, M. Ogihara, and H. Zhang. Data Streaming Algorithms for Estimating Entropy of Network Traffic. In Proc. ACM SIGMETRICS, 2006.

Digital Library

[31]

Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani. Counter Braids: A Novel Counter Architecture for Per-Flow Measurement. In Proc. SIGMETRICS, 2008.

Digital Library

[32]

M. P. Collins and M. K. Reiter. Hit-listWorm Detection and Bot Identification in Large Networks Using Protocol Graphs. In Proc. RAID, 2007.

Digital Library

[33]

H. Madhyastha and B. Krishnamurthy. A Generic Language for Application-Specific Flow Sampling. ACM CCR, 38(2):7--15, Apr. 2008.

Digital Library

[34]

J. Mai, C.-N. Chuah, A. Sridharan, T. Ye, and H. Zang. Is Sampled Data Sufficient for Anomaly Detection? In Proc. IMC, 2006.

Digital Library

[35]

A. Ramachandran, S. Seetharaman, and N. Feamster. Fast Monitoring of Traffic Subpopulations. In Proc. IMC, 2008.

Digital Library

[36]

B. Ribeiro, D. Towsley, T. Ye, and J. Bolot. Fisher information of sampled packets: an application to flow size estimation. In Proc. IMC, 2006.

Digital Library

[37]

S. Kumar and P. Crowley. Segmented Hash: An Efficient Hash Table Implementation for High Performance Networking Subsystems. In Proc. ACM ANCS, 2005.

Digital Library

[38]

V. Sekar, N. Duffield, K. van der Merwe, O. Spatscheck, and H. Zhang. LADS: Large-scale Automated DDoS Detection System. In Proc. USENIX ATC, 2006.

Digital Library

[39]

V. Sekar, M. K. Reiter,W. Willinger, H. Zhang, R. Kompella, and D. G. Andersen. cSamp: A System for Network-Wide Flow Monitoring. In Proc. NSDI, 2008.

Digital Library

[40]

V. Sekar, A. Gupta, M. K. Reiter and H. Zhang. Coordinated Sampling sans Origin-Destination Identifiers: Algorithms and Analysis. In Proc. COMSNSETS, 2010.

Digital Library

[41]

S. Venkataraman, D. Song, P. B. Gibbons, and A. Blum. New Streaming Algorithms for Fast Detection of Superspreaders . In Proc. NDSS, 2005.

[42]

Y. Xie, V. Sekar, D. A. Maltz, M. K. Reiter, and H. Zhang. Worm Origin Identification Using Random Moonwalks. In Proc. IEEE Symposium on Security and Privacy, 2005.

Digital Library

[43]

K. Xu, Z.-L. Zhang, and S. Bhattacharya. Profiling Internet Backbone Traffic: Behavior Models and Applications. In Proc. ACM SIGCOMM, 2005.

Digital Library

[44]

Y. Gao, Y. Zhao, R. Schweller, S. Venkataraman, Y. Chen, D. Song and M.-Y. Kao. Detecting Stealthy Attacks Using Online Histograms. In Proc. IWQoS, 2007.

[45]

L. Yuan, C.-N. Chuah, and P. Mohapatra. ProgME: Towards Programmable Network MEasurement. In Proc. SIGCOMM, 2007.

Digital Library

[46]

Q. Zhao, J. Xu, and Z. Liu. Design of a novel statistics counter architecture with optimal space and time efficiency. In Proc. ACM SIGMETRICS, 2006.

Digital Library

Cited By

Seufert MDietz KWehner NGeißler SSchüler JWolz MHotho ACasas PHoßfeld TFeldmann A(2024) Marina : Realizing ML-Driven Real-Time Network Traffic Monitoring at Terabit Scale IEEE Transactions on Network and Service Management10.1109/TNSM.2024.338239321:3(2773-2790)Online publication date: Jun-2024
https://doi.org/10.1109/TNSM.2024.3382393
Sun HHuang QLee PBai WZhu FBao Y(2024)Distributed Network Telemetry With Resource Efficiency and Full AccuracyIEEE/ACM Transactions on Networking10.1109/TNET.2023.332734532:3(1857-1872)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2023.3327345
Femi-Oyewole FOsamor VOkunbor D(2024)Survey on Predictive Algorithms to Detect Insider Threat on a Network Using Different Combination of Machine Learning Algorithms2024 International Conference on Science, Engineering and Business for Driving Sustainable Development Goals (SEB4SDG)10.1109/SEB4SDG60871.2024.10630366(1-14)Online publication date: 2-Apr-2024
https://doi.org/10.1109/SEB4SDG60871.2024.10630366
Show More Cited By

Index Terms

Revisiting the case for a minimalist approach for network flow monitoring
1. Networks
  1. Network services
    1. Network management
    2. Network monitoring

Recommendations

Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices
SIGMETRICS '05: Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems

The traffic volume between origin/destination (OD) pairs in a network, known as traffic matrix, is essential for efficient network provisioning and traffic engineering. Existing approaches of estimating the traffic matrix, based on statistical inference ...
Fair sampling across network flow measurements
SIGMETRICS '12: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE joint international conference on Measurement and Modeling of Computer Systems

Sampling is crucial for controlling resource consumption by internet traffic flow measurements. Routers use Packet Sampled NetFlow, and completed flow records are sampled in the measurement infrastructure. Recent research, motivated by the need of ...
Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices
Performance evaluation review

The traffic volume between origin/destination (OD) pairs in a network, known as traffic matrix, is essential for efficient network provisioning and traffic engineering. Existing approaches of estimating the traffic matrix, based on statistical inference ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '10: Proceedings of the 10th ACM SIGCOMM conference on Internet measurement

November 2010

496 pages

ISBN:9781450304832

DOI:10.1145/1879141

Program Chair:
Mark Allman
International Computer Science Institute

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

USENIX Assoc: USENIX Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IMC '10

Sponsor:

IMC '10: Internet Measurement Conference

November 1 - 30, 2010

Melbourne, Australia

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Upcoming Conference

IMC '24

Sponsor:
sigcomm
sigcomm

ACM Internet Measurement Conference

November 4 - 6, 2024

Madrid , AA , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
421
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Seufert MDietz KWehner NGeißler SSchüler JWolz MHotho ACasas PHoßfeld TFeldmann A(2024) Marina : Realizing ML-Driven Real-Time Network Traffic Monitoring at Terabit Scale IEEE Transactions on Network and Service Management10.1109/TNSM.2024.338239321:3(2773-2790)Online publication date: Jun-2024
https://doi.org/10.1109/TNSM.2024.3382393
Sun HHuang QLee PBai WZhu FBao Y(2024)Distributed Network Telemetry With Resource Efficiency and Full AccuracyIEEE/ACM Transactions on Networking10.1109/TNET.2023.332734532:3(1857-1872)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2023.3327345
Femi-Oyewole FOsamor VOkunbor D(2024)Survey on Predictive Algorithms to Detect Insider Threat on a Network Using Different Combination of Machine Learning Algorithms2024 International Conference on Science, Engineering and Business for Driving Sustainable Development Goals (SEB4SDG)10.1109/SEB4SDG60871.2024.10630366(1-14)Online publication date: 2-Apr-2024
https://doi.org/10.1109/SEB4SDG60871.2024.10630366
Liu HSun XChen XHuang QZhang DZhou HWu CLiu XKhan M(2024) : Low-latency and reliable event collection in network measurement Journal of Network and Computer Applications10.1016/j.jnca.2024.103904228(103904)Online publication date: Aug-2024
https://doi.org/10.1016/j.jnca.2024.103904
Chen HChen PWang BYu XChen XMa DZheng Z(2024)Graph neural network based robust anomaly detection at service level in SDN driven microservice systemComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2023.110135239:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.comnet.2023.110135
Yang KWu YMiao RYang TLiu ZXu ZQiu RZhao YLv HJi ZXie GSchulzrinne HKohler EMaltz DMisra V(2023)ChameleMon: Shifting Measurement Attention as Network State ChangesProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604850(881-903)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604850
Chen XLiu HZhang DHuang QZhou HWu CYang Q(2023)Eliminating Control Plane Overload via Measurement Task PlacementIEEE/ACM Transactions on Networking10.1109/TNET.2022.322342031:4(1717-1731)Online publication date: Aug-2023
https://doi.org/10.1109/TNET.2022.3223420
Chen XLiu HHuang QZhang DZhou HWu CLiu XYang Q(2023)Toward Low-Latency and Accurate State Synchronization for Programmable NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2022.321844631:3(1400-1415)Online publication date: Jun-2023
https://doi.org/10.1109/TNET.2022.3218446
Drozdenko BPowell M(2022)Utilizing Deep Learning Techniques to Detect Zero Day Exploits in Network Traffic Flows2022 IEEE 13th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON)10.1109/UEMCON54665.2022.9965695(0163-0172)Online publication date: 26-Oct-2022
https://doi.org/10.1109/UEMCON54665.2022.9965695
Sadrhaghighi SDolati MGhaderi MKhonsari A(2022)FlowShark: Sampling for High Flow Visibility in SDNsIEEE INFOCOM 2022 - IEEE Conference on Computer Communications10.1109/INFOCOM48880.2022.9796658(160-169)Online publication date: 2-May-2022
https://doi.org/10.1109/INFOCOM48880.2022.9796658
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents