research-article

Fast monitoring of traffic subpopulations

Authors:

Anirudh Ramachandran,

Srinivasan Seetharaman,

Vijay VaziraniAuthors Info & Claims

IMC '08: Proceedings of the 8th ACM SIGCOMM conference on Internet measurement

Pages 257 - 270

https://doi.org/10.1145/1452520.1452551

Published: 20 October 2008 Publication History

Abstract

Network accounting, forensics, security, and performance monitoring applications often need to examine detailed traces from subsets of flows ("subpopulations"), where the application desires flexibility in specifying the subpopulation (e.g., to detect a portscan, the application must observe many packets between a source and a destination with one packet to each port). However, the dynamism and volume of network traffic on many high-speed links necessitates traffic sampling, which adversely affects subpopulation monitoring: because many subpopulations of interest to operators are low-volume flows, conventional sampling schemes (e.g., uniform random sampling) miss much of the subpopulation's traffic. Today's routers and network devices provide scant support for monitoring specific traffic subpopulations.

This paper presents the design, implementation, and evaluation of FlexSample, a traffic monitoring engine that dynamically extracts traffic from subpopulations that operators define using conditions on packet header fields. FlexSample uses a fast, flexible counter array to provide rough estimates of packets' membership in respective subpopulations. Based on these coarse estimates, FlexSample then makes per-packet sampling decisions to sample proportionately from each subpopulation (as specified by a network operator), subject to an overall sampling constraint. We apply FlexSample to extract subpopulations such as port scans and traffic to high-degree nodes and find that it is able to capture significantly more packets from these subpopulations than conventional approaches.

References

[1]

Idle-scanning and Related IPID Games. http://nmap.org/idlescan.html.

[2]

Original posting describing FTP bounce scan. http://nmap.org/hobbit.ftpbounce.txt.

[3]

Arbor Networks. http://www.arbornetworks.com.

[4]

P. Barford, J. Kline, D. Plonka, and A. Ron. A Signal Analysis of Network Traffic Anomalies. In Proc. ACM SIGCOMM Internet Measurement Workshop, Marseille, France, Nov. 2002.

Digital Library

[5]

D. Brauckhoff, B. Tellenbach, A. Wagner, A. Lakhina, and M. May. Impact of Traffic Sampling on Anomaly Detection Metrics. In Proc. ACM SIGCOMM Internet Measurement Conference, Rio de Janeiro, Brazil, Oct. 2006.

Digital Library

[6]

G. Cantieni, G. Iannaccone, P. Thiran, C. Barakat, and C. Diot. Reformulating the monitor placement problem: Optimal network-wide sampling. Intel Research Technical Report, Feb. 2006.

[7]

B.-Y. Choi and S. Bhattacharyya. On the Accuracy and Overhead of Cisco Sampled NetFlow. In Proceedings of ACM SIGMETRICS Workshop on Large Scale Network Inference (LSNI), June 2005.

[8]

K. C. Claffy, G. C. Polyzos, and H.-W. Braun. Application of sampling methodologies to network traffic characterization. In Proc. ACM SIGCOMM, pages 194--203, San Francisco, CA, Sept. 1993.

Digital Library

[9]

N. Duffield. A Framework for Packet Selection and Reporting. IETF Internet Draft draft-ietf-psamp-framework-12.txt, June 2007.

[10]

N. Duffield, C. Lund, and M. Thorup. Charging from Sampled Network Usage. In Proc. ACM SIGCOMM Internet Measurement Workshop, San Fransisco, CA, Nov. 2001.

Digital Library

[11]

N. Duffield, C. Lund, and M. Thorup. Estimating flow distributions from sampled flow statistics. In Proc. ACM SIGCOMM, pages 325--336, Karlsruhe, Germany, Aug. 2003.

Digital Library

[12]

N. Duffield, C. Lund, and M. Thorup. Predicting Resource Usage and Estimation Accuracy in an IP Flow Measurement Collection Infrastructure. In Proc. ACM SIGCOMM Internet Measurement Conference, pages 179--191, Miami, FL, Oct. 2003.

Digital Library

[13]

N. Duffield, F. L. Presti, V. Paxson, and D. Towsley. Inferring Link Loss Using Striped Unicast Probes. In Proc. IEEE INFOCOM, Anchorage, AK, Apr. 2001.

[14]

C. Estan, K. Keys, D. Moore, and G. Varghese. Building a Better NetFlow. In Proc. ACM SIGCOMM, Portland, OR, Aug. 2004.

Digital Library

[15]

C. Estan and G. Varghese. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Transactions on Computer Systems, 21(3):270--313, Aug. 2003.

Digital Library

[16]

L. Fan, P. Cao, J. Almeida, and A. Z. Broder. Summary cache: A scalable wide-area Web cache sharing protocol. In Proc. ACM SIGCOMM, pages 254--265, Vancouver, Canada, Sept. 1998.

Digital Library

[17]

A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, and F. True. Deriving Traffic Demands for Operational IP Networks: Methodology and Experience. IEEE/ACM Transactions on Networking, 9(3):257--270, June 2001.

Digital Library

[18]

N. Hohn and D. Veitch. Inverting sampled traffic. In Proc. ACM SIGCOMM Internet Measurement Conference, Miami, FL, Oct. 2003.

Digital Library

[19]

Y. Huang and J. Pullen. Countering Denial of Service Attacks using Congestion Triggered Packet Sampling and Filtering. In Proceedings of International Conference on Computer Communications and Networks, pages 490--494, 2001.

[20]

InMon sFlow. http://www.inmon.com/technology.

[21]

Juniper traffic sampling and forwarding overview. http://www. juniper.net/techpubs/software/junos/junos71/swconfig71-policy/html/sampling-overview.html.

[22]

T. Karagiannis, K. Papagiannaki, and M. Faloutsos. BLINC: multilevel traffic classification in the dark. In Proc. ACM SIGCOMM, pages 229--240, Philadelphia, PA, Aug. 2005.

Digital Library

[23]

R. Kompella and C. Estan. The Power of Slicing in Internet Flow Measurement. In Proc. ACM SIGCOMM Internet Measurement Conference, Berkeley, CA, Oct. 2005.

Digital Library

[24]

A. Kumar, M. Sung, J. Xu, and J. Wang. Data streaming algorithms for efficient and accurate estimation of flow size distribution. In Proc. ACM SIGMETRICS, pages 177--188, New York, NY, June 2004.

Digital Library

[25]

A. Kumar, M. Sung, J. Xu, J. Wang, and E. W. Zegura. A Data Streaming Algorithm for Estimating Subpopulation Flow Size Distribution. In Proc. ACM SIGMETRICS, Banff, Canada, June 2005.

Digital Library

[26]

A. Kumar and J. Xu. Sketch Guided Sampling -- Using On-Line Estimates of Flow Size for Adaptive Data Collection. In Proc. IEEE INFOCOM, Barcelona, Spain, Mar. 2006.

[27]

Y. Lu, S. Dharmapurikar, A. K. Kabbani, A. Montanari, and B. Prabhakar. Counter Braids: An Efficient Minimum-Space Statistics Counter Architecture. In To appear in the Proceedings of ACM SIGMETRICS, June 2008.

Digital Library

[28]

H. V. Madhyastha and B. Krishnamurthy. A Generic Language for Application-Specific Flow Sampling. ACM Computer Communication Review, 38(2), April 2008.

Digital Library

[29]

J. Mai, C.-N. Chuah, A. Sridharan, T. Ye, and H. Zang. Is Sampled Data Sufficient for Anomaly Detection? In Proc. ACM SIGCOMM Internet Measurement Conference, Rio de Janeiro, Brazil, Oct. 2006.

Digital Library

[30]

G. Maier, R. Sommer, H. Dreger, A. Feldmann, V. Paxson, and F. Schneider. Enriching Network Security Analysis with Time Travel. In Proc. ACM SIGCOMM, Seattle, WA, Aug. 2008.

Digital Library

[31]

Cisco NetFlow. http://www.cisco.com/en/US/products/ps6601/products_ios_protocol_group_home.html.

[32]

A. Ramachandran and N. Feamster. Understanding the network-level behavior of spammers. In Proc. ACM SIGCOMM, Pisa, Italy, Sept. 2006. An earlier version appeared as Georgia Tech TR GT-CSS-2006-001.

Digital Library

[33]

H. Ringberg, A. Soule, and M. Caeser. Behavior Of Bots In Traffic Traces. Technical report, Princeton University, 2008. Number forthcoming.

[34]

L. A. Sanchez, W. C. Milliken, A. C. Snoeren, F. Tchakountio, C. E. Jones, S. T. Kent, C. Partridge, and W. T. Strayer. Hardware Support for a Hash-Based IP Traceback. In Proceedings of DARPA Information Survivability Conference and Exposition (DISCEX), 2001.

[35]

V. Sekar, M. K. Reiter, W. Willinger, H. Zhang, R. R. Kompella, and D. G. Andersen. cSamp: A system for network-wide flow monitoring. In Proc. 5th USENIX NSDI, San Francisco, CA, Apr. 2008.

Digital Library

[36]

H. Song, S. Dharmapurikar, J. Turner, and J. Lockwood. Fast Hash Table Lookup Using Extended Bloom Filter: An Aid To Network Processing. In Proc. ACM SIGCOMM, Philadelphia, PA, Aug. 2005.

Digital Library

[37]

G. Varghese. Network Algorithmics: An Interdisciplinary Approach to Designing Fast Networked Devices. Morgan Kaufmann Publishers Inc., 2004.

Digital Library

[38]

F. Vaskovich. Nmap stealth port scanner. http://www.insecure.org/nmap/index.html, 2002.

[39]

Average spam message size at record low. http://www.virusbtn.com/news/2008/04_03a.xml?rss.

[40]

Report: 95 percent of all email has that spammy smell. http://arstechnica.com/news.ars/post/20071212-report-95-percent-of-all-e-mail-has-that-spammy-smell.html.

[41]

K. Xu, Z.-L. Zhang, and S. Bhattacharyya. Profiling Internet Backbone Traffic: Behavior Models and Applcations. In Proc. ACM SIGCOMM, Philadelphia, PA, Aug. 2005.

Digital Library

[42]

B. Yang, R. Karri, and D. A. McGrew. Divide and Concatenate: An Architectural Level Optimization Technique for Universal Hash Functions. In Proceedings of the Design Automation Conference, San Diego, CA, 2004.

Digital Library

[43]

L. Yuan, C.-N. Chuah, and P. Mohapatra. ProgME: Towards Programmable Network MEasurement. In Proc. ACM SIGCOMM, Kyoto, Japan, Aug. 2007.

Digital Library

[44]

Y. Zhang, M. Roughan, C. Lund, and D. Donoho. An Information-Theoretic Approach to Traffic Matrix Estimation. In Proc. ACM SIGCOMM, pages 301--312, Karlsruhe, Germany, Aug. 2003.

Digital Library

[45]

Y. Zhang, S. Singh, S. Sen, N. Duffield, and C. Lund. Online Identification of Hierarchical Heavy Hitters: Algorithms, Evaluation, and Applications. In Proc. ACM SIGCOMM Internet Measurement Conference, Taormina, Sicily, Italy, Oct. 2004.

Digital Library

[46]

T. Zseby, M. Molina, N. Duffield, S. Niccolini, and F. Raspall. Sampling and Filtering Techniques for IP Packet Selection, Internet-Draft, draft-ietf-psamp-sample-tech-07.txt, Work in Progress, 2005.

Cited By

Fan WXiao FHan LHe XWang J(2024)Joint Optimization of Measurement Point Intelligent Selection and End-to-End Network Traffic Calculation in DatacentersIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.327868011:3(2438-2449)Online publication date: May-2024
https://doi.org/10.1109/TNSE.2023.3278680
Li DTian NQiu KChang HYu XZhao J(2024)VotePipe: Efficient Heavy Hitter Detection in Programmable Data PlaneFrontiers of Networking Technologies10.1007/978-981-97-3890-8_11(146-166)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-981-97-3890-8_11
Manousis ACheng ZBasat RLiu ZSekar V(2022)Enabling efficient and general subpopulation analytics in multidimensional data streamsProceedings of the VLDB Endowment10.14778/3551793.355186715:11(3249-3262)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.14778/3551793.3551867
Show More Cited By

Index Terms

Fast monitoring of traffic subpopulations
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
    2. Metrics
2. Networks
  1. Network services
    1. Network monitoring

Recommendations

Traffic statistics collection with FleXam
SIGCOMM'14

One of the limitations of wildcard rules in Software Defined Networks, such as OpenFlow, is losing visibility. FleXam is a flexible sampling extension for OpenFlow that allows the controller to define which packets should be sampled, what parts of each ...
Traffic statistics collection with FleXam
SIGCOMM '14: Proceedings of the 2014 ACM conference on SIGCOMM

One of the limitations of wildcard rules in Software Defined Networks, such as OpenFlow, is losing visibility. FleXam is a flexible sampling extension for OpenFlow that allows the controller to define which packets should be sampled, what parts of each ...
A new structure-preserving method of sampling for predicting self-similar traffic

The paper presents a structure-preserving method of sampling self-similar traffic with an application to network monitoring and resource provisioning. Based on the observation of the self-similarity of Internet traffic, we propose a new sampling ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '08: Proceedings of the 8th ACM SIGCOMM conference on Internet measurement

October 2008

352 pages

ISBN:9781605583341

DOI:10.1145/1452520

Program Chairs:
Konstantina Papagiannaki
Intel Research Pittsburgh, USA
,
Zhi-Li Zhang
University of Minnesota, USA

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IMC08

Sponsor:

IMC08: Internet Measurement Conference

October 20 - 22, 2008

Vouliagmeni, Greece

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Upcoming Conference

IMC '24

Sponsor:
sigcomm
sigcomm

ACM Internet Measurement Conference

November 4 - 6, 2024

Madrid , AA , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
909
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fan WXiao FHan LHe XWang J(2024)Joint Optimization of Measurement Point Intelligent Selection and End-to-End Network Traffic Calculation in DatacentersIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.327868011:3(2438-2449)Online publication date: May-2024
https://doi.org/10.1109/TNSE.2023.3278680
Li DTian NQiu KChang HYu XZhao J(2024)VotePipe: Efficient Heavy Hitter Detection in Programmable Data PlaneFrontiers of Networking Technologies10.1007/978-981-97-3890-8_11(146-166)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-981-97-3890-8_11
Manousis ACheng ZBasat RLiu ZSekar V(2022)Enabling efficient and general subpopulation analytics in multidimensional data streamsProceedings of the VLDB Endowment10.14778/3551793.355186715:11(3249-3262)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.14778/3551793.3551867
Alikhanov JJang RAbuhamad MMohaisen DNyang DNoh Y(2022)Investigating the Effect of Traffic Sampling on Machine Learning-Based Network Intrusion Detection ApproachesIEEE Access10.1109/ACCESS.2021.313731810(5801-5823)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2021.3137318
Li QLiu YLiu ZZhang PPang C(2021)Efficient Forwarding Anomaly Detection in Software-Defined NetworksIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.306813532:11(2676-2690)Online publication date: 1-Nov-2021
https://doi.org/10.1109/TPDS.2021.3068135
Michel OSonchack JCusack GNazari MKeller ESmith J(2021)Software Packet-Level Network Analytics at Cloud ScaleIEEE Transactions on Network and Service Management10.1109/TNSM.2021.305865318:1(597-610)Online publication date: Mar-2021
https://doi.org/10.1109/TNSM.2021.3058653
Kfoury ECrichigno JBou-Harb E(2021)An Exhaustive Survey on P4 Programmable Data Plane Switches: Taxonomy, Applications, Challenges, and Future TrendsIEEE Access10.1109/ACCESS.2021.30867049(87094-87155)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3086704
Jang RMin DMoon SMohaisen DNyang D(2020)SketchFlow: Per-Flow Systematic Sampling Using Sketch Saturation EventIEEE INFOCOM 2020 - IEEE Conference on Computer Communications10.1109/INFOCOM41043.2020.9155252(1339-1348)Online publication date: Jul-2020
https://doi.org/10.1109/INFOCOM41043.2020.9155252
Xuan STang DChung ICho YDu XYang W(2020)Network Traffic Sampling System Based on Storage Compression for Application Classification DetectionIEEE Access10.1109/ACCESS.2020.29842588(63106-63120)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2984258
Martins Rda Silva Villaça RVerdi F(2020)BitMatrix: A Multipurpose Sketch for Monitoring of Multi-tenant NetworksJournal of Network and Systems Management10.1007/s10922-020-09556-7Online publication date: 30-Jul-2020
https://doi.org/10.1007/s10922-020-09556-7
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents