research-article

Free access

SketchVisor: Robust Network Measurement for Software Packet Processing

Authors:

Patrick P. C. Lee,

Gong ZhangAuthors Info & Claims

SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data Communication

Pages 113 - 126

https://doi.org/10.1145/3098822.3098831

Published: 07 August 2017 Publication History

Abstract

Network measurement remains a missing piece in today's software packet processing platforms. Sketches provide a promising building block for filling this void by monitoring every packet with fixed-size memory and bounded errors. However, our analysis shows that existing sketch-based measurement solutions suffer from severe performance drops under high traffic load. Although sketches are efficiently designed, applying them in network measurement inevitably incurs heavy computational overhead.

We present SketchVisor, a robust network measurement framework for software packet processing. It augments sketch-based measurement in the data plane with a fast path, which is activated under high traffic load to provide high-performance local measurement with slight accuracy degradations. It further recovers accurate network-wide measurement results via compressive sensing. We have built a SketchVisor prototype on top of Open vSwitch. Extensive testbed experiments show that SketchVisor achieves high throughput and high accuracy for a wide range of network measurement tasks and microbenchmarks.

Supplementary Material

WEBM File (sketchvisorrobustnetworkmeasurementforsoftwarepacketprocessing.webm)

Download
92.53 MB

References

[1]

O. Alipourfard, M. Moshref, and M. Yu. Re-evaluating Measurement Algorithms in Software. In Proc. of HotNets, 2015.

Digital Library

[2]

Z. Bar-Yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. Counting Distinct Elements in a Data Stream. In Proc. of RANDOM, 2002.

[3]

T. Benson, A. Akella, and D. A. Maltz. Network Traffic Characteristics of Data Centers in the Wild. In Proc. of IMC, 2010.

Digital Library

[4]

F. Bonomi, M. Mitzenmacher, R. Panigrahy, S. Singh, and G. Varghese. An Improved Construction for Counting Bloom Filters. In Proc. of ESA, 2006.

Digital Library

[5]

Caida Anonymized Internet Traces 2015 DCaida. http://www.caida.org/data/passive/passive_2015_dataset.xml.

[6]

E. Candes, J. Romberg, and T. Tao. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2):489--509, 2006.

Digital Library

[7]

E. Candes and T. Tao. Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? IEEE Transactions on Information Theory, 52(12):5406--5425, 2006.

Digital Library

[8]

M. Charikar, K. Chen, and M. Farach-Colton. Finding Frequent Items in Data Streams. Theoretical Computer Science, 312(1):3--15, 2004.

Digital Library

[9]

Y.-C. Chen, L. Qiu, Y. Zhang, G. Xue, and Z. Hu. Robust Network Compressive Sensing. In Proc. of MOBICOM, 2014.

Digital Library

[10]

Cisco Nexus 1000V Switch. http://www.cisco.com/c/en/us/products/switches/nexus-1000v-switch-vmware-vsphere/index.html.

[11]

G. Cormode and M. Garofalakis. Sketching Streams Through the Net: Distributed Approximate Query Tracking. In Proc. of VLDB, 2005.

[12]

G. Cormode and M. Hadjieleftheriou. Methods for Finding Frequent Items in Data Streams. The VLDB Journal, 19(1):3--20, 2010.

Digital Library

[13]

G. Cormode and S. Muthukrishnan. What's New: Finding Significant Differences in Network Data Streams. In Proc. of IEEE INFOCOM, 2004.

[14]

G. Crmode and S. Muthukrishnan. An Improved Data Stream Summary: The Count-Min Sketch and its Applications. Journal of Algorithms, 55(1):58--75, 2005.

Digital Library

[15]

X. Dimitropoulos, P. Hurley, and A. Kind. Probabilistic Lossy Counting: An Efficient Algorithm for Finding Heavy Hitters. ACM SIGCOMM Computer Communication Review, 38(1):5--5, 2008.

Digital Library

[16]

M. Dobrescu, N. Egi, K. Argyraki, B.-g. Chun, K. Fall, G. Iannaccone, A. Knies, M. Manesh, and S. Ratnasamy. RouteBricks: Exploiting Parallelism to Scale Software Routers. In Proc. of SOSP, 2009.

Digital Library

[17]

DPDK. http://dpdk.org/.

[18]

C. Eckart and G. Young. The Approximation of One Matrix by Another of Lower Rank. Psychometrika, 1(3):211--218, 1936.

[19]

C. Estan and G. Varghese. New Directions in Traffic Measurement and Accounting: Focusing on the Elephants, Ignoring the Mice. ACM Trans. on Computer Systems, 21(3):270--313, 2003.

Digital Library

[20]

P. Flajolet and G. Nigel Martin. Probabilistic Counting Algorithms for Data Base Applications. Journal of Computer and System Sciences, 31(2):182--209, 1985.

Digital Library

[21]

N. Handigol, B. Heller, V. Jeyakumar, D. Mazières, and N. McKeown. I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks. In Proc. of NSDI, 2014.

[22]

Q. Huang and P. P. C. Lee. A Hybrid Local and Distributed Sketching Design for Accurate and Scalable Heavy Key Detection in Network Data Streams. Computer Networks, 91:298--315, 2015.

Digital Library

[23]

L. Jose, M. Yu, and J. Rexford. Online Measurement of Large Traffic Aggregates on Commodity Switches. In USENIX HotICE, 2011.

[24]

S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken. The Nature of Data Center Traffic: Measurements & Analysis. In Proc. of IMC, 2009.

Digital Library

[25]

T. Koponen, K. Amidon, P. Balland, M. Casado, A. Chanda, B. Fulton, I. Ganichev, J. Gross, P. Ingram, E. Jackson, A. Lambeth, R. Lenglet, S.-H. Li, A. Padmanabhan, J. Pettit, B. Pfaff, R. Ramanathan, S. Shenker, A. Shieh, J. Stribling, P. Thakkar, D. Wendlandt, A. Yip, and R. Zhang. Network Virtualization in Multi-tenant Datacenters. In Proc. of NSDI, 2014.

Digital Library

[26]

A. Kumar, M. Sung, J. J. Xu, and J. Wang. Data Streaming Algorithms for Efficient and Accurate Estimation of Flow Size Distribution. In Proc. of SIGMETRICS, 2004.

Digital Library

[27]

P. P. C. Lee, T. Bu, and G. Chandranmenon. A Lock-Free, Cache-Efficient Multi-Core Synchronization Mechanism for Line-Rate Network Traffic Monitoring. In Proc. of IPDPS, 2010.

[28]

Y. Li, R. Miao, C. Kim, and M. Yu. FlowRadar: A Better NetFlow for Data Centers. In Proc. of NSDI, 2016.

[29]

X. Liu, M. Shirazipour, M. Yu, and Y. Zhang. MOZART: Temporal Coordination of Measurement. In Proc. of SOSR, 2016.

Digital Library

[30]

Z. Liu, A. Manousis, G. Vorsanger, V. Sekar, and V. Braverman. One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon. In Proc. of SIGCOMM, 2016.

Digital Library

[31]

N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner. OpenFlow: Enabling Innovation in Campus Networks. ACM SIGCOMM Computer Communication Review, 38(2):69, 2008.

Digital Library

[32]

Microsoft Hyper-V Virtual Switch. https://technet.microsoft.com/en-us/library/hh831823.aspx.

[33]

J. Misra and D. Gries. Finding Repeated Elements. Science of Computer Programming, 2(2):143--152, 1982.

[34]

M. Mitzenmacher. Compressed Bloom Filters. In Proc. of PODC, 2001.

Digital Library

[35]

M. Mitzenmacher, R. Pagh, and N. Pham. Efficient Estimation for High Similarities Using Odd Sketches. In Proc. of WWW, 2014.

Digital Library

[36]

M. Moshref, M. Yu, R. Govindan, and A. Vahdat. DREAM: Dynamic Resource Allocation for Software-defined Measurement. In Proc. of SIGCOMM, 2014.

Digital Library

[37]

M. Moshref, M. Yu, R. Govindan, and A. Vahdat. SCREAM: Sketch Resource Allocation for Software-defined Measurement. In Proc. of CoNEXT, 2015.

Digital Library

[38]

M. Moshref, M. Yu, R. Govindan, and A. Vahdat. Trumpet: Timely and Precise Triggers in Data Centers. In Proc. of SIGCOMM, 2016.

Digital Library

[39]

S. Narayana, M. T. Arashloo, J. Rexford, and D. Walker. Compiling Path Queries. In Proc. of NSDI, 2016.

[40]

NetFlow. https://www.ietf.org/rfc/rfc3954.txt.

[41]

OpenvSwitch. http://openvswitch.org.

[42]

perf. http://perf.wiki.kernel.org.

[43]

J. Rasley, B. Stephens, C. Dixon, E. Rozner, W. Felter, K. Agarwal, J. Carter, and R. Fonseca. Planck: Millisecond-scale Monitoring and Control for Commodity Networks. In Proc. of SIGCOMM, 2014.

Digital Library

[44]

B. Recht, M. Fazel, and P. A. Parrilo. Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization. SIAM Review, 52(3):471--501, 2010.

Digital Library

[45]

L. Rizzo. Netmap: A Novel Framework for Fast Packet I/O. In Proc. of ATC, 2012.

[46]

R. Schweller, Z. Li, Y. Chen, Y. Gao, A. Gupta, Y. Zhang, P. Dinda, M. Y. Kao, and G. Memik. Reversible Sketches: Enabling Monitoring and Analysis over High-Speed Data Streams. IEEE/ACM Trans. on Networking, 15(5):1059--1072, 2007.

Digital Library

[47]

V. Sekar, M. K. Reiter, W. Willinger, H. Zhang, R.R. Kompella, and D.G. Andersen. cSAMP: A System for Network-Wide Flow Monitoring. In Proc. of USENIX NSDI, 2008.

Digital Library

[48]

V. Sekar, M. K. Reiter, and H. Zhang. Revisiting the Case for a Minimalist Approach for Network Flow Monitoring. In Proc. of IMC, 2010.

Digital Library

[49]

sFlow. http://www.sflow.org/.

[50]

Snort. https://www.snort.org.

[51]

SPAN. http://www.cisco.com/c/en/us/tech/lan-switching/switched-port-analyzer-span/index.html.

[52]

J. Suh, T. T. Kwon, C. Dixon, W. Felter, and J. Carter. OpenSample: A Low-Latency, Sampling-Based Measurement Platform for Commodity SDN. In Proc. of ICDCS, 2014.

Digital Library

[53]

svdcomp. http://www.public.iastate.edu/~dicook/JSS/paper/code/svd.c.

[54]

K. Thompson, G. J. Miller, and R. Wilder. Wide-Area Internet Traffic Patterns and Characteristics. IEEE Network, 11:10--23, 1997.

Digital Library

[55]

K.-Y. Whang, B. T. Vander-Zanden, and H. M. Taylor. A Linear-time Probabilistic Counting Algorithm for Database Applications. ACM Trans. Database Systems, 15(2):208--229, 1990.

Digital Library

[56]

M. Yu, L. Jose, and R. Miao. Software Defined Traffic Measurement with OpenSketch. In Proc. of NSDI, 2013.

[57]

L. Yuan, C.-N. Chuah, and P. Mohapatra. ProgME: Towards Programmable Network Measurement. In Proc. of SIGCOMM, 2007.

Digital Library

[58]

ZeroMQ. http://zeromq.org.

[59]

Y. Zhang, L. Breslau, V. Paxson, and S. Shenker. On the Characteristics and Origins of Internet Flow Rates. In Proc. of ACM SIGCOMM, 2002.

Digital Library

[60]

Y. Zhang, M. Roughan, N. Duffield, and A. Greenberg. Fast Accurate Computation of Large-Scale IP Traffic Matrices from Link Loads. In Proc. of SIGMETRICS, 2003.

Digital Library

[61]

Y. Zhang, M. Roughan, W. Willinger, and L. Qiu. Spatio-Temporal Compressive Sensing and Internet Traffic Matrices. In Proc. of SIGCOMM, 2009.

Digital Library

[62]

Y. Zhu, N. Kang, J. Cao, A. Greenberg, G. Lu, R. Mahajan, D. Maltz, L. Yuan, M. Zhang, B. Y. Zhao, and H. Zheng. Packet-Level Telemetry in Large Datacenter Networks. In Proc. of SIGCOMM, 2015.

Digital Library

Cited By

Ben Basat REinziger GHan WTayh B(2024)SQUID: Faster Analytics via Sampled Quantile EstimationProceedings of the ACM on Networking10.1145/36768732:CoNEXT3(1-23)Online publication date: 21-Aug-2024
https://doi.org/10.1145/3676873
Zheng HHuang CHan XZheng JWang XTian CDou WChen GSekar VYu MSeneviratne AVeitch D(2024)μMon: Empowering Microsecond-level Network Monitoring with WaveletsProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672236(274-290)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672236
Lu JPan THe SMiao MZhou GQi YZhang SSong ESun XZhao HLyu BZhu S(2024)CloudSentry: Two-Stage Heavy Hitter Detection for Cloud-Scale Gateway Overload ProtectionIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330185235:4(616-633)Online publication date: Apr-2024
https://doi.org/10.1109/TPDS.2023.3301852
Show More Cited By

Index Terms

SketchVisor: Robust Network Measurement for Software Packet Processing
1. Networks
  1. Network performance evaluation
    1. Network measurement

Recommendations

Sketchlearn: relieving user burdens in approximate measurement with automated statistical inference
SIGCOMM '18: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

Network measurement is challenged to fulfill stringent resource requirements in the face of massive network traffic. While approximate measurement can trade accuracy for resource savings, it demands intensive manual efforts to configure the right ...
Accelerating network measurement in software

Network measurement plays an important role for many network functions such as detecting network anomalies and identifying big flows. However, most existing measurement solutions fail to achieve high performance in software as they often incorporate ...
Bandwidth-efficient Microburst Measurement in Large-scale Datacenter Networks
APNet '22: Proceedings of the 6th Asia-Pacific Workshop on Networking

Microburst measurement is essential for diagnosing and mitigating performance problems in datacenter networks. The key is to efficiently identify the flows that contribute the most to queue buildup. However, because the existing microburst measurement ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data Communication

August 2017

515 pages

ISBN:9781450346535

DOI:10.1145/3098822

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SIGCOMM '17

Sponsor:

SIGCOMM

SIGCOMM '17: ACM SIGCOMM 2017 Conference

August 21 - 25, 2017

CA, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

176
Total Citations
View Citations
4,041
Total Downloads

Downloads (Last 12 months)414
Downloads (Last 6 weeks)36

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ben Basat REinziger GHan WTayh B(2024)SQUID: Faster Analytics via Sampled Quantile EstimationProceedings of the ACM on Networking10.1145/36768732:CoNEXT3(1-23)Online publication date: 21-Aug-2024
https://doi.org/10.1145/3676873
Zheng HHuang CHan XZheng JWang XTian CDou WChen GSekar VYu MSeneviratne AVeitch D(2024)μMon: Empowering Microsecond-level Network Monitoring with WaveletsProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672236(274-290)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672236
Lu JPan THe SMiao MZhou GQi YZhang SSong ESun XZhao HLyu BZhu S(2024)CloudSentry: Two-Stage Heavy Hitter Detection for Cloud-Scale Gateway Overload ProtectionIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330185235:4(616-633)Online publication date: Apr-2024
https://doi.org/10.1109/TPDS.2023.3301852
Seufert MDietz KWehner NGeißler SSchüler JWolz MHotho ACasas PHoßfeld TFeldmann A(2024) Marina : Realizing ML-Driven Real-Time Network Traffic Monitoring at Terabit Scale IEEE Transactions on Network and Service Management10.1109/TNSM.2024.338239321:3(2773-2790)Online publication date: Jun-2024
https://doi.org/10.1109/TNSM.2024.3382393
Ren JWang YLiu YWang XWang S(2024)Server-Assisted Traffic Measurement for Programmable Data Center NetworksIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.339729111:5(4729-4743)Online publication date: Sep-2024
https://doi.org/10.1109/TNSE.2024.3397291
Li LXie KPei SWen JLiang WXie G(2024)CS-Sketch: Compressive Sensing Enhanced Sketch for Full Traffic MeasurementIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.330512511:3(2338-2352)Online publication date: May-2024
https://doi.org/10.1109/TNSE.2023.3305125
Misa CDurairajan RRejaie RWillinger W(2024)DynATOS+: A Network Telemetry System for Dynamic Traffic and Query WorkloadsIEEE/ACM Transactions on Networking10.1109/TNET.2024.336743232:4(2810-2825)Online publication date: Aug-2024
https://doi.org/10.1109/TNET.2024.3367432
Li FLv YYan YGao CWang XCao J(2024)Learning-Based Sketch for Adaptive and High-Performance Network MeasurementIEEE/ACM Transactions on Networking10.1109/TNET.2024.336417632:3(2571-2585)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3364176
Sun HHuang QLee PBai WZhu FBao Y(2024)Distributed Network Telemetry With Resource Efficiency and Full AccuracyIEEE/ACM Transactions on Networking10.1109/TNET.2023.332734532:3(1857-1872)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2023.3327345
Li WPatras P(2024)P-Sketch: A Fast and Accurate Sketch for Persistent Item LookupIEEE/ACM Transactions on Networking10.1109/TNET.2023.330689732:2(987-1002)Online publication date: Apr-2024
https://doi.org/10.1109/TNET.2023.3306897
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents