Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/2934872.2934906acmconferencesArticle/Chapter ViewAbstractPublication PagescommConference Proceedingsconference-collections
research-article
Free access

One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon

Published: 22 August 2016 Publication History
  • Get Citation Alerts
  • Abstract

    Network management requires accurate estimates of metrics for traffic engineering (e.g., heavy hitters), anomaly detection (e.g., entropy of source addresses), and security (e.g., DDoS detection). Obtaining accurate estimates given router CPU and memory constraints is a challenging problem. Existing approaches fall in one of two undesirable extremes: (1) low fidelity general-purpose approaches such as sampling, or (2) high fidelity but complex algorithms customized to specific application-level metrics. Ideally, a solution should be both general (i.e., supports many applications) and provide accuracy comparable to custom algorithms. This paper presents UnivMon, a framework for flow monitoring which leverages recent theoretical advances and demonstrates that it is possible to achieve both generality and high accuracy. UnivMon uses an application-agnostic data plane monitoring primitive; different (and possibly unforeseen) estimation algorithms run in the control plane, and use the statistics from the data plane to compute application-level metrics. We present a proof-of-concept implementation of UnivMon using P4 and develop simple coordination techniques to provide a ``one-big-switch'' abstraction for network-wide monitoring. We evaluate the effectiveness of UnivMon using a range of trace-driven evaluations and show that it offers comparable (and sometimes better) accuracy relative to custom sketching solutions.

    Supplementary Material

    MP4 File (p101.mp4)

    References

    [1]
    Caida internet traces 2014 sanjose. http://goo.gl/uP5aqG.
    [2]
    Caida internet traces 2015 chicago. http://goo.gl/xgIUmF.
    [3]
    Intel flexpipe. http://goo.gl/H5qPP2.
    [4]
    Netfpga technical specifications. http://netfpga.org/1G_specs.html.
    [5]
    Opensketch simulation library. https://goo.gl/kyQ80q.
    [6]
    P4 behavioral simulator. https://github.com/p4lang/p4factory.
    [7]
    P4 specification. http://goo.gl/5ttjpA.
    [8]
    Why big data needs big buffer switches. https://goo.gl/ejWUIq.
    [9]
    N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. In Proc., STOC, 1996.
    [10]
    N. Bandi, A. Metwally, D. Agrawal, and A. El Abbadi. Fast data stream algorithms using associative memories. In Proc., SIGMOD, 2007.
    [11]
    T. Benson, A. Anand, A. Akella, and M. Zhang. Microte: Fine grained traffic engineering for data centers. In Proc., CoNEXT, 2011.
    [12]
    P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, and D. Walker. P4: Programming protocol-independent packet processors. SIGCOMM Comput. Commun. Rev., July 2014.
    [13]
    P. Bosshart, G. Gibb, H.-S. Kim, G. Varghese, N. McKeown, M. Izzard, F. Mujica, and M. Horowitz. Forwarding metamorphosis: Fast programmable match-action processing in hardware for sdn. In Proc., SIGCOMM, 2013.
    [14]
    V. Braverman and S. R. Chestnut. Universal Sketches for the Frequency Negative Moments and Other Decreasing Streaming Sums. In APPROX/RANDOM, 2015.
    [15]
    V. Braverman, S. R. Chestnut, R. Krauthgamer, and L. F. Yang. Streaming symmetric norms via measure concentration. CoRR, 2015.
    [16]
    V. Braverman, S. R. Chestnut, D. P. Woodruff, and L. F. Yang. Streaming space complexity of nearly all functions of one variable on frequency vectors. In Proc., PODS, 2016.
    [17]
    V. Braverman, J. Katzman, C. Seidell, and G. Vorsanger. An optimal algorithm for large frequency moments using o(n̂(1-2/k)) bits. In APPROX/RANDOM, 2014.
    [18]
    V. Braverman, Z. Liu, T. Singh, N. V. Vinodchandran, and L. F. Yang. New bounds for the CLIQUE-GAP problem using graph decomposition theory. In In Proc., MFCS, 2015.
    [19]
    V. Braverman and R. Ostrovsky. Zero-one frequency laws. In Proc., STOC, 2010.
    [20]
    V. Braverman and R. Ostrovsky. Approximating large frequency moments with pick-and-drop sampling. In APPROX/ROMDOM, 2013.
    [21]
    V. Braverman and R. Ostrovsky. Generalizing the layering method of indyk and woodruff: Recursive sketches for frequency-based vectors on streams. In APPROX/RAMDOM. 2013.
    [22]
    V. Braverman, R. Ostrovsky, and A. Roytman. Zero-one laws for sliding windows and universal sketches. In APPROX/RANDOM, 2015.
    [23]
    A. Chakrabarti, S. Khot, and X. Sun. Near-optimal lower bounds on the multi-party communication complexity of set disjointness. In IEEE CCC, 2003.
    [24]
    M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. In Automata, Languages and Programming. 2002.
    [25]
    B. Claise. Cisco systems netflow services export version 9. RFC 3954.
    [26]
    G. Cormode and S. Muthukrishnan. An improved data stream summary: The count-min sketch and its applications. J. Algorithms, 2005.
    [27]
    S. Dasgupta and A. Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Struct. Algorithms, Jan. 2003.
    [28]
    M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. SIAM J. Comput., June 2002.
    [29]
    R. Dementiev, T. Willhalm, O. Bruggeman, P. Fay, P. Ungerer, A. Ott, P. Lu, J. Harris, P. Kerly, P. Konsor, A. Semin, M. Kanaly, R. Brazones, and R. Shah. Intel performance counter monitor - a better way to measure cpu utilization. http://goo.gl/tQ5gxa.
    [30]
    N. Duffield, C. Lund, and M. Thorup. Estimating flow distributions from sampled flow statistics. In Proc., SIGCOMM, 2003.
    [31]
    C. Estan and G. Varghese. New directions in traffic measurement and accounting. In Proc., SIGCOMM, 2002.
    [32]
    A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, and F. True. Deriving traffic demands for operational ip networks: Methodology and experience. IEEE/ACM Trans. Netw., June 2001.
    [33]
    P. Indyk, A. McGregor, I. Newman, and K. Onak. Open problems in data streams, property testing, and related topics. 2011.
    [34]
    N. Kang, Z. Liu, J. Rexford, and D. Walker. Optimizing the "one big switch" abstraction in software-defined networks. In Proc., CoNEXT, 2013.
    [35]
    S. Knight, H. Nguyen, N. Falkner, R. Bowden, and M. Roughan. The internet topology zoo. Selected Areas in Communications, IEEE Journal on, october 2011.
    [36]
    B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen. Sketch-based change detection: methods, evaluation, and applications. In Proc., ACM SIGCOMM IMC, 2003.
    [37]
    A. Kumar, M. Sung, J. J. Xu, and J. Wang. Data streaming algorithms for efficient and accurate estimation of flow size distribution. In Proc., SIGMETRICS, 2004.
    [38]
    A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. In Proc., SIGMETRICS/Performance, 2006.
    [39]
    Z. Liu, G. Vorsanger, V. Braverman, and V. Sekar. Enabling a "risc" approach for software-defined monitoring using universal streaming. In Proc., ACM HotNets, 2015.
    [40]
    N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner. Openflow: Enabling innovation in campus networks. SIGCOMM Comput. Commun. Rev., Mar. 2008.
    [41]
    M. Moshref, M. Yu, R. Govindan, and A. Vahdat. SCREAM: Sketch Resource Allocation for Software-defined Measurement. In Proc., CoNEXT, 2015.
    [42]
    B. Pfaff, J. Pettit, T. Koponen, K. Amidon, M. Casado, and S. Shenker. Extending networking into the virtualization layer. In Proc., HotNets, 2009.
    [43]
    A. Ramachandran, S. Seetharaman, N. Feamster, and V. Vazirani. Fast monitoring of traffic subpopulations. In Proc., IMC, 2008.
    [44]
    R. Schweller, A. Gupta, E. Parsons, and Y. Chen. Reversible sketches for efficient and accurate change detection over network data streams. In Proc., IMC, 2004.
    [45]
    V. Sekar, M. K. Reiter, and H. Zhang. Revisiting the case for a minimalist approach for network flow monitoring. In Proc., IMC, 2010.
    [46]
    Y. Xie, V. Sekar, D. A. Maltz, M. K. Reiter, and H. Zhang. Worm origin identification using random moonwalks. In S&P. IEEE Computer Society, 2005.
    [47]
    M. Yu, L. Jose, and R. Miao. Software defined traffic measurement with opensketch. In Proc., NSDI, 2013.
    [48]
    L. Yuan, C.-N. Chuah, and P. Mohapatra. Progme: towards programmable network measurement. IEEE/ACM TON, 2011.
    [49]
    Y. Zhang. An adaptive flow counting method for anomaly detection in sdn. In Proc., CoNEXT, 2013.
    [50]
    H. C. Zhao, A. Lall, M. Ogihara, O. Spatscheck, J. Wang, and J. Xu. A data streaming algorithm for estimating entropies of od flows. In Proc., IMC, 2007.
    [51]
    H. C. Zhao, A. Lall, M. Ogihara, and J. J. Xu. Global iceberg detection over distributed data streams. In Proc., ICDE, 2010.

    Cited By

    View all
    • (2024)An Accurate and Invertible Sketch for Super Spread DetectionElectronics10.3390/electronics1301022213:1(222)Online publication date: 3-Jan-2024
    • (2024)SAROS: A Self-Adaptive Routing Oblivious Sampling Method for Network-wide Heavy Hitter DetectionProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663429(142-148)Online publication date: 3-Aug-2024
    • (2024)SmartNIC Security Isolation in the Cloud with S-NICProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3650071(851-869)Online publication date: 22-Apr-2024
    • Show More Cited By

    Index Terms

    1. One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        SIGCOMM '16: Proceedings of the 2016 ACM SIGCOMM Conference
        August 2016
        645 pages
        ISBN:9781450341936
        DOI:10.1145/2934872
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 22 August 2016

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Flow Monitoring
        2. Sketching
        3. Streaming Algorithm

        Qualifiers

        • Research-article

        Conference

        SIGCOMM '16
        Sponsor:
        SIGCOMM '16: ACM SIGCOMM 2016 Conference
        August 22 - 26, 2016
        Florianopolis, Brazil

        Acceptance Rates

        SIGCOMM '16 Paper Acceptance Rate 39 of 231 submissions, 17%;
        Overall Acceptance Rate 554 of 3,547 submissions, 16%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)857
        • Downloads (Last 6 weeks)87

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)An Accurate and Invertible Sketch for Super Spread DetectionElectronics10.3390/electronics1301022213:1(222)Online publication date: 3-Jan-2024
        • (2024)SAROS: A Self-Adaptive Routing Oblivious Sampling Method for Network-wide Heavy Hitter DetectionProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663429(142-148)Online publication date: 3-Aug-2024
        • (2024)SmartNIC Security Isolation in the Cloud with S-NICProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3650071(851-869)Online publication date: 22-Apr-2024
        • (2024)Snatch: Online Streaming Analytics at the Network EdgeProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629577(349-369)Online publication date: 22-Apr-2024
        • (2024)CloudSentry: Two-Stage Heavy Hitter Detection for Cloud-Scale Gateway Overload ProtectionIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330185235:4(616-633)Online publication date: Apr-2024
        • (2024)Differentially Private Top-$k$ Flows Estimation Mechanism in Network TrafficIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.330925011:3(2462-2472)Online publication date: May-2024
        • (2024)CS-Sketch: Compressive Sensing Enhanced Sketch for Full Traffic MeasurementIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.330512511:3(2338-2352)Online publication date: May-2024
        • (2024)Unbiased Real-Time Traffic SketchingIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.328400411:3(2371-2383)Online publication date: May-2024
        • (2024)From CountMin to Super kJoin Sketches for Flow Spread EstimationIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.327966511:3(2353-2370)Online publication date: May-2024
        • (2024)Learning-Based Sketch for Adaptive and High-Performance Network MeasurementIEEE/ACM Transactions on Networking10.1109/TNET.2024.336417632:3(2571-2585)Online publication date: Jun-2024
        • Show More Cited By

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Get Access

        Login options

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media