research-article

Public Access

I Know What You Did Last Summer: Network Monitoring using Interval Queries

Authors:

Vladimir BravermanAuthors Info & Claims

Proceedings of the ACM on Measurement and Analysis of Computing Systems, Volume 3, Issue 3

Article No.: 61, Pages 1 - 28

https://doi.org/10.1145/3366709

Published: 17 December 2019 Publication History

Abstract

Modern network telemetry systems collect and analyze massive amounts of raw data in a space efficient manner. These require advanced capabilities such as drill down queries that allow iterative refinement of the search space. We present a first integral solution that (i) enables multiple measurement tasks inside the same data structure, (ii) supports specifying the time frame of interest as part of its queries, and (iii) is sketch-based and thus space efficient. Namely, our approach allows the user to define both the measurement task (e.g., heavy hitters, entropy estimation, count distinct, etc.) and the time frame of relevance (e.g., 5PM-6PM) at query time. Our approach provides accuracy guarantees and is the only space-efficient solution that offers such capabilities. Finally, we demonstrate how our system can be used for accurately pinpointing the start of a realistic DDoS attack.

References

[1]

The CAIDA Anonymized Internet Trace, equinix-chicago 2016-06--21, Dir. A.

[2]

The CAIDA Anonymized Internet Trace equinix-nyc 2018-03--15, Dir. A.

[3]

The CAIDA Anonymized Internet Trace, equinix-sanjose 2014-03--20, Dir. B.

[4]

Charu C Aggarwal. Data Streams: Models and Algorithms, volume 31. Springer Science & Business Media, 2007.

[5]

Noga Alon, Yossi Matias, and Mario Szegedy. The Space Complexity of Approximating the Frequency Moments. J. Comp. and sys. sciences, 1999.

[6]

Eran Assaf, Ran Ben-Basat, Gil Einziger, and Roy Friedman. Pay for a Sliding Bloom Filter and Get Counting, Distinct Elements, and Entropy for Free. In IEEE INFOCOM, 2018.

[7]

Ziv Bar-Yossef, Thathachar S Jayram, Ravi Kumar, and D Sivakumar. An Information Statistics Approach to Data Stream and Communication Complexity. Journal of Computer and System Sciences, 2004.

Digital Library

[8]

Ran Ben Basat, Gil Einziger, Isaac Keslassy, Ariel Orda, Shay Vargaftik, and Erez Waisbard. Memento: Making Sliding Windows Efficient for Heavy Hitters. In ACM CoNEXT, 2018.

[9]

Ran Ben Basat, Roy Friedman, and Rana Shahout. Heavy Hitters over Interval Queries. In PVLDB, 2019. Also available on arXiv:1804.10740.

[10]

R. Ben-Basat, G. Einziger, R. Friedman, and Y. Kassner. Heavy Hitters in Streams and Sliding Windows. In IEEE INFOCOM, 2016.

[11]

Ran Ben-Basat, Gil Einziger, Roy Friedman, and Yaron Kassner. Randomized Admission Policy for Efficient Top-k and Frequency Estimation. In IEEE INFOCOM, 2017.

[12]

Ran Ben-Basat, Gil Einziger, Roy Friedman, Marcelo Caggiani Luizelli, and Erez Waisbard. Constant Time Updates in Hierarchical Heavy Hitters. ACM SIGCOMM, 2017.

Digital Library

[13]

Theophilus Benson, Aditya Akella, and David A. Maltz. Network Traffic Characteristics of Data Centers in the Wild. In ACM IMC, 2010.

[14]

Theophilus Benson, Ashok Anand, Aditya Akella, and Ming Zhang. MicroTE: Fine Grained Traffic Engineering for Data Centers. In ACM CoNEXT, 2011.

Digital Library

[15]

Arnab Bhattacharyya, Palash Dey, and David P Woodruff. An Optimal Algorithm for $L_1$-Heavy Hitters in Insertion Streams and Related Problems. In ACM PODS, 2016.

Digital Library

[16]

V. Braverman. Sliding window algorithms. Encyc. of Algorithms, 2004.

[17]

Vladimir Braverman, Stephen R Chestnut, Nikita Ivkin, Jelani Nelson, Zhengyu Wang, and David P Woodruff. BPTree: an $L_2 $ Heavy Hitters Algorithm using Constant Memory. arXiv:1603.00759, 2016.

[18]

Vladimir Braverman, Stephen R Chestnut, Nikita Ivkin, and David P Woodruff. Beating CountSketch for Heavy Hitters in Insertion Streams. In ACM STOC, 2016.

Digital Library

[19]

Vladimir Braverman, Stephen R Chestnut, David P Woodruff, and Lin F Yang. Streaming Space Complexity of Nearly All Functions of One Variable on Frequency Vectors. In ACM PODS, 2016.

Digital Library

[20]

Vladimir Braverman, Ran Gelles, and Rafail Ostrovsky. How to Catch $L_2$-heavy-hitters on Sliding Windows. Theoretical Computer Science, 2014.

[21]

Vladimir Braverman, Elena Grigorescu, Harry Lang, David P Woodruff, and Samson Zhou. Nearly Optimal Distinct Elements and Heavy Hitters on Sliding Windows. arXiv preprint arXiv:1805.00212, 2018.

[22]

Vladimir Braverman and Rafail Ostrovsky. Smooth Histograms for Sliding Windows. In IEEE FOCS, 2007.

Digital Library

[23]

Vladimir Braverman and Rafail Ostrovsky. Generalizing the Layering Method of Indyk and Woodruff: Recursive Sketches for Frequency-Based Vectors on Streams. In APPROX/RANDOM, 2013.

[24]

Amit Chakrabarti, Subhash Khot, and Xiaodong Sun. Near-optimal Lower Bounds on the Multi-party Communication Complexity of Set Disjointness. In IEEE CCC, 2003.

[25]

Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding Frequent Items in Data Streams. In ICALP, 2002.

Digital Library

[26]

Xiaoqi Chen, Shir Landau Feibish, Yaron Koral, Jennifer Rexford, and Ori Rottenstreich. Catching the Microburst Culprits with Snappy. In Proceedings of the Afternoon Workshop on Self-Driving Networks, SelfDN 2018, 2018.

[27]

Edith Cohen. All-Distances Sketches, Revisited: HIP Estimators for Massive Graphs Analysis. IEEE Trans. Knowl. Data Eng., 2015.

[28]

Graham Cormode and Marios Hadjieleftheriou. Methods for Finding Frequent Items in Data Streams. J. VLDB, 2010.

[29]

Graham Cormode and Shan Muthukrishnan. An Improved Data Stream Summary: The Count-min Sketch and Its Applications. J. Algorithms, 2005.

[30]

Mayur Datar, Aristides Gionis, Piotr Indyk, and Rajeev Motwani. Maintaining Stream Statistics over Sliding Windows. SIAM J. Comp., 2002.

[31]

Jisa David and Ciza Thomas. DDoS Attack Detection using Fast Entropy Approach on Flow-based Network Traffic. Procedia Computer Science, 2015.

[32]

G. Einziger, B. Fellman, and Y. Kassner. Independent Counter Estimation Buckets . In IEEE INFOCOM, 2015.

[33]

Cristian Estan, Stefan Savage, and George Varghese. Automatically Inferring Patterns of Resource Consumption in Network Traffic. In ACM SIGCOMM, 2003.

Digital Library

[34]

FD.io. Vector packet processing, 2018.

[35]

Shir Landau Feibish, Yehuda Afek, Anat Bremler-Barr, Edith Cohen, and Michal Shagam. Mitigating DNS Random Subdomain DDoS Attacks by Distinct Heavy Hitters Sketches. In ACM/IEEE HotWeb 2017.

[36]

Laura Feinstein, Dan Schnackenberg, Ravindra Balupari, and Darrell Kindred. Statistical Approaches to DDoS Attack Detection and Response. In Proceedings DARPA information survivability conference and exposition, 2003.

[37]

Éric Fusy and Frédéric Giroire. Estimating the Number of Active Flows in a Data Stream over a Sliding Window. In ANALCO, 2007.

[38]

Moshe Gabel, Daniel Keren, and Assaf Schuster. Anarchists, Unite: Practical Entropy Approximation for Distributed Streams. In ACM KDD, 2017.

[39]

Pedro Garcia-Teodoro, Jesus E. Diaz-Verdejo, Gabriel Macia-Fernandez, and E. Vazquez. Anomaly-Based Network Intrusion Detection: Techniques, Systems and Challenges. Computers and Security, 2009.

[40]

Sangjin Han, Keon Jang, Aurojit Panda, Shoumik Palkar, Dongsu Han, and Sylvia Ratnasamy. SoftNIC: A Software NIC to Augment Hardware. Technical report, 2015.

[41]

Hazar Harmouch and Felix Naumann. Cardinality Estimation: An Experimental Survey. J. VLDB, 2017.

[42]

Stefan Heule, Marc Nunkesser, and Alexander Hall. HyperLogLog in Practice: Algorithmic Engineering of a State of the Art Cardinality Estimation Algorithm. In ACM EDBT, 2013.

Digital Library

[43]

Piotr Indyk and David Woodruff. Optimal Approximations of the Frequency Moments of Data Streams. In ACM STOC, 2005.

Digital Library

[44]

Nikita Ivkin, Edo Liberty, Kevin Lang, Zohar Karnin, and Vladimir Braverman. Streaming quantiles algorithms with small space and update time. arXiv preprint arXiv:1907.00236, 2019.

[45]

Nikita Ivkin, Zhuolong Yu, Vladimir Braverman, and Xin Jin. Qpipe: Quantiles sketch fully in the data plane. In ACM CoNEXT, 2019.

Digital Library

[46]

Atul Kant Kaushik, Emmanuel S. Pilli, and R. C. Joshi. "Network Forensic Analysis by Correlation of Attacks with Network Attributes". In Information and Communication Technologies, 2010.

[47]

Ilan Kremer, Noam Nisan, and Dana Ron. On Randomized One-round Communication Complexity. Computational Complexity, 1999.

[48]

Krishan Kumar, RC Joshi, and Kuldip Singh. A Distributed Approach using Entropy to Detect DDoS Attacks in ISP Domain. In IEEE ICDCS, 2007.

[49]

Xuemin Lin, Hongjun Lu, Jian Xu, and Jeffrey Xu Yu. Continuously Maintaining Quantile Summaries of the Most Recent N Elements over a Data Stream. ICDE, 2004.

Digital Library

[50]

Zaoxing Liu, Ran Ben-Basat, Gil Einziger, Yaron Kassner, Vladimir Braverman, Roy Friedman, and Vyas Sekar. NitroSketch: Robust and General Sketch-based Monitoring in Software Switches. In ACM SIGCOMM, 2019.

Digital Library

[51]

Zaoxing Liu, Antonis Manousis, Gregory Vorsanger, Vyas Sekar, and Vladimir Braverman. One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon. In ACM SIGCOMM, 2016.

Digital Library

[52]

A. Metwally, D. Agrawal, and A. El Abbadi. Efficient Computation of Frequent and Top-k Elements in Data Streams. In ICDT, 2005.

[53]

Rui Miao, Hongyi Zeng, Changhoon Kim, Jeongkeun Lee, and Minlan Yu. SilkRoad: Making Stateful Layer-4 Load Balancing Fast and Cheap Using Switching ASICs. In ACM SIGCOMM, 2017.

Digital Library

[54]

Jayadev Misra and David Gries. Finding Repeated Elements. Science of computer programming, 1982.

[55]

Masoud Moshref, Minlan Yu, Ramesh Govindan, and Amin Vahdat. DREAM: Dynamic Resource Allocation for Software-defined Measurement. In ACM SIGCOMM, 2014.

Digital Library

[56]

Michael Müter and Naim Asaj. Entropy-based Anomaly Detection for In-vehicle Networks. In 2011 IEEE Intelligent Vehicles Symposium (IV), 2011.

[57]

Shanmugavelayutham Muthukrishnan. Data streams: Algorithms and applications. Foundations and Trends in TCS, 2005.

[58]

AS Navaz, V Sangeetha, and C Prabhadevi. Entropy based anomaly detection system to prevent DDoS attacks in cloud. arXiv preprint arXiv:1308.6745, 2013.

[59]

George Nychis, Vyas Sekar, David G. Andersen, Hyong Kim, and Hui Zhang. An Empirical Evaluation of Entropy-based Traffic Anomaly Detection. In ACM IMC, 2008.

Digital Library

[60]

Odysseas Papapetrou, Minos Garofalakis, and Antonios Deligiannakis. Sketching Distributed Sliding-window Data Streams. The VLDB Journal, 2015.

[61]

Ben Pfaff, Justin Pettit, Teemu Koponen, Ethan Jackson, Andy Zhou, Jarno Rajahalme, Jesse Gross, Alex Wang, Joe Stringer, Pravin Shelar, Keith Amidon, and Martin Casado. The Design and Implementation of Open vSwitch. In USENIX NSDI, 2015.

Digital Library

[62]

Vyas Sekar, Nick G Duffield, Oliver Spatscheck, Jacobus E van der Merwe, and Hui Zhang. LADS: Large-scale Automated DDoS Detection System. In USENIX ATC, 2006.

[63]

Haya Shulman and Michael Waidner. Towards Forensic Analysis of Attacks with DNSSEC. In IEEE SPW, 2014.

Digital Library

[64]

Zhewei Wei, Ge Luo, Ke Yi, Xiaoyong Du, and Ji-Rong Wen. Persistent data sketching. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 795--810. ACM, 2015.

Digital Library

[65]

Li Yang, Wu Hao, Pan Tian, Dai Huichen, Lu Jianyuan, and Liu Bin. CASE: Cache-assisted Stretchable Estimator for High Speed Per-flow Measurement. In IEEE INFOCOM, 2016.

[66]

Sen Yang, Bill Lin, and Jun Xu. Safe Randomized Load-Balanced Switching By Diffusing Extra Loads. ACM Meas. Anal. Comput. Syst., 2007.

[67]

Ke Yi and Qin Zhang. Optimal Tracking of Distributed Heavy Hitters and Quantiles. Algorithmica, 2013.

Digital Library

Cited By

Cornacchia ABianchi GBianco AGiaccone P(2022)Staggered HLL: Near-continuous-time cardinality estimation with no overheadComputer Communications10.1016/j.comcom.2022.06.038193(168-175)Online publication date: Sep-2022
https://doi.org/10.1016/j.comcom.2022.06.038
Liu ZNamkung HAgarwal AManousis ASteenkiste PSeshan SSekar V(2021)Sketchy With a Chance of Adoption: Can Sketch-Based Telemetry Be Ready for Prime Time?2021 IEEE 7th International Conference on Network Softwarization (NetSoft)10.1109/NetSoft51509.2021.9492582(9-16)Online publication date: 28-Jun-2021
https://doi.org/10.1109/NetSoft51509.2021.9492582
Rothchild DPanda AUllah EIvkin NStoica IBraverman VGonzalez JArora RDaumé HSingh A(2020)FetchSGDProceedings of the 37th International Conference on Machine Learning10.5555/3524938.3525702(8253-8265)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.5555/3524938.3525702
Show More Cited By

Index Terms

I Know What You Did Last Summer: Network Monitoring using Interval Queries

Recommendations

Sketching distributed sliding-window data streams

While traditional data management systems focus on evaluating single, ad hoc queries over static data sets in a centralized setting, several emerging applications require (possibly, continuous) answers to queries on dynamic data that is widely ...
I Know What You Did Last Summer: Network Monitoring using Interval Queries

Modern telemetry systems require advanced analytic capabilities such as drill down queries. These queries can be used to detect the beginning and end of a network anomaly by efficiently refining the search space. We present the first integral solution ...
Summarization Method for Multiple Sliding Window Aggregate Queries
STFSSD '09: Proceedings of the 2009 Software Technologies for Future Dependable Distributed Systems

The data stream management systems use continuous queries. The continuous query has a independent queue per each query. This situation causes to use much duplicate memories. To solve this problem, there are two resource sharing methods which are the ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Measurement and Analysis of Computing Systems

Proceedings of the ACM on Measurement and Analysis of Computing Systems Volume 3, Issue 3

SIGMETRICS

December 2019

525 pages

EISSN:2476-1249

DOI:10.1145/3376928

Editors:
Augustin Chaintreau
Columbia University
,
Leana Golubchik
University of Southern California
,
Zhi-Li Zhang
University of Minnesota

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 December 2019

Published in POMACS Volume 3, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Israeli Science Foundation grant
Israel Cyber Directorate
NSF CAREER
the Lynne and William Frankel Center for Computing Science at Ben-Gurion University
National Science Foundation
the Cyber Security Research Center
Office of Naval Research
DARPA/MTO
Zuckerman Foundation
Technion Hiroshi Fujiwara Cyber Security Research Center

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
657
Total Downloads

Downloads (Last 12 months)89
Downloads (Last 6 weeks)20

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cornacchia ABianchi GBianco AGiaccone P(2022)Staggered HLL: Near-continuous-time cardinality estimation with no overheadComputer Communications10.1016/j.comcom.2022.06.038193(168-175)Online publication date: Sep-2022
https://doi.org/10.1016/j.comcom.2022.06.038
Liu ZNamkung HAgarwal AManousis ASteenkiste PSeshan SSekar V(2021)Sketchy With a Chance of Adoption: Can Sketch-Based Telemetry Be Ready for Prime Time?2021 IEEE 7th International Conference on Network Softwarization (NetSoft)10.1109/NetSoft51509.2021.9492582(9-16)Online publication date: 28-Jun-2021
https://doi.org/10.1109/NetSoft51509.2021.9492582
Rothchild DPanda AUllah EIvkin NStoica IBraverman VGonzalez JArora RDaumé HSingh A(2020)FetchSGDProceedings of the 37th International Conference on Machine Learning10.5555/3524938.3525702(8253-8265)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.5555/3524938.3525702
Ivkin NBen Basat RLiu ZEinziger GFriedman RBraverman V(2020)I Know What You Did Last SummerACM SIGMETRICS Performance Evaluation Review10.1145/3410048.341008448:1(61-62)Online publication date: 9-Jul-2020
https://dl.acm.org/doi/10.1145/3410048.3410084
Qiu YHsu KXing JChen A(2020)A Feasibility Study on Time-aware Monitoring with Commodity SwitchesProceedings of the Workshop on Secure Programmable Network Infrastructure10.1145/3405669.3405821(22-27)Online publication date: 10-Aug-2020
https://dl.acm.org/doi/10.1145/3405669.3405821
Ivkin NBasat RLiu ZEinziger GFriedman RBraverman VYeh EMarkopoulou ATay Y(2020)I Know What You Did Last SummerAbstracts of the 2020 SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer Systems10.1145/3393691.3394193(61-62)Online publication date: 8-Jun-2020
https://dl.acm.org/doi/10.1145/3393691.3394193
Yang MZhang JGadre ALiu ZKumar SSekar V(2020)JoltikProceedings of the 26th Annual International Conference on Mobile Computing and Networking10.1145/3372224.3419204(1-14)Online publication date: 16-Apr-2020
https://dl.acm.org/doi/10.1145/3372224.3419204
Saidi SMaghsoudlou AFoucard DSmaragdakis GPoese IFeldmann A(2020)Exploring Network-Wide Flow Data With FlowyagerIEEE Transactions on Network and Service Management10.1109/TNSM.2020.303427817:4(1988-2006)Online publication date: Dec-2020
https://doi.org/10.1109/TNSM.2020.3034278
Wei VIvkin NBraverman VSzalay A(2020)Sketch and Scale Geo-distributed tSNE and UMAP2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9377843(996-1003)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9377843

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents