article

Spreader classification based on optimal dynamic bit sharing

Authors:

Yan QiaoAuthors Info & Claims

IEEE/ACM Transactions on Networking (TON), Volume 21, Issue 3

Pages 817 - 830

https://doi.org/10.1109/TNET.2012.2218255

Published: 01 June 2013 Publication History

Abstract

Spreader classification is an online traffic measurement function that has many important applications. In order to keep up with ever-higher line speed, the recent research trend is to implement such functions in fast but small on-die SRAM. However, the mismatch between the huge amount of Internet traffic to be monitored and limited on-die memory space presents a significant technical challenge. In this paper, we propose an Efficient Spreader Classification (ESC) scheme based on dynamic bit sharing, a compact information storage method. We design a maximum likelihood estimation method to extract per-source information from the compact storage and determine the heavy spreaders. Our new scheme ensures that false positive/negative ratios are bounded. Moreover, given an arbitrary set of bounds, we develop a systematic approach to determine the optimal system parameters that minimize the amount of memory needed to meet the bounds. Experiments based on a real Internet traffic trace demonstrate that the proposed spreader classification scheme reduces memory consumption by 3-20 times when compared to the best existing work. We also investigate a new multi-objective spreader classification problem and extend our classification scheme to solve it.

References

[1]

W. D. Gardner, "Researchers transmit optical data at 16.4 Tbps," Inf. Week, Feb. 2008 [Online]. Available: http://www.informationweek.com/researchers-transmit-optical-data-at-164/206900603

[2]

H. Song, F. Hao, M. Kodialam, and T. Lakshman, "IPv6 lookups using distributed and load balanced bloom filters for 100 Gbps core router line cards," in Proc. IEEE INFOCOM, Apr. 2009, pp. 2518-2526.

[3]

C. Hermsmeyer, H. Song, R. Schlenk, R. Gemelli, and S. Bunse, "Towards 100G packet processing: Challenges and technologies," Bell Labs Tech. J., vol. 14, no. 2, pp. 57-80, 2009.

[4]

Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani, "Counter braids: A novel counter architecture for per-flow measurement," in Proc. ACM SIGMETRICS, Jun. 2008, pp. 121-132.

[5]

Q. Zhao, J. Xu, and A. Kumar, "Detection of super sources and destinations in high-speed networks: Algorithms, analysis and evaluation," IEEE J. Sel. Areas Commun., vol. 24, no. 10, pp. 1840-1852, Oct. 2006.

[6]

M. Yoon, T. Li, S. Chen, and J. Peir, "Fit a spread estimator in small memory," in Proc. IEEE INFOCOM, Apr. 2009, pp. 504-512.

[7]

Q. Zhao, J. Xu, and Z. Liu, "Design of a novel statistics counter architecture with optimal space and time efficiency," in Proc. ACM SIGMETRICS/Perform. , 2006, pp. 323-334.

[8]

S. Venkatataman, D. Song, P. Gibbons, and A. Blum, "New streaming algorithms for fast detection of superspreaders," in Proc. NDSS, Feb. 2005, p. 6.

[9]

K. Hwang, B. Vander-Zanden, and H. Taylor, "A linear-time probabilistic counting algorithm for database applications," Trans. Database Syst., vol. 15, no. 2, pp. 208-229, Jun. 1990.

[10]

C. Estan, G. Varghese, and M. Fish, "Bitmap algorithms for counting active flows on high-speed links," IEEE/ACM Trans. Netw., vol. 14, no. 5, pp. 925-937, Oct. 2006.

[11]

J. Cao, Y. Jin, A. Chen, T. Bu, and Z. Zhang, "Identifying high cardinality internet hosts," in Proc. IEEE INFOCOM, Apr. 2009, pp. 810-818.

[12]

S. Staniford, J. Hoagland, and J. McAlerney, "Practical automated detection of stealthy portscans," J. Comput. Secur., vol. 10, pp. 105-136, 2002.

[13]

D. Plonka, "FlowScan: A network traffic flow reporting and visualization tool," in Proc. USENIX LISA, 2000, pp. 305-318.

[14]

T. Li, S. Chen, and Y. Ling, "Fast and compact per-flow traffic measurement through randomized counter sharing," in Proc. IEEE INFOCOM, 2011, pp. 1799-1807.

[15]

D. Shah, S. Iyer, B. Prabhakar, and N. McKeown, "Maintaining statistics counters in router line cards," IEEE Micro, vol. 22, no. 1, pp. 76-81, Jan.-Feb. 2002.

[16]

S. Ramabhadran and G. Varghese, "Efficient implementation of a statistics counter architecture," in Proc. ACM SIGMETRICS, Jun. 2003, pp. 261-271.

[17]

Y. Lu and B. Prabhakar, "Robust counting via counter braids: An error-resilient network measurement architecture," in Proc. IEEE INFOCOM, Apr. 2009, pp. 522-530.

[18]

T. Li, S. Chen, and Y. Qiao, "Origin-destination flow measurement in high-speed networks," in Proc. IEEE INFOCOM, Mini-Conf., 2012, pp. 2526-2530.

[19]

M. Charikar,K. Chen, and M. Farach-Colton, "Finding frequent items in data streams," in Proc. ICALP, Jul. 2002, p. 784.

[20]

G. Cormode and S. Muthukrishnan, "Space efficient mining of multigraph streams," in Proc. ACM PODS, Jun. 2005, pp. 271-282.

[21]

E. Demaine, A. Lopez-Ortiz, and J. Ian-Munro, "Frequency estimation of internet packet streams with limited space," in Proc. ESA, Sep. 2002, pp. 348-360.

[22]

X. Dimitropoulos, P. Hurley, and A. Kind, "Probabilistic lossy counting: An efficient algorithm for finding heavy hitters," Comput. Commun. Rev., vol. 38, no. 1, pp. 7-16, 2008.

[23]

C. Estan and G. Varghese, "New directions in traffic measurement and accounting," in Proc. ACM SIGCOMM, Aug. 2002, pp. 323-336.

[24]

P. Gibbons and Y. Matias, "New sampling-based summary statistics for improving approximate query answers," in Proc. ACM SIGMOD, Jun. 1998, pp. 331-342.

[25]

G. Manku and R. Motwani, "Approximate frequency counts over data streams," in Proc. VLDB, Aug. 2002, pp. 346-357.

[26]

Y. Zhang, S. Singh, S. Sen, N. Duffield, and C. Lund, "Online identification of hierarchical heavy hitters: Algorithms, evaluation, and application," in Proc. ACM SIGCOMM IMC, Oct. 2004, pp. 101-114.

[27]

N. Bandi, D. Agrawal, and A. Abbadi, "Fast algorithms for heavy distinct hitters using associative memories," in Proc. IEEE ICDCS, Jun. 2007, p. 6.

[28]

Q. Zhao, A. Kumar, and J. Xu, "Joint data streaming and sampling techniques for detection of super sources and destinations," in Proc. USENIX/ACM Internet Meas. Conf., Oct. 2005, p. 7.

[29]

B. H. Bloom, "Space/time trade-offs in hash coding with allowable errors," Commun. ACM, vol. 13, no. 7, pp. 422-426, 1970.

[30]

A. Broder and M. Mitzenmacher, "Network applications of Bloom filters: A survey," Internet Math., vol. 1, no. 4, pp. 485-509, 2002.

Cited By

Han HJing XYan ZPedrycz W(2023)ExtendedSketch+Information Fusion10.1016/j.inffus.2022.12.00992:C(300-312)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1016/j.inffus.2022.12.009
Han HYan ZJing XPedrycz W(2022)Applications of sketches in network traffic measurementInformation Fusion10.1016/j.inffus.2021.12.00782:C(58-85)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1016/j.inffus.2021.12.007
Tang LHuang QLee P(2020)SpreadSketch: Toward Invertible and Network-Wide Detection of SuperspreadersIEEE INFOCOM 2020 - IEEE Conference on Computer Communications10.1109/INFOCOM41043.2020.9155541(1608-1617)Online publication date: 6-Jul-2020
https://dl.acm.org/doi/10.1109/INFOCOM41043.2020.9155541
Show More Cited By

Index Terms

Spreader classification based on optimal dynamic bit sharing

Recommendations

Modeling, Architecture, and Applications for Emerging Memory Technologies

Editor's note:Spin-transfer torque RAM and phase-change RAM are vying to become the next-generation embedded memory, offering high speed, high density, and nonvolatility. This article discusses new opportunities and challenges presented by these two ...
Embedded Memories: Progress and a Look into the Future

Memories are categorized as embedded memories (e-memories) and stand-alone memories. E-memories favor high speed rather than low cost. In addition, they must maintain compatibility with the logic process, because they must be cofabricated on the same ...
Traffic measurement and analysis in an ATM-based internet backbone

This paper reports our measurements and analysis of traffic characteristics in an Internet backbone ATM network. In order to utilize network resource efficiently while satisfying the quality of service requirement, it is important to understand the ...

Comments

Information & Contributors

Information

Published In

cover image IEEE/ACM Transactions on Networking

IEEE/ACM Transactions on Networking Volume 21, Issue 3

June 2013

336 pages

ISSN:1063-6692

Issue’s Table of Contents

Publisher

IEEE Press

Publication History

Published: 01 June 2013

Accepted: 23 July 2012

Revised: 21 June 2012

Received: 16 May 2011

Published in TON Volume 21, Issue 3

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
116
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Han HJing XYan ZPedrycz W(2023)ExtendedSketch+Information Fusion10.1016/j.inffus.2022.12.00992:C(300-312)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1016/j.inffus.2022.12.009
Han HYan ZJing XPedrycz W(2022)Applications of sketches in network traffic measurementInformation Fusion10.1016/j.inffus.2021.12.00782:C(58-85)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1016/j.inffus.2021.12.007
Tang LHuang QLee P(2020)SpreadSketch: Toward Invertible and Network-Wide Detection of SuperspreadersIEEE INFOCOM 2020 - IEEE Conference on Computer Communications10.1109/INFOCOM41043.2020.9155541(1608-1617)Online publication date: 6-Jul-2020
https://dl.acm.org/doi/10.1109/INFOCOM41043.2020.9155541
Wang JLiu WZheng LLi ZLiu Z(2019)A novel algorithm for detecting superpoints based on reversible virtual bitmapsJournal of Information Security and Applications10.1016/j.jisa.2019.10240349:COnline publication date: 1-Dec-2019
https://dl.acm.org/doi/10.1016/j.jisa.2019.102403
Xu JDing WHu XGong Q(2019) VATEComputer Communications10.1016/j.comcom.2019.02.005138:C(20-31)Online publication date: 15-Apr-2019
https://dl.acm.org/doi/10.1016/j.comcom.2019.02.005
Xiao QChen SZhou YChen MLuo JLi TLing Y(2017)Cardinality Estimation for Elephant FlowsIEEE/ACM Transactions on Networking10.1109/TNET.2017.275384225:6(3738-3752)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1109/TNET.2017.2753842
Liu WQu WGong JLi K(2016)Detection of Superpoints Using a Vector Bloom FilterIEEE Transactions on Information Forensics and Security10.1109/TIFS.2015.250326911:3(514-527)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1109/TIFS.2015.2503269
Xiao QChen SChen MLing Y(2015)Hyper-Compact Virtual Estimators for Big Network Data Based on Register SharingACM SIGMETRICS Performance Evaluation Review10.1145/2796314.274587043:1(417-428)Online publication date: 15-Jun-2015
https://dl.acm.org/doi/10.1145/2796314.2745870
Xiao QChen SChen MLing YLin BXu JSengupta SShah D(2015)Hyper-Compact Virtual Estimators for Big Network Data Based on Register SharingProceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems10.1145/2745844.2745870(417-428)Online publication date: 15-Jun-2015
https://dl.acm.org/doi/10.1145/2745844.2745870

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents