research-article

pcapIndex: an index for network packet traces with legacy compatibility

Authors:

Francesco Fusco,

Xenofontas Dimitropoulos,

Michail Vlachos,

Luca DeriAuthors Info & Claims

ACM SIGCOMM Computer Communication Review, Volume 42, Issue 1

Pages 47 - 53

https://doi.org/10.1145/2096149.2096156

Published: 16 January 2012 Publication History

Get Access

Abstract

Long-term historical analysis of captured network traffic is a topic of great interest in network monitoring and network security. A critical requirement is the support for fast discovery of packets that satisfy certain criteria within large-scale packet repositories. This work presents the first indexing scheme for network packet traces based on compressed bitmap indexing principles. Our approach supports very fast insertion rates and results in compact index sizes. The proposed indexing methodology builds upon libpcap, the de-facto reference library for accessing packet-trace repositories. Our solution is therefore backward compatible with any solution that uses the original library. We experience impressive speedups on packet-trace search operations: our experiments suggest that the index-enabled libpcap may reduce the packet retrieval time by more than 1100 times.

References

[1]

Endace Measurement Systems. http://www.endace.com.

Google Scholar

[2]

TCPDUMP/LIBPCAP public repository. http://www.tcpdump.org/.

Google Scholar

[3]

E. W. Bethel, S. Campbell, E. Dart, K. Stockinger, and K. Wu. Accelerating Network Traffic Analysis Using Query-Driven Visualization. In Proc. of 2006 IEEE Symposium on Visual Analytics Science and Technology, pages 115--122, 2006.

Crossref

Google Scholar

[4]

F. Deliàge and T. B. Pedersen. Position list word aligned hybrid: optimizing space and performance for compressed bitmaps. In Proc. of the 13th Int. Conf. on Extending Database Technology, pages 228--239, 2010.

Digital Library

Google Scholar

[5]

L. Deri, V. Lorenzetti, and S. Mortimer. Collection and Exploration of Large Data Monitoring Sets Using Bitmap Databases. In 2nd Int. Workshop on Traffic Monitoring and Analysis(TMA), pages 73--86, 2010.

Digital Library

Google Scholar

[6]

P. J. Desnoyers and P. Shenoy. Hyperion: high volume stream archival for retrospective querying. In Proc. of the USENIX Annual Technical Conf., 2007.

Digital Library

Google Scholar

[7]

F. Fusco, M. P. Stoecklin, and M. Vlachos. Net-fli: on-the-fly compression, archiving and indexing of streaming network traffic. Proc. VLDB Endow., 3, 2010.

Digital Library

Google Scholar

[8]

C. R. Kalmanek et al. Darkstar: Using exploratory data mining to raise the bar on network reliability and performance. In 7th Int. Workshop on Design of Reliable Communication Networks, 2009.

Crossref

Google Scholar

[9]

G. Maier, R. Sommer, H. Dreger, A. Feldmann, V. Paxson, and F. Schneider. Enriching network security analysis with time travel. In Proc. of the ACM SIGCOMM 2008 Conf. on Data communication, pages 183--194, 2008.

Digital Library

Google Scholar

[10]

S. McCanne and V. Jacobson. The BSD packet filter: a new architecture for user-level packet capture. In Proc. of the USENIX Winter Conf., pages 2--2, 1993.

Digital Library

Google Scholar

[11]

K. Wu, E. Otoo, and A. Shoshani. On the Performance of Bitmap Indices for High Cardinality Attributes. In Proc. of the 13th Int. Conf. on Very Large Data Dases (VLDB), pages 24--35, 2004.

Digital Library

Google Scholar

[12]

K. Wu, E. J. Otoo, and A. Shoshani. Optimizing bitmap indices with efficient compression. ACM Trans. Database Syst., 31:1--38, March 2006.

Digital Library

Google Scholar

Cited By

View all

Swarnkar MKumar RBaidyo RG H(2023)LiteEx: A Lightweight Feature Extraction Tool for Captured Network Traces2023 15th International Conference on COMmunication Systems & NETworkS (COMSNETS)10.1109/COMSNETS56262.2023.10041389(243-251)Online publication date: 3-Jan-2023
https://doi.org/10.1109/COMSNETS56262.2023.10041389
Jiang CWang JLi Y(2021)An Efficient Indexing Scheme for Network Traffic Collection and Retrieval SystemElectronics10.3390/electronics1002019110:2(191)Online publication date: 15-Jan-2021
https://doi.org/10.3390/electronics10020191
Wang YZhang GJiang HXie GCarle GOtt J(2021)Compact-indexProceedings of the 17th International Conference on emerging Networking EXperiments and Technologies10.1145/3485983.3494858(90-103)Online publication date: 2-Dec-2021
https://dl.acm.org/doi/10.1145/3485983.3494858
Show More Cited By

Index Terms

pcapIndex: an index for network packet traces with legacy compatibility
1. Networks
  1. Network services
    1. Network monitoring

Recommendations

PcapWT

Network packet tracing has been used for many different purposes during the last few decades, such as network software debugging, networking performance analysis, forensic investigation, and so on. Meanwhile, the size of packet traces becomes larger, as ...
SQUID: a practical 100% throughput scheduler for crosspoint buffered switches

Crosspoint buffered switches are emerging as the focus of research in high-speed routers. They have simpler scheduling algorithms and achieve better performance than bufferless crossbar switches. Crosspoint buffered switches have a buffer at each ...
Excess buffer requirement for EPD schemes in ATM networks

It is known that the performance of TCP over ATM can be significantly degraded if the bandwidth is occupied by cells belonging to packets that are already corrupted by cell loss due to buffer overflow. The Early Packet Discard (EPD) mechanism is a well ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review

ACM SIGCOMM Computer Communication Review Volume 42, Issue 1

January 2012

88 pages

ISSN:0146-4833

DOI:10.1145/2096149

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 January 2012

Published in SIGCOMM-CCR Volume 42, Issue 1

Check for updates

Author Tag

packet indexing

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
316
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Swarnkar MKumar RBaidyo RG H(2023)LiteEx: A Lightweight Feature Extraction Tool for Captured Network Traces2023 15th International Conference on COMmunication Systems & NETworkS (COMSNETS)10.1109/COMSNETS56262.2023.10041389(243-251)Online publication date: 3-Jan-2023
https://doi.org/10.1109/COMSNETS56262.2023.10041389
Jiang CWang JLi Y(2021)An Efficient Indexing Scheme for Network Traffic Collection and Retrieval SystemElectronics10.3390/electronics1002019110:2(191)Online publication date: 15-Jan-2021
https://doi.org/10.3390/electronics10020191
Wang YZhang GJiang HXie GCarle GOtt J(2021)Compact-indexProceedings of the 17th International Conference on emerging Networking EXperiments and Technologies10.1145/3485983.3494858(90-103)Online publication date: 2-Dec-2021
https://dl.acm.org/doi/10.1145/3485983.3494858
D'Alconzo ADrago IMorichetta AMellia MCasas P(2019)A Survey on Big Data for Network Traffic Monitoring and AnalysisIEEE Transactions on Network and Service Management10.1109/TNSM.2019.293335816:3(800-813)Online publication date: Sep-2019
https://doi.org/10.1109/TNSM.2019.2933358
Piskozub MSpolaor RMartinovic I(2019)CompactFlow: A Hybrid Binary Format for Network Flow DataInformation Security Theory and Practice10.1007/978-3-030-41702-4_12(185-201)Online publication date: 11-Dec-2019
https://dl.acm.org/doi/10.1007/978-3-030-41702-4_12
Vallentin MPaxson VSommer RArgyraki KIsaacs R(2016)VASTProceedings of the 13th Usenix Conference on Networked Systems Design and Implementation10.5555/2930611.2930634(345-362)Online publication date: 16-Mar-2016
https://dl.acm.org/doi/10.5555/2930611.2930634
Lee JLee SLee JYi YPark KLu SRiedel E(2015)FloSISProceedings of the 2015 USENIX Conference on Usenix Annual Technical Conference10.5555/2813767.2813800(445-457)Online publication date: 8-Jul-2015
https://dl.acm.org/doi/10.5555/2813767.2813800
Kim YKonow RDujovne DTurletti TDabbous WNavarro G(2015)PcapWTComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2014.12.00779:C(91-102)Online publication date: 14-Mar-2015
https://dl.acm.org/doi/10.1016/j.comnet.2014.12.007
Choi SPark HLee JKim JKim I(2015)Performance Comparison of Relational Databases and Columnar Databases Using Bitmap Index for Fast Search of 10Gbps Network FlowsAdvances in Computer Science and Ubiquitous Computing10.1007/978-981-10-0281-6_25(171-175)Online publication date: 18-Dec-2015
https://doi.org/10.1007/978-981-10-0281-6_25
Fusco FVlachos MDimitropoulos XDeri LPapagiannaki KGummadi KPartridge C(2013)Indexing million of packets per second using GPUsProceedings of the 2013 conference on Internet measurement conference10.1145/2504730.2504756(327-332)Online publication date: 23-Oct-2013
https://dl.acm.org/doi/10.1145/2504730.2504756
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

PcapWT

SQUID: a practical 100% throughput scheduler for crosspoint buffered switches

Excess buffer requirement for EPD schemes in ATM networks

Comments

Published In

Publisher

Publication History

Check for updates

Author Tag

Qualifiers

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

PcapWT

SQUID: a practical 100% throughput scheduler for crosspoint buffered switches

Excess buffer requirement for EPD schemes in ATM networks

Comments

Information

Published In

Publisher

Publication History

Check for updates

Author Tag

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations