Long term data storage issues for situational awareness
Author: 
John McHughAuthors Info & Claims
John McHughAuthors Info & ClaimsArticle No.: 14, Pages 1 - 4
Abstract
Network traffic archives are useful for a number of purposes ranging from forensic studies to retrospective studies of the evolution of network traffic characteristics. The sheer volume of data that might be useful, if retained, imposes stresses on data storage and management systems. This is exacerbated by the fact that a substantial portion of network traffic is essentially noise and is interesting primarily at an aggregate level as the archive ages, while the remainder may remain interesting at the packet or flow level for an indefinite period. This paper discusses two cases, high volume scans and very infrequent traffic, where lossy compression may be applied to make substantial reductions in the volume of data retained while minimizing the risk of loosing interesting records. In addition, it discusses data structures, based of space and time efficient hashing methods that can be used to index network data using very large, sparse, index spaces such as those presented by IPv6 or by connection tuples that contain multiple IP addresses, along with service and protocol information.
References
[1]
]]B. H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422--426, 1970.
[2]
]]Fabiano C. Botelho and Nivio Ziviani. External perfect hashing for very large key sets. In Proceedings of CIKM'07, Lisboa, Portugal, 2007. ACM.
[3]
]]Carrie Gates, Michael Collins, Michael Duggan, Andrew Kompanek, and Mark Thomas. More NetFlow tools for performance and security. In Proceedings of the 18th Large Installation Systems Administration Conference (LISA 2004), pages 121--132, 2004.
[4]
]]Carrie Gates and John McHugh. The contact surface: A technique for exploring internet scale emergent behaviors. In Diego Zamboni, editor, DIMVA, volume 5137 of Lecture Notes in Computer Science, pages 228--246. Springer, 2008.
[5]
]]John McHugh. Sets, bags and rock and roll; analyzing large sets of network data. In Proceedings of ESORICS 2004, LNCS, pages 407--422. Springer, 2004.
[6]
]]Úlfar Erlingsson, Mark Manasse, and Frank McSherry. A cool and practical alternative to traditional hash tables. In Proceedings of the 7th Workshop on Distributed Data and Structures (WDAS'06), Santa Clara, CA, January 2006.
Index Terms
- Long term data storage issues for situational awareness
Recommendations
Situational Awareness in Context
CONTEXT 2013: Proceedings of the 8th International and Interdisciplinary Conference on Modeling and Using Context - Volume 8175In this paper we analyze the relationship between context and situational awareness with the aim to get a better understanding of how context information influences situation assessment. The analysis is based on previous research on situational ...
Cuckoo bags for exploring multikey data
CSIIRW '10: Proceedings of the Sixth Annual Workshop on Cyber Security and Information Intelligence ResearchAs the transition from IPv4 to hybrid IPv4/IPv6 networks begins, our structures for representing IP addresses for analysis are lagging behind. There is a need for tools that can develop behavioral models of network traffic that include communications ...
RasterZip: compressing network monitoring data with support for partial decompression
IMC '12: Proceedings of the 2012 Internet Measurement ConferenceNetwork traffic archival solutions are fundamental for a number of emerging applications that require: a) efficient storage of high-speed streams of traffic records and b) support for interactive exploration of massive datasets. Compression is a ...
Comments
Information & Contributors
Information
Published In
April 2009
952 pages
ISBN:9781605585185
DOI:10.1145/1558607
Copyright © 2009 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 April 2009
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
CSIIRW '09
CSIIRW '09: Fifth Cyber Security and Information Intelligence Research Workshop
April 13 - 15, 2009
Tennessee, Oak Ridge, USA
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 223Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 14 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in