Article

The network data handling war: MySQL vs. NfDump

Authors:

Aiko PrasAuthors Info & Claims

EUNICE'10: Proceedings of the 16th EUNICE/IFIP WG 6.6 conference on Networked services and applications: engineering, control and management

Pages 167 - 176

Published: 28 June 2010 Publication History

Abstract

Network monitoring plays a crucial role in any network management environment. Especially nowadays, with network speed and load constantly increasing, more and more data needs to be collected and efficiently processed. In highly interactive network monitoring systems, a quick response time from information sources turns out to be a crucial requirement. However, for data sets in the order of several GBs, this goal becomes difficult to achieve. In this paper, we present our operational experience in dealing with large amounts of network data. In particular, we focus on MySQL and NfDump, testing their capabilities under different usage scenarios and increasing data set sizes.

References

[1]

Steinder, M., Sethi, A.S.: A survey of fault localization techniques in computer networks. Science of Computer Programming 53(2), 165-194 (2004).

[2]

Casey, E.: Network traffic as a source of evidence: tool strengths, weaknesses, and future needs. Digital Investigation 1(1), 28-43 (2004).

Digital Library

[3]

Tcpdump/libpcap (April 2010), http://www.tcpdump.org/

[4]

Claise, B.: Cisco Systems NetFlow Services Export Version 9. RFC 3954, Informational (2004).

[5]

MySQL (April 2010), http://www.mysql.com/

[6]

PostgreSQL (April 2010), http://www.postgresql.org/

[7]

Tcptrace (April 2010), http://www.tcptrace.org/

[8]

NfDump (April 2010), http://nfdump.sourceforge.net/

[9]

Liu, X., Heo, J., Sha, L.: Modeling 3-Tiered Web Applications. In: Proc. of the 13th IEEE Int. Symp. on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 307-310 (2005).

Digital Library

[10]

Hofstede, R., Fioreze, T.: SURFmap: A Network Monitoring Tool Based on the Google Maps API. In: Application session proc. of the 11th IFIP/IEEE Int. Symp. on Integrated Network Management, pp. 676-690. IEEE Computer Society Press, Los Alamitos (2009).

Digital Library

[11]

Li, Y., Slagell, A., Luo, K., Yurcik, W.: CANINE: A combined conversion and anonymization tool for processing NetFlows for security. In: Proc. of 10th Int. Conf. on Telecommunication Systems, Modeling and Analysis (2005).

[12]

Minarik, P., Dymacek, T.: NetFlow Data Visualization Based on Graphs. In: Goodall, J.R., Conti, G., Ma, K.-L. (eds.) VizSec 2008. LNCS, vol. 5210, pp. 144-151. Springer, Heidelberg (2008).

Digital Library

[13]

University of Twente (April 2010), http://www.utwente.nl

[14]

Siekkinen, M., Biersack, E.W., Urvoy-Keller, G., Goebel, V., Plagemann, T.: In-TraBase: Integrated traffic analysis based on a database management system. In: Proc. of the End-to-End Monitoring Techniques and Services, Washington, DC, USA, pp. 32-46. IEEE Computer Society, Los Alamitos (2005).

Digital Library

[15]

Kobayashi, A., Matsubara, D., Kimura, S., Saitou, M., Hirokawa, Y., Sakamoto, H., Ishibashi, K., Yamamoto, K.: A Proposal of Large-Scale Traffic Monitoring System Using Flow Concentrators. In: Kim, Y.-T., Takano, M. (eds.) APNOMS 2006. LNCS, vol. 4238, pp. 53-62. Springer, Heidelberg (2006).

Digital Library

[16]

Lim, K.S., Stadler, R.: Real-time views of network traffic using decentralized management. In: Proc. of the 9th IFIP/IEEE Int. Symp. on Integrated Network Management, Nice, France, pp. 119-132 (2005).

[17]

Quittek, J., Zseby, T., Claise, B., Zander, S.: Requirements for IP Flow Information Export (IPFIX). RFC 3917, Informational (2004).

[18]

Schwartz, B., Zaitsev, P., Tkachenko, V., Zawodny, J., Lentz, A., Balling, D.J.: High performance MySQL, 2nd edn. O'Reilly, Sebastopol (2008).

Digital Library

Cited By

Fusco FVlachos MDimitropoulos XByers JKurose JMahajan RSnoeren A(2012)RasterZipProceedings of the 2012 Internet Measurement Conference10.1145/2398776.2398783(51-64)Online publication date: 14-Nov-2012
https://dl.acm.org/doi/10.1145/2398776.2398783
Velan PKrejčí R(2012)Flow information storage assessment using IPFIXcolProceedings of the 6th IFIP WG 6.6 international autonomous infrastructure, management, and security conference on Dependable Networks and Services10.1007/978-3-642-30633-4_21(155-158)Online publication date: 4-Jun-2012
https://dl.acm.org/doi/10.1007/978-3-642-30633-4_21

Index Terms

The network data handling war: MySQL vs. NfDump
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Networks
  1. Network services
    1. Network monitoring

Index terms have been assigned to the content through auto-classification.

Recommendations

Composition attack against social network data

This paper deals with applying a composition attack to anonymized social network data.A new algorithm for the composition attack is proposed.Experiments on synthetic scale-free networks show the usability of the algorithm.The algorithm pairs rightly 20%...
Performance evaluation of a hybrid IP/SDN network in data centre network architectures

The current Internet is facing many challenges to accommodate the growing demands of cloud computing. Software‐defined networking (SDN) is an emerging paradigm, proposed to be used in this context in order to investigate these issues. SDN virtualises ...
Handling data quality in entity resolution
IQIS '05: Proceedings of the 2nd international workshop on Information quality in information systems

Entity resolution (ER) is a problem that arises in many information integration scenarios: We have two or more sources containing records on the same set of real-world entities (e.g., customers).However, there are no unique identifiers that tell us what ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

EUNICE'10: Proceedings of the 16th EUNICE/IFIP WG 6.6 conference on Networked services and applications: engineering, control and management

June 2010

274 pages

ISBN:3642139701

Editors:
Finn Arve Aagesen
Norwegian University of Science and Technology, Department of Telematics, Trondheim, Norway
,
Svein Johan Knapskog
Norwegian University of Science and Technology, Centre for Quantifiable Quality of Service in Communication Systems, Trondheim, Norway

Sponsors

UNINETT
NTNU: Norwegian University of Science and Technology

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 28 June 2010

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fusco FVlachos MDimitropoulos XByers JKurose JMahajan RSnoeren A(2012)RasterZipProceedings of the 2012 Internet Measurement Conference10.1145/2398776.2398783(51-64)Online publication date: 14-Nov-2012
https://dl.acm.org/doi/10.1145/2398776.2398783
Velan PKrejčí R(2012)Flow information storage assessment using IPFIXcolProceedings of the 6th IFIP WG 6.6 international autonomous infrastructure, management, and security conference on Dependable Networks and Services10.1007/978-3-642-30633-4_21(155-158)Online publication date: 4-Jun-2012
https://dl.acm.org/doi/10.1007/978-3-642-30633-4_21

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents