Applying big data warehousing and visualization techniques on pingER data
Pages 67 - 72
Abstract
Nowadays, the Internet has turned into a crucial piece of our cutting edge society. It is a stage of exploration, financial development, democratic participation and speech. The operations of the Internet have prompted a huge development and collection of information known as Big Data. Therefore, it is important to monitor and measure the Quality of Service (QoS) of Internet traffic. The SLAC National Accelerator Laboratory started the PingER project in 1995 to measure the End-to-End Internet performance history of servers and routers worldwide. The project involves measurements of the 700 monitored sites in over 160 countries. PingER Monitoring Agents (MAs) ping a list of monitored sites after every 30 minutes to obtain Round Trip Time (RTT) values revealing interesting information about Internet performance (e.g., RTT, jitter, packet loss and unreachability) major events (e.g., fiber cuts, earthquakes, and social upheavals). Thus, the project has collected a vast amount of historical Internet Performance data worldwide since 1995. Currently, the data is stored in flat text files, making it difficult to analyze collectively. In addition, this simplistic format limits the analytical potential of this data. In this paper, we propose an approach to process, store, analyze and visualize PingER data. A Data warehouse is created which combines Hadoop Big Data techniques. The data are processed by using Sci-cumulus MR workflow, stored in HDFS, analyzed by Impala queries and visualized by using Google API's. This approach makes PingER data more accessible and enhances its potential contribution to ongoing research and application development.
References
[1]
R. L. Cottrell, "Tutorial on Internet Monitoring & PingER at SLAC" {Online} Available at http://pinger.seecs.edu.pk/tutorial/tutorial.html 07 july,2015.
[2]
W. Fan and A. Bifet, "Mining Big Data: Current Status, and Forecast to the Future," Special Interest Group on Knowledge Discovery in Data (SIGKDD), vol. 14, no. 2, pp. 1--5.
[3]
Q. Liu, B. Ribeiro, A. H. Sung, and D. Suryakumar, "Mining the Big Data: The Critical Feature Dimension Problem," 2014 IIAI 3rd Int. Conf. Adv. Appl. Informatics, no. 2010, pp. 499--504, Aug. 2014.
[4]
S.-H. Liao, P.-H. Chu, and P.-Y. Hsiao, "Data mining techniques and applications - A decade review from 2000 to 2011," Expert Syst. Appl., vol. 39, no. 12, pp. 11303--11311, Sep. 2012.
[5]
Y. Qi, "Data Analysis Analysis Visualization in Media Big Data", 2015 IEEE/ACIS 14th Int. Conf. on Comp. and Info. Science (ICIS), pp. 0--3, 2015.
[6]
X. Bai, D. White, and D. Sundaram, "Context adaptive visualization for effective business intelligence," 2013 15th IEEE Int. Conf. Commun. Technol., pp. 786--790, Nov. 2013.
[7]
R. F. Souza, L. Cottrell, B. White, M. L. Campos, and M. Mattoso, "Linked Open Data Publication Strategies: Application in Networking Performance Measurement Data," 2014 ASE BIGDATA/SOCIALCOM/CYBERSECURITY conf. stanford university, pp. 1--7, 2014.
[8]
Nabi, "Implementation of Relational archive site for PingER" {Online}. Available at https://confluence.slac.stanford.edu/display/IEPM/Implementation+of+Relational+archive+site+for+PingER. 26 July, 2011.
[9]
T. M. S. Barbosa, R. Souza, S. M. S. Cruz, M. L. Campos, and R. Les Cottrell, "Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance," 2015 NETAPPS 4th Int. Conf. on Internet Applications, Protocols and Services, pp. 31--36, 2015.
[10]
R. Kimball, "The Data Warehouse Lifecycle Toolkit: Expert Methods for Designing, Developing, and Deploying Data Warehouses Architecture," p. 771, 1998.
[11]
A. Zoss, "Introduction to Data Visualization: Visualization Types" {Online}. Available at http://guides.library.duke.edu/datavis/vis_types. 8 December, 2015.
[12]
S. Chaudhuri, and U. Dayal, "An overview of data warehousing and OLAP technology," ACM SIGMOD Special Interest Group on Management Of Data, vol. 26, no 1, pp. 65--74, 1997.
[13]
X. Wu, X. Zhu, and S. Member, "Data Mining with Big Data," IEEE trans. on knowl. and data eng., vol. 26, no. 1, pp. 97--107, 2014.
[14]
Burbank, D., The 5 V's of Big Data {Online}. Available at http://enterprisearchitects.com/the-5v-s-of-big-data/ (May, 2016).
[15]
Q. Fu, W. Liu, T. Xue, H. Gu, S. Zhang, and C. Wang, "A BIG DATA PROCESSING METHODS FOR VISUALIZATION," 2014 IEEE 3rd Int. Conf. on Cloud Comp. and Intelligence Systems, 2014.
[16]
K. Krishnan, "Big Data Processing Architectures", In Data Warehousing in the Age of Big Data. P.29--42 Part 1 (2nd Ed.) Elsevier.2013
[17]
B. Hedlund, "Understanding Hadoop Clusters and the Network" {Online}. Available at http://bradhedlund.com/2011/09/10/understanding-hadoop-clusters-and-the-network/. 10 September, 2011.
[18]
R. Vijayakumari, R. Kirankumar, and K. G. Rao, "Comparative analysis of Google File System and Hadoop Distributed File System," Int. Journal of Advanced Trends in Computer Science and Engineering, vol. 3, no. 1, pp. 553--558, 2014.
[19]
Apache HDFS, "HDFS Architecture Guide" {Online}. Available at http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html.
[20]
E. Indarto, "Data Mining" {Online}. Available at http://recommendersystems.readthedocs.io/en/latest/datamining.html. 5 July, 2013.
[21]
SciCumulus, "SciCumulus/C2 - Parallel Scientific Workflow Management System" {Online}. Available at https://scicumulusc2.wordpress.com/starter-guide-2/. 20 May, 2016.
[22]
Cloudera, "Cloudera Data Management" {Online}. Available at http://www.cloudera.com/documentation/enterprise/latest/topics/datamgmt.html. 20 May, 2016.
[23]
Cloudera, "Cloudera Impala" {Online}. Available at http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html. 20 May, 2016.
[24]
Cloudera, "Using the Parquet File Format with Impala Tables" {Online}. Available at http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/latest/topics/impala_parquet.html. 20 May, 2016.
Recommendations
Efficient Batch Processing of Related Big Data Tasks using Persistent MapReduce Technique
VisionNet'16: Proceedings of the Third International Symposium on Computer Vision and the InternetThe data generated by today's enterprises has been increasing at exponential rates in size from most recent couple of years. Also, the need to process and break down the substantial volumes of data has likewise expanded. In order to handle this enormous ...
Big data and ICT applications: A study
ICTCS '16: Proceedings of the Second International Conference on Information and Communication Technology for Competitive StrategiesBig Data is used to manage the data due to their large size and complexity, because it can't be handled with the traditional methods and the current technology or tools used for that. Big Data mining is populated with 5 V's volume, variability, velocity,...
An experimental survey on big data frameworks
AbstractRecently, increasingly large amounts of data are generated from a variety of sources.Existing data processing technologies are not suitable to cope with the huge amounts of generated data. Yet, many research works focus on Big Data, a buzzword ...
Highlights- An overview of most popular Big Data frameworks.
- A categorization of the presented frameworks and techniques.
- An extensive set of experiments to evaluate the studied Big Data frameworks.
- A description of best practices related ...
Comments
Information & Contributors
Information
Published In
Copyright © 2016 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 December 2016
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
UCC '16
Sponsor:
UCC '16: 9th International Conference on Utility and Cloud Computing
December 6 - 9, 2016
Shanghai, China
Acceptance Rates
Overall Acceptance Rate 27 of 93 submissions, 29%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 169Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)1
Reflects downloads up to 01 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in