poster

Efficiently querying archived data using Hadoop

Authors:

Rajeev Gupta,

Himanshu Gupta,

Ullas Nambiar,

Mukesh MohaniaAuthors Info & Claims

CIKM '10: Proceedings of the 19th ACM international conference on Information and knowledge management

Pages 1301 - 1304

https://doi.org/10.1145/1871437.1871606

Published: 26 October 2010 Publication History

Get Access

Abstract

The need to analyze structured data for various business intelligence applications such as customer churn analysis, social network analysis, telecom network monitoring etc., is well known. However, the potential size to which such data will scale in future will make solutions that revolve around data warehouses hard to scale. As data sizes grow the movement of data from the warehouse to archives becomes more frequent. Current file based archive models make the archived data unusable for any type of insight extraction. In this paper, we present an active archival solution for data warehouses that makes use of Hadoop distributed file system (HDFS) to store the data in an always available and cost-effective manner. We investigate various structured data storage schemes within HDFS and empirical evaluations show that a combination of Universal scheme model and column store is best suited for the active archival solution.

References

[1]

Apache Foundation. Hadoop. http://hadoop.apache.org/core/.

Google Scholar

[2]

Hive- Hadoop wiki. http://wiki.apache.org/hadoop/Hive

Google Scholar

[3]

JSON. http://www.json.org

Google Scholar

[4]

Jaql Project hosting. http://code.google.com/p/jaql/

Google Scholar

[5]

M. Poess and R.O. Nambiar and D. Walrath. Why You Should Run TPC-DS: A Workload Analysis. In Proceedings of VLDB, 2007.

Digital Library

Google Scholar

[6]

M. Vardi. The Universal-Relation Data Model for Logical Independence. IEEE Software, Vol.5, No.2, 1988.

Digital Library

Google Scholar

[7]

M. Stonebraker et al. C-STORE: A Column-oriented DBMS. In Proceedings of VLDB, 2005.

Digital Library

Google Scholar

[8]

http://en.wikipedia.org/wiki/Communications_in_India

Google Scholar

Cited By

View all

Xiao LZou BZhu CZeng MChen Z(2021)HFBT: An Efficient Hierarchical Fault-tolerant Method for Cloud Storage System2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00019(27-36)Online publication date: Sep-2021
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00019
Chen YZhou YTaneja SQin XHuang J(2017)aHDFS: An Erasure-Coded Data Archival System for Hadoop ClustersIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2017.270668628:11(3060-3073)Online publication date: 1-Nov-2017
https://doi.org/10.1109/TPDS.2017.2706686
Nithya MMaheshwari N(2017)Load rebalancing for Hadoop Distributed File System using distributed hash table2017 International Conference on Intelligent Sustainable Systems (ICISS)10.1109/ISS1.2017.8389317(939-943)Online publication date: Dec-2017
https://doi.org/10.1109/ISS1.2017.8389317
Show More Cited By

Index Terms

Efficiently querying archived data using Hadoop
1. Information systems
  1. Data management systems
    1. Information integration
      1. Data warehouses
  2. Information systems applications
    1. Decision support systems
      1. Data warehouses

Recommendations

Medical Big Data Warehouse: Architecture and System Design, a Case Study: Improving Healthcare Resources Distribution

The huge increases in medical devices and clinical applications which generate enormous data have raised a big issue in managing, processing, and mining this massive amount of data. Indeed, traditional data warehousing frameworks can not be effective ...
Evaluating SQL-on-Hadoop for Big Data Warehousing on Not-So-Good Hardware
IDEAS '17: Proceedings of the 21st International Database Engineering & Applications Symposium

Big Data is currently conceptualized as data whose volume, variety or velocity impose significant difficulties in traditional techniques and technologies. Big Data Warehousing is emerging as a new concept for Big Data analytics. In this context, SQL-on-...
Data Warehouse with Big Data Technology for Higher Education
Abstract
Nowadays, data warehouse tools and technologies cannot handle the load and analytic process of data into meaningful information for top management. Big data technology should be implemented to extend the existing data warehouse solutions. ...

Comments

Information & Contributors

Information

Published In

CIKM '10: Proceedings of the 19th ACM international conference on Information and knowledge management

October 2010

2036 pages

ISBN:9781450300995

DOI:10.1145/1871437

General Chair:
Jimmy Huang
York University, Canada
,
Program Chairs:
Nick Koudas
University of Toronto, Canada
,
Gareth Jones
Dublin City University, Ireland
,
Xindong Wu
University of Vermont, USA
,
Kevyn Collins-Thompson
Microsoft Research, USA
,
Aijun An
York University, Canada

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

CIKM '10

Sponsor:

CIKM '10: International Conference on Information and Knowledge Management

October 26 - 30, 2010

ON, Toronto, Canada

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
867
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Xiao LZou BZhu CZeng MChen Z(2021)HFBT: An Efficient Hierarchical Fault-tolerant Method for Cloud Storage System2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00019(27-36)Online publication date: Sep-2021
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00019
Chen YZhou YTaneja SQin XHuang J(2017)aHDFS: An Erasure-Coded Data Archival System for Hadoop ClustersIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2017.270668628:11(3060-3073)Online publication date: 1-Nov-2017
https://doi.org/10.1109/TPDS.2017.2706686
Nithya MMaheshwari N(2017)Load rebalancing for Hadoop Distributed File System using distributed hash table2017 International Conference on Intelligent Sustainable Systems (ICISS)10.1109/ISS1.2017.8389317(939-943)Online publication date: Dec-2017
https://doi.org/10.1109/ISS1.2017.8389317
Ghuli PShettar R(2014)A novel approach to implement a shop bot on distributed web crawler2014 IEEE International Advance Computing Conference (IACC)10.1109/IAdCC.2014.6779439(882-886)Online publication date: Feb-2014
https://doi.org/10.1109/IAdCC.2014.6779439
Lee TBae HKim H(2014)Join processing with threshold-based filtering in MapReduceThe Journal of Supercomputing10.1007/s11227-014-1179-969:2(793-813)Online publication date: 1-Aug-2014
https://dl.acm.org/doi/10.1007/s11227-014-1179-9
Zhu QQin Z(2012)HyDBProceedings of the 2012 IEEE 14th International Conference on High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems10.1109/HPCC.2012.84(580-587)Online publication date: 25-Jun-2012
https://dl.acm.org/doi/10.1109/HPCC.2012.84

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Medical Big Data Warehouse: Architecture and System Design, a Case Study: Improving Healthcare Resources Distribution

Evaluating SQL-on-Hadoop for Big Data Warehousing on Not-So-Good Hardware

Data Warehouse with Big Data Technology for Higher Education