research-article

Fusion insight librA: huawei's enterprise cloud data analytics platform

Proceedings of the VLDB Endowment, Volume 11, Issue 12

Pages 1822 - 1834

https://doi.org/10.14778/3229863.3229870

Published: 01 August 2018 Publication History

Abstract

Huawei Fusion Insight Libr A (FI-MPPDB) is a petabyte scale enterprise analytics platform developed by the Huawei data-base group. It started as a prototype more than five years ago, and is now being used by many enterprise customers over the globe, including some of the world's largest financial institutions. Our product direction and enhancements have been mainly driven by customer requirements in the fast evolving Chinese market.

This paper describes the architecture of FI-MPPDB and some of its major enhancements. In particular, we focus on top four requirements from our customers related to data analytics on the cloud: system availability, auto tuning, query over heterogeneous data models on the cloud, and the ability to utilize powerful modern hardware for good performance. We present our latest advancements in the above areas including online expansion, auto tuning in query optimizer, SQL on HDFS, and intelligent JIT compiled execution. Finally, we present some experimental results to demonstrate the effectiveness of these technologies.

References

[1]

Amazon Athena - Amazon. https://aws.amazon.com/athena/.

[2]

Amazon Redshift - Amazon. https://aws.amazon.com/redshift/.

[3]

Amazon Redshift Spectrum - Amazon. https://aws.amazon.com/redshift/spectrum/.

[4]

Apache - HBase. https://hbase.apache.org/.

[5]

Apache - Hive. https://hive.apache.org/.

[6]

Apache - ORC. https://orc.apache.org/.

[7]

Apache - Parquet. https://parquet.apache.org/.

[8]

Apache HAWQ - Apache. http://spark.apache.org/sql/.

[9]

FusionInsight MPPDB - Huawei. http://e.huawei.com/us/products/cloud-computing-dc/cloud-computing/bigdata/fusioninsight.

[10]

Greenplum Database - Pivotal. http://www.greenplum.com.

[11]

Hadoop - Apache. http://hadoop.apache.org/.

[12]

MemSQL. https://www.memsql.com/.

[13]

Oracle Autonomous Database Strategy White Paper. http://www.oracle.com/us/products/database/autonomous-database-strategy-wp-4124741.pdf.

[14]

Postgres-XC. https://sourceforge.net/projects/postgres-xc/.

[15]

Snowflake - Snowflake. https://www.snowflake.net/.

[16]

Spark SQL & DataFrames - Apache. http://spark.apache.org/sql/.

[17]

Transwarp Inceptor. http://www.transwarp.cn/.

[18]

S. Agrawal, S. Chaudhuri, L. Kollar, A. Marathe, V. Narasayya, and M. Syamala. Database Tuning Advisor for Microsoft SQL Server 2005. In Proceedings of the 30th VLDB Conference, Toronto, Canada, pages 1110--1121, 2004.

[19]

D. V. Aken, A. Pavlo, G. J. Gordon, and B. Zhang. Automatic Database Management System Tuning Through Large-scale Machine Learning. In Proceedings of SIGMOD 2017, Chicago, USA, pages 1009--1024, 2017.

Digital Library

[20]

J. Cohen, J. Eshleman, B. Hagenbuch, J. Ken, C. Pedrotti, G. Sherry, and F. Waas. Online Expansion of Large-scale Data Warehouses. In PVLDB, 4(12):1249--1259, 2011.

[21]

C. Garcia-Alvarado, V. Raghavan, S. Narayanan, and F. M. Waas. Automatic Data Placement in MPP Databases. In Proceedings of the 2012 IEEE 28th International Conference on Data Engineering Workshops, 2012.

Digital Library

[22]

O. Ivanov and S. Bartunov. Adaptive Query Optimization in PostgreSQL. In PGCon 2017 Conference, Ottawa, Canada, 2017.

[23]

M. Z. J. Sompolski and P. Boncz. Vectorization vs. compilation in query execution. In In Proc. of the 7th International Workshop on Data Management on New Hardware, 2011.

Digital Library

[24]

M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, A. Choi, J. Erickson, M. Grund, D. Hecht, M. Jacobs, I. Joshi, L. Kuff, D. Kumar, A. Leblang, N. Li, H. Robinson, D. Rorke, S. Rus, J. Russell, D. Tsirogiannis, S. Wanderman-milne, and M. Yoder. Impala: A modern, open-source sql engine for hadoop. In In Proc. CIDR15, 2015.

[25]

H. Liu, M. Xu, Z. Yu, V. Corvinell, and C. Zuzarte. Cardinality Estimation using Neural Networks. In Proceeds of the 25th Annual International Conference on Computer Science and Software Engineering (CASCON15), pages 53--59, 2015.

Digital Library

[26]

H. Lu and R. Setiono. Effective Query Size Estimation Using Neural Networks. In Applied Intelligenve, pages 173--183, May 2002.

Digital Library

[27]

T. Neumann. Efficiently compiling efficient query plans for modern hardware. In PVLDB, 4(9):539--550, 2011.

Digital Library

[28]

V. Reddy Gankidi, N. Teletia, J. Patel, A. Halverson, and D. J. DeWitt. Indexing hdfs data in pdw: Splitting the data from the index. In PVLDB, 7(13):1520--1528, 2014.

Digital Library

[29]

S. Shankar, R. Nehme, J. Aguilar-Saborit, A. Chung, M. Elhemali, A. Halverson, E. Robinson, M. Subramanian, D. DeWitt, and C. Galindo-Legaria. Query Optimization in Microsoft SQL Server PDW. In Proceedings of SIGMOD 2012, Scottsdale, USA, pages 767--776, 2012.

Digital Library

[30]

M. Stillger, G. Lohman, V. Markl, and M. Kandil. LEO - DB2s LEarning Optimizer. In Proceedings of the 27th VLDB Conference, Roma, Italy, 2001.

Digital Library

[31]

F. Waas. Beyond Conventional Data Warehousing -Massively Parallel Data Processing with Greenplum Database. In In Proc. BIRTE, 2008.

[32]

W. Wang, M. Zhang, G. Chen, H. V. Jagadish, B. C. Ooi, and K.-L. Tan. Database Meets Deep Learning: Challenges and Opportunities. In SIGMOD Record, Vol. 45, No2, 2016.

Digital Library

Cited By

Cao HHuang CLiu SHu HZhao MZhou XTu YQian W(2025)Aion: Live Migration for In-Memory Databases with Zero Downtime and Reduced Redundant Data TransferData Science and Engineering10.1007/s41019-024-00276-5Online publication date: 15-Jan-2025
https://doi.org/10.1007/s41019-024-00276-5
Wang YFeng YXi CWang BTang BGeng Y(2024)Development of an Intelligent Coal Production and Operation Platform Based on a Real-Time Data Warehouse and AI ModelEnergies10.3390/en1720520517:20(5205)Online publication date: 19-Oct-2024
https://doi.org/10.3390/en17205205
Ye WLi S(2024)Digital protection analysis of national traditional sports health cultural heritage based on big data in the era of data cloudMultimedia Tools and Applications10.1007/s11042-024-18371-083:27(69739-69758)Online publication date: 2-Feb-2024
https://doi.org/10.1007/s11042-024-18371-0
Show More Cited By

Fusion insight librA: huawei's enterprise cloud data analytics platform
1. Information systems
  1. Data management systems
    1. Database management system engines
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory

Recommendations

Libra: Improved Partitioning Strategies for Massive Comparative Metagenomics Analysis
ScienceCloud'18: Proceedings of the 9th Workshop on Scientific Cloud Computing

Big-data analytics platforms, such as Hadoop, are appealing for scientific computation because they are ubiquitous, well-supported, and well-understood. Unfortunately, load-balancing is a common challenge of implementing large-scale scientific computing ...
Cloud-based big data analytics for customer insight-driven design innovation in SMEs
Abstract
Fast development of IT and ICT facilitate customers to post a large volume of their concerns and expectation online, which are widely accepted to be a valuable resource for product designers. However, it is found that only a small ...
HD Insight: For Starters

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 11, Issue 12

August 2018

426 pages

ISSN:2150-8097

Editors:
Sihem Amer-Yahia
University of Grenoble Alpes, CNRS
,
Jian Pei
Simon Fraser University

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 August 2018

Published in PVLDB Volume 11, Issue 12

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
187
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)1

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cao HHuang CLiu SHu HZhao MZhou XTu YQian W(2025)Aion: Live Migration for In-Memory Databases with Zero Downtime and Reduced Redundant Data TransferData Science and Engineering10.1007/s41019-024-00276-5Online publication date: 15-Jan-2025
https://doi.org/10.1007/s41019-024-00276-5
Wang YFeng YXi CWang BTang BGeng Y(2024)Development of an Intelligent Coal Production and Operation Platform Based on a Real-Time Data Warehouse and AI ModelEnergies10.3390/en1720520517:20(5205)Online publication date: 19-Oct-2024
https://doi.org/10.3390/en17205205
Ye WLi S(2024)Digital protection analysis of national traditional sports health cultural heritage based on big data in the era of data cloudMultimedia Tools and Applications10.1007/s11042-024-18371-083:27(69739-69758)Online publication date: 2-Feb-2024
https://doi.org/10.1007/s11042-024-18371-0
Chen JShi RChen HZhang LLi RDing WFan LWang HXiong MChen YDong BGuo KLin YLiu XShi HWang PWang ZYang YZhao JZhou DZuo ZLiang Y(2023)Krypton: Real-Time Serving and Analytical SQL Engine at ByteDanceProceedings of the VLDB Endowment10.14778/3611540.361154516:12(3528-3542)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611545
Chen JDing YLiu YLi FZhang LZhang MWei KCao LZou DLiu YZhang LShi RDing WWu KLuo SSun JLiang Y(2022)ByteHTAPProceedings of the VLDB Endowment10.14778/3554821.355483215:12(3411-3424)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554832
Zhang X(2022)Application of Fusion of Fuzzy Mathematical Clustering Analysis in Enterprise Financial Management Cloud PlatformMathematical Problems in Engineering10.1155/2022/53545732022(1-9)Online publication date: 10-Sep-2022
https://doi.org/10.1155/2022/5354573
Kang JCai LLi FZhou XCao WCai SShao DIves ZBonifati AEl Abbadi A(2022)Remus: Efficient Live Migration for Distributed Databases with Snapshot IsolationProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3526047(2232-2245)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3526047
Floratos SGhazal ASun JChen JZhang X(2021)DBSpinner: Making a Case for Iterative Processing in Databases2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00273(2399-2410)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00273
Feng GLi A(2021)Intelligent Automated Operation and Operation Management of Large Enterprise Cloud Data Center Based on Artificial Intelligence2020 International Conference on Data Processing Techniques and Applications for Cyber-Physical Systems10.1007/978-981-16-1726-3_65(531-538)Online publication date: 2-Jun-2021
https://doi.org/10.1007/978-981-16-1726-3_65
Avni HAliev AAmor OAvitzur ABronshtein IGinot EGoikhman SLevy ELevy ILu FMishali LMo YPachter NSivov DVeeraraghavan VVexler VWang LWang P(2020)Industrial-strength OLTP using main memory and many coresProceedings of the VLDB Endowment10.14778/3415478.341553713:12(3099-3111)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3415478.3415537
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents