How to Quickly and Easily Draw Value from Big Data Sources_Q3 symposia(Moa)
- 1. How to Quickly and Easily Draw Value from Big Data Sources Moacyr Passador Senior Sales Engineer
- 2. DevelopersAnalystsConsumers Data Scientists Business Users IT Users Types of BI Users The Old BI World Today’s BI World Business Users are getting more involved in producing analytical content The Role Of Business Users In BI Today Has Greatly Evolved
- 3. Relational Databases MapReduce & NoSQL Multi-Dimensional & Other BI Tools Cloud Applications Departmental Data Social Media Business Users Today Want Direct Access To More Data To make insightful decisions on their own, business users demand instant access to Data from multiple enterprise sources &
- 4. >100x More content creation and consumption 5-10x More Content 5-10x More Content Creators 5-10x More Sharing More productive More content per creator More producers More users can create content More collaborative Peer-to-peer sharing & Adoption Of Self-service Analytics By Business Users Increases Productivity
- 5. Introduction to Big Data
- 6. What is Big Data, Really? The Three Vs of Big Data According to Gartner Volume • Orders of magnitude bigger than conventional data (Terabytes, Petabytes, Exabytes) • Cost-prohibitive or practically impossible to store, manage or analyze in typical database software Variety • Structured, semi-structured, unstructured formats • Diverse sources - complex event processing, application logs, machine sensors, social media data Velocity • Speed of ingesting incoming data streams • Processing and real-time analysis of streaming and complex event data Volume Variety Velocity
- 7. Four broad categories of Big Data sources and their value Traditional sources becoming bigger Company, Government, Financial sector, Business and consumer studies, Surveys, Polls All business performance drivers – Operational efficiency, Revenue management, Strategic planning SOURCE VALUE Digital exhaust from interactions Online click-stream, Application logs, Call/service records, ID scans, Security cameras New revenue sources, Consumer promotions, Risk management, Fraud detection SOURCE VALUE Web 2.0 phenomenon Content generated from social media posts, tweets, blogs, pictures, videos, ratings Customer engagement, Customer service, Brand management, Viral marketing SOURCE VALUE Internet of things Machine generated sensor data and “connected device” communication Operational efficiency, Cost control, Risk avoidance SOURCE VALUE Business Oriented Use Cases for Big Data SOURCE VALUE
- 8. Data Lakes and MicroStrategy
- 9. • Manicured, Often Relational • Known Data Volumes • Expected Formats • Little to No Change DATA SOURCES ETL DATA WAREHOUSE BI & ANALYTICS • Complex Structures • Rigid Transformations • Extensive Monitoring Required • Transformed Historical to Read Structures • Flat, Canned Access to Data • Report Chaos • Extensive Data Load Delays • Inflexible with new sources Traditional Approaches And Current State Of A DWH
- 10. • Increase in Data types • Rising Data Volumes • Pressure on the DWH • Constant change DATA SOURCES ETL DATA WAREHOUSE BI & ANALYTICS • Delay in reacting to new sources • Monitoring is abandoned • Transformations cant keep pace • Repaid, Adjust and Redesign ETL • Reports become invalid • Delay in updates • Users seek silos • Business is disconnected w/ IT Challenges With Traditional DWHs With Growing Data Demands
- 11. A storage repository, usually in Hadoop, that holds a vast amount of raw data in its native format until it is needed • Low cost • Flexible and Loosely governed • No need to decide up front what data to store or for how long • Contains a mix of structured, semi-structuredand unstructured data • Allows for freeform data exploration without having to wait for ETL “If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.” -- James Dixon What is a Data Lake?
- 12. Data is an asset • Today many organizations discard data due to cost of storage even though business value may be mined from the data in the future • A Data Lake allows you to store and process data essentially indefinitely Unification • Data Discovery, Data Science and Enterprise BI are treated as silos in many organizations today • A Data Lake can help unify these concepts and allow cross team collaboration Utility • A Data Lake creates the possibility of answering future questions of your data without knowing the question in advance Why Have A Data Lake?
- 13. Building the lake is easy / using it is hard Governance is still key • Without a meaningful strategy for maintaining data quality this strategy can quickly fail • You can quickly create a Data Swamp (dirty) or a Data Graveyard (useless) Sandbox strategy Beware Of The Swamp • Segment portions of your lake for experiments, testing and data that may fall outside of your standard governance process
- 14. Analytical Datamarts Relational Databases MDM ETLand Governance Data Warehouse Structured Enterprise Data Operational Data Enterprise Metadata MicroStrategy Analytics Platform Enterprise Applications Traditional Data Architecture
- 15. Data Lake Enterprise Metadata Prime ELTwith Governance ETL Relational Data Enterprise Applications Cloud-based data Web Logs Flat Files Analytical Datamarts MicroStrategy Analytics Platform In Comes The Data Lake
- 16. ⎸07062016 Build Analytics and Mobility Applications Using Data Stored with Big Data Hadoop Vendors SQL on Hadoop NoSQL Databases Elastic Map Reduce The MicroStrategy platform empowers organizations to build applications that leverage big data and Hadoop distributions. All of the major Hadoop distributions are certified to work with MicroStrategy and once connected, data stored in Hadoop becomes just like any other data source. Users can connect using Hive, Pig, or proprietary SQL-on-Hadoop connectors like Cloudera Impala or IBM BigInsights. The MicroStrategy big data engine can natively tap into HDFS, generating schema on-read and making Hadoop suitable for ad- hoc querying. It also enables parallel loading of data from HDFS, resulting in high-performance data loading. MicroStrategy’s native connectivity saves users from the tedious process of ETL from HDFS to Hive and helps to overcome the ODBC overhead associated with Hive. MicroStrategy’s data wrangling capability lets users cleanse and refine their big data directly in MicroStrategy’s data discovery tool. Big DataEnterprise assets Mobility apps that source information from multiple locations, and submit transactions to your ERP systems Analytics applications that blend data from databases and big data and deliver insights to users via reports, dashboards, and apps 16 MicroStrategy Platform
- 17. ⎸07062016 MicroStrategy allows organizations toeasily accessandanalyzedatain all shapes andsizes, from a singleplace. Business and IT users caneasily blend multiple data-sources includingbig data sources in seconds.From personal spreadsheets to cloudsources,to evenHDFS,big data access is madequick and easy withnativeHDFS connectors or viaanyflavor of hiveproducts includingCloudera,Hortonworks, MapR,Spark andmore. Batch SQL: Fulfill your batch processing needs with certified Hive/ODBC drivers from different Hadoop distributors: Cloudera, Hortonworks, MapR, and Amazon EMR Interactive SQL: Leverage advanced SQL on Hadoop technologies for interactive queries such as Cloudera Impala, MapR Drill, Apache Spark, IBM BigInsights, Pivotal HAWQ, and Facebook Presto No SQL: Connect, query, and analyze data from No SQL sources such as HBase and Cassandra Search: Dynamically search on semi-structured and unstructured data with MicroStrategy’s integration to Apache Solr and Splunk 17 MicroStrategy Analytics Platform Distributed File Systems (HDFS, Amazon S3, GFS…) No SQLBatch SQL Interactive SQL Search Big Data Analytics for Most Common Scenarios Big data is just another data source
- 19. 5 Differentiators for Big Data Analytics “Leverage MicroStrategy's improved data discovery capabilities, as an alternative to augmenting your BI portfolio with products such as Tableau and Qlik, to lower cost of ownership and improve enterprise self-serviceviaa single, broader solution.” Gartner Research Note – December 11, 2015
- 20. Enterprise data access with complete data governance Self-service data exploration and production dashboards User accessible advanced and predictive analytics Analysis of semi-structured and unstructured data Real-time analysis from live updating data 1 2 3 4 5 Five Differentiators for MicroStrategy in Big DataAnalytics The MicroStrategy Analytics Platform enables every business user to get these capabilities
- 21. Optimized Access to Your Entire Big Data Ecosystem as If It Were a Single Database Data Warehouse Appliances MapReduce & NOSQLDatabases Relational Databases Multidimensional Databases Columnar Databases SaaS-BasedApp Data HANA BigInsights Parallel Data Warehouse Elastic Map Reduce Analysis Services Redshift BringAllRelevant DatatoDecision Makers Distribution Zendesk HDFS Generic Web Services SOAP REST User /Departmental Data 1. Enterprise data access with complete data governance
- 22. Stunning Visualizations Instant Query Results Effortless Dashboards No IT Needed Lightning fast insights, easy for everyone 2. Self-service data exploration and production dashboards
- 23. User / Departmental Data Data Warehouse Appliances MapReduce Databases Relational Databases Multidimensional Databases Columnar Databases SaaS-Based App Data MicroStrategy Multisource Engine 2 & 3 Join data on-the-fly. No need to move it to a staging database first. Access your entire Big Data ecosystem as if it were a single database Combine Data from Multiple Sources 2 3 1 2 1 2 3
- 24. MicroStrategy Supports Data Discovery at Scale Start with Departmental teams and grow exponentially to publish to 1000s of users without fear 24 PUBLISH Team Department Enterprise Value Chain 10s 100s 1,000s 10,000+ MicroStrategy Desktop Proven Scalability Built-in clustering, failover, and comprehensive administrative tools for performance optimization In-Memory Performance Tested sub-second response times on web and mobile, even at highest user volume Advanced Monitoring Admin tools to automate, report and alert on system utilization Content Personalization Users only see relevant data, and only access functionality they are authorized to use
- 25. Extend MicroStrategy’s Sophisticated Analytical Capabilities with 3rd party statistics toolkits 25 Industry’s most powerful SQLEngine and 300+ native analytical functions Projections RelationshipAnalysis Benchmarking TrendAnalysis Data Summarization AnalyticalMaturity What is likely to happen based on past history? What factors influence activity or behavior? How are we doing versus comparables? What direction are we headed in? What is happening in the aggregate? Optimization What do we want to happen? World’s most popular advancedanalytics tool. Free, opensource. More Specialty Tools 3. Business User Friendly Predictive and Advanced Analytics
- 26. Streaming AnalyticsInteractive Search Text Analytics Quickly investigate: • Website logs • Application usage • Surveys and free form text fields • Event and error monitoring logs 80% of data in most businesses is unstructured and this proportion will keep on rising 4. Analysis of semi-structured and unstructured data Find keyword and event occurrences in any data Apply semantic and syntactic models to text data Assess rapidly changing data streams Extract relevant information to: • Optimize search engine marketing • Understand sentiment on topics • Get a 360 degree view of customers • Detect fraud Analyze an array of data from: • Sensors and devices • Images, audio, and video • Email and document management systems • Other operational and transactional data
- 27. New live data update technology 5. Real-time analysis from live updating data 2 7
- 28. ⎸07062016 Native HDFS Connector Native Big Data Wrangling Support for search sources In-memory parallel architecture MicroStrategy Tableau Qlik Power BI IBM Cognos SAP BOBJ Oracle OBIEE MicroStrategy enables organizations to quickly harness the value of big data by deploying analytics at scale Big Data Analytics: Product Differentiators 28
- 30. Powerful data preparation for more accurate analysis Empowering data analysts to deliver deeper insights with intuitive and integrated data wrangling capabilities Data integration in the hands of every user Access and combine data from multiple sources “on-the-fly” to drive more productivity MicroStrategy Prime – Analyze more data in memory In-memory engine tightly coupled to the underlying DB MicroStrategy Multi-Source Effectively navigate data across multiple data sources MicroStrategy Enabling Technologies for Big Data
- 31. Empowering data analysts to deliver deeper insights with intuitive and integrated data wrangling capabilities 31 Streamlined workflows to parse and preparedata Hundreds of inbuilt functions to profile andcleandata Multi-Table in-memory support from different sources Automatically parse and prepare data with every refresh Create custom groups On the fly andwithout coding Local /URL Files Hadoop Data Preparation NewIntegratedDataPreparationCapability Salesforce Twitter/Facebook Powerful Data Preparation For More Accurate Analysis
- 32. Source 1 Source 2 Source 3 Source 4 Public Data DataBlending Data Access Live Connection In-memory Data OR Other BI Tools SaaS Data Native HDFS Access and combine data from multiple sources “on-the-fly” to drive more productivity Data Integration In The Hands Of Every User
- 33. Data Upload 4x Faster Server With MicroStrategy 9.x Serial access to in-memory data Database OLAP Cube With MicroStrategy 10 Multi-threadedaccess to in-memory data Database PRIME Server 2B 2B | 2B . . . 2B Data Volumes 80x Larger ………Core 1 Core 2 Core 16 CPU ………Core 1 Core 2 Core 16 CPU Data Interactions 50% faster Bottleneck …… Up to 8 parallelthreads In-memory engine tightly coupled to the underlying DB MicroStrategy Prime – Analyze More Data In Memory
- 34. Seamlessly traverse multiple DBs End user agnostic Effectively use aggregates Automatic navigation Works with different source types Move from Hadoop toRelational Metadata driven No need towrite SQL Effectively Navigate Data Across Multiple Data Sources MicroStrategy Multi-Source
- 35. Data lakes are a powerful tool MicroStrategy is ready to support your hybrid architecture Meet the needs of multiple user personas Enterprise BI capabilities combined with ad-hoc data discovery A scalable solution for all workloads Highly scalable in memory engine to combine data from multiple sources MicroStrategy Multi-Source Effectively navigate data across multiple data sources Summary
- 36. Questions