Data Driven Advanced Analytics using Denodo Platform on AWS
•
0 likes•84 views
The document discusses challenges with data-driven cloud modernization and how the Denodo platform can help address them. It outlines Denodo's capabilities like universal connectivity, data services APIs, security and governance features. Example use cases are presented around real-time analytics, centralized access control and transitioning to the cloud. Key benefits of the Denodo data virtualization approach are that it provides a logical view of data across sources and enables self-service analytics while reducing costs and IT dependencies.
Report
Share
Data Driven Advanced Analytics using Denodo Platform on AWS
- 2. Agenda 1. Introduction to Data Driven Everything on AWS 2. Challenges with Data Driven Cloud Modernization 3. Addressing Challenges with Denodo Platform for AWS 4. Denodo Platform Use Cases and Data Architectures. 5. Key Takeaways | Q&A
- 3. 3 Modernizing leads to maximum innovation velocity and optimal total cost of ownership On-premises Lift and shift Move to managed databases Modernize with purpose-built databases Innovation velocity Total cost of ownership (TCO) Break-free from legacy databases
- 4. 4
- 5. 5
- 6. 6 What are customers building? Backup & restore Non-disruptive Easy place to start Integrated with all major vendors Archive & compliance Media workflows Tape replacement Public Sector, FinServ, Healthcare/Life Sciences Home directories Simple to move Not sensitive to latency Significant cost savings Data lakes Variety of analytics tools Built for streaming data Data visualization Business- critical applications Integrated with major vendors Fully managed infrastructure Lift-and-shift migrations
- 7. 7 AWS Customer- Analytics Challenges in a Distributed Data Landscape Point-to-point data integration approaches are challenging: § Extracting and moving data increases latency and cost, and decreases quality, thus lacking unified data access § Every project solves data access and integration in a different way, increasing IT dependency § Solutions are tightly coupled to data sources, impacting flexibility, agility and overall governance DATA SOURCE DATA CONSUMER Data Governance Tools DB, DW & Data Lakes Files BI Dashboard Report and Tools Data Science & Machine Learning Apps Mobile & Enterprise Apps Microservices Apps Cloud DB & SaaS Streaming Data & IoT Cube
- 8. 8 • The business wants more useful data • Timely, curated, usable • IT can’t keep up • 67% of companies use less than half of their data* • IT stuck in old school thinking about data management • ‘Business as usual’ The ‘Useful Data’ Gap * Source: Denodo Global Cloud Survey 2022
- 9. 9 AWS Analytics Data Strategy, Keynote Remember, re:Invent 2022
- 10. 10 10 Businesses need a new approach to connect data silos in real-time to support various applications, insights, and analytics.
- 11. 11 Modern Data Architectural Patterns & Data Driven Analytics Data Mesh Data Lakehouse Data Lake Data Fabric Cloud Data Warehouse
- 12. 12 Real World Data Lake Example – AWS Trusted Data Zone Raw Data Zone Refined Data Zone Transformation Transformation Data Consumers Networking, Infrastructure & Security Data Ingestion Data Sources Data Catalog and Search – Asset Registry Workflow Orchestration, DevOps and CI/CD
- 13. 13 Denodo + AWS – Simple and Complementary Recipe! • Embrace distributed data landscape • Embrace the fact that data resides in multiple locations or systems – on-prem, hybrid, multi- cloud. All data needs to be managed with consistency • Use a Logical approach to manage it • Consumers access data through a centralized semantic model, decoupled from data location and physical schemas, that can enforce security and governance requirements
- 14. 14 Denodo Platform: ONE Logical Platform for All Your Data Ease of Use Fast Query Response Integrated, Active Data Catalog Universal Connectivity Modern Data Services API Layer Dynamic Data Masking Automated Cloud Management Key Differentiators 83% reduction in time-to-revenue 67% reduction in data preparation effort 65% decrease in delivery times over ETL Source: Forrester Total Economic ImpactTM of Data Virtualization, 2021 Hybrid/ Multi-Cloud Security & Governance Al/ML Recommendations Advanced Semantics Data Catalog Discover / Explore / Document BI Tools SQL / MDX Data Science Tools Data as a Service RESTful / Odata GraphQL/ GeoJSON Files Cubes Cloud Stores Traditional DB & DW INTEGRATE MANAGE DELIVER Disparate data in any location, format or latency Related data with a universal semantic model and AI / ML functionality enabling vital data governance And democratize data using BI & data science tools, data catalogs, and APIs Data Lake & NoSQL Query Optimization & Acceleration
- 15. On- Premise Transition to Cloud Hybrid Multi- Cloud Stages of the Cloud Journey
- 16. • • • • • • • Transition (or Migration) to the Cloud Challenges
- 17. 17 Reduce the Business Impact 1 - Transition to the AWS Cloud (Minimize Business Disruption) Business Need § Transition to cloud – migrate EDW § Real-time analytics from Business Users and Data Scientists § Security and governance across multiple analytical tools need to be centralized § Acts as a single semantic layer § Homogeneous data access regardless of back-end technology § No need to deal with new languages and APIs: access to SFDC, Excel, Amazon Redshift, Oracle, Hadoop, other SaaS APIs, etc. § Consistent business data model across all consumers and reporting tools § Reusability of analytical objects across multiple tools and consuming applications § Abstracts access to disparate data sources § Change in the data sources buffered minimizing the impact on consumer business applications § New technology adoption with minimal impact on the business § Minimizes impact on consumers § Minimizes cross-environment connectivity § reducing risks of unauthorized access to data § Amazon Athena § Amazon S3 Buckets § Amazon Redshift § Amazon Aurora § AWS PaaS - RDBMs Denodo AWS
- 18. 18 Transition to Cloud | Cloud Migration Acceleration Denodo becomes the common access layer for all on- premise and cloud systems: Access to all data from a single system The data can be accessed directly from the original system, without the need for replication The data can still be easily replicated and hidden if necessary Simplify data aggregation, regardless of the location or format of the data Allows semantic models definition, independent of the original formats and structures Advanced security for all data Documentation and usage statistics included in the Data Catalog
- 19. • • • • Hybrid (Cloud and On-Premise) Data Integration – Customer 360 / Single View
- 20. 20 AWS Cloud Modernization - LeasePlan Data Hub Architecture ` DATA ACQUISITION DATA SOURCES DATA STORE (RAW) ANALYTICS WAREHOUSE DATA SCIENCE DATA FABRIC DATA CONSUMER Next Gen Data Management (Meta-data, data quality, governance) Meta data management, data quality, data governance as central components guarding the overall data-asset of the corporation to allow trusted access to data for utilisation across the enterprise Structured Unstructured ETL/ELT ORCHESTRATION STREAMING Native Extraction No ETL Tool(s) AWS Kinesis Airflow SAP BW/4HANA + HANA Native Raw Quality Integration Consumption Glacier Archive BW/4HANA + HANA Native NG Finance 1 NG Insurance NG Procurement NG Marketing NG Sales NG Service NG Commerce NG Fleet Ops NG Supplier Engagement NG Policy Mgt. NG Portals NG Contact Center Legacy – NOLS/ DB2/AS400 etc. Other External Data: Telematics, IoT, GA, Social feeds, streams Analytics for Cloud Analysis for Office AWS SageMaker Power BI Role Based Access Control Caching
- 21. 21 Take the right decision on accurate data 2 - Real-Time Analytics for Business Users Business Need § Transition to cloud – migrate EDW § Real-time analysis from Business Users and Data Scientists § Security and governance across multiple analytical tools need to be centralized § Enables Self-Service BI § IT delivers a governed layer of “business views” to business users § Business users can generate any report over those IT-governed business views § Business views can be adapted for every type of user making use of the same terminology and naming conventions for every Line of Business § Incorporate geospatial, IoT, and other streaming data, to enable real-time data services § Accelerate cloud analytics with Amazon’s elastic infrastructure (EC2, auto-scaling) § Data is immediately available for use without delays § Integrate and Manage data across Amazon Redshift, Amazon RDS, Amazon S3 in real-time to drive advanced analytics § Source data to Amazon Lambda serverless processes and expose them as data source for BI-Analytics § Visualize data and reports in real time with QuickSights Denodo AWS
- 22. 22 How Does Denodo Platform Work? Development Lifecycle Mgmt Monitoring & Audit Governance Security Development Tools and SDK Scheduled Tasks Data Caching Query Optimizer JDBC/ODBC/ADO.Net SOAP / REST WS U Customer 360 View Virtual Data Mart View J Application Layer Business Layer Unified View Unified View Unified View Unified View A J J Derived View Derived View J J S Transformation & Cleansing Data Source Layer Base View Base View Base View Base View Base View Base View Base View Abstraction
- 23. 23 FAA – Federal Aviation Administration – Streamline Operations/Analytics ü Reduced the IT Operations Cost by 99.8%, while accelerating data access by 96%. ü To reduce costs and streamline IT operations, the U.S. Federal Aviation Administration (FAA) wanted to consolidate multiple IT organizations – each supporting different mission areas – into a single office reporting to a single CIO. FAA leveraged the Denodo platform on AWS to:
- 24. 24 Across multiple analytical tools 3 - Centralized Security and Governance Business Need § Transition to cloud – migrate EDW § Real-time analytics from Business Users and Data Scientists § Security and governance across multiple analytical tools need to be centralized § Unified Security Layer § Global Tag-based Policy Engine § Role-based authorization to all tables in the virtual layer (RBAC) § Attribute-based access control (ABAC) § Security is moved outside the reporting layer to avoid security bypasses § Centralized access point simplifies operations and auditing § Data Masking / Obfuscation § Centralized Governance Layer § Centralized metadata catalog accessible for both technical and business users § Data Source refresh, change impact analysis, full data lineage, etc. § Protects data sources from uncontrolled access through query throttling, limiting #concurrent queries over them, limiting resulting datasets sizes, enabling the cache for minimizing the access to data sources for some views, etc. Denodo AWS Services § Datawarehouse Built for the cloud § Athena § Redshift § Secured, Managed Access § With Amazon Resource Manager § Identity Management & SSO Amazon IAM
- 25. 25 Data Fabric Overview Core Principles: ü Data Integration ü Data Governance ü Data Democratization ü Data Intelligence ü Data Interoperability
- 26. 26 Data Mesh Powered by Denodo Data Virtualization SQL Operational EDW Data Lakes Files SaaS APIs REST GraphQL OData Event Product Customer Location Employee Common Domain Event Management Human Resources MDX 2.Domains connect their data sources ❷ 1.Each domain is given a separate virtual schema. A common domain may be useful to centralized data products common across domains ❶ 3.Metadata is mapped to relational views. No data is replicated ❸ 4.Domains SMEs can model their Data Products. Products can be used to define other products ❹ 5.For execution, Products can be served directly from their sources, or replicated to a central location, like a lake ❺ 6.A central team can set guidelines and governance to ensure interoperability ❻ 7.Products can be access via SQL, MDX or exposed as an API. No coding is required ❼ 8.Infrastructure can easily scale out in a cluster ❽ New architectural paradigm for data management | distributed organizational paradigm | Domains in charge of Data Products
- 27. 27 Data Fabric & Data Mesh Powered by Data Virtualization
- 28. Summary and Takeaways Benefits of Logical Data Architectures
- 29. Benefits of a Logical Data Architecture “Now, we can do weekly releases. We’re able to add new data sources within 2 to 3 hours. We’re about 60% faster than we were in the old world.” VP of data and analytics, real estate “To me, it all boils down to speed to insights. Not having to wait to get the question that you have top-of-mind answered with data is huge.” VP of data and analytics, real estate 29
- 30. 30 Try Denodo Platform on AWS – Get Started Today! • 30 days Free Trial of Denodo Professional via AWS Marketplace • AWS Marketplace Transactable Pay-Go/Private Offers • Denodo – AWS Test Drives (free hands-on learning in 2 hours) : Denodo-AWS BI Denodo-AWS Data Science Visit Denodo Platform and AWS https://www.denodo.com/en/denodo-platform/denodo-platform-for-aws
- 31. Q&A
- 32. Thanks! www.denodo.com info@denodo.com © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.