The Importance of DataOps in a Multi-Cloud World
- 1. A P R I L 4 , 2 0 1 9 KEVIN PETRIE SR DIRECTOR, ATTUNITY DATAOPS FOR MULTI- CLOUD STRATEGIES DATAVERSITY WEBINAR
- 2. 2© 2018 Attunity 2© 2017 Attunity LEADING provider of Streaming CDC Support most sources with best performance and least impact LEADING cloud DB migration technology Already moved over 120,000 databases to public Cloud platforms LEADING in agility and platform coverage Pre-packaged automation of complex processes and modern UX to accelerate delivery by “data people” THE LEADING PLATFORM FOR DELIVERING DATA EFFICIENTLY AND IN REAL-TIME TO CLOUDS, DATA LAKES, AND STREAMING ARCHITECTURES ATTUNITY: MODERN DATA INTEGRATION
- 3. 3© 2018 Attunity 3© 2019 Attunity SOURCES CLOUD Amazon RDS (SQL Server, Oracle, MySQL, Postgres) Amazon Aurora (MySQL) Amazon Redshift Azure SQL Server M1 (Q1) COMPREHENSIVE PLATFORM INTEGRATION SAP ECC ERP CRM SRM GTS MDG S/4HANA (on Oracle, SQL, DB2, HANA) DATABASE Oracle SQL Server DB2 iSeries DB2 z/OS DB2 LUW MySQL PostgeSQL Sybase ASE Informix ODBC EDW Exadata Teradata Netezza Vertica Pivotal MAINFRAME DB2 z/OS IMS/DB VSAM FLAT FILES Delimited (e.g., CSV, TSV) TARGETS FLAT FILES Delimited (e.g., CSV, TSV) STREAMING Kafka Amazon Kinesis Azure Event Hubs MapR Streams SAP HANA EDW Exadata Teradata Netezza Vertica Sybase IQ SAP HANA Microsoft PDW GOOGLE Cloud SQL (MySQL, Postgres) Cloud Storage Dataproc PubSub (‘19) Big Query (Q2) DATA LAKE Hortonworks Cloudera MapR Amazon EMR Azure HDInsight Google Dataproc DATABASE Oracle SQL Server DB2 LUW MySQL PostgreSQL Sybase ASE Informix MemSQL Compose support AZURE DBaaS (SQL DB) DBaaS (MySQL, Postgres) ADLS BLOB HDInsight Event Hub SQL DW Snowflake (Q1) Databricks (Q2) AWS RDS (MySQL, Postgres, MariaDB, Oracle, SQL Server) Aurora (MySQL, Postgres) S3 EMR Kinesis Redshift Snowflake (Q1) Databricks (Q2) SaaS Salesforce (Q2)
- 4. 4© 2018 Attunity 4© 2019 Attunity DBaaS STORAGE HADOOP STREAMING DWaaS OTHER DWaaS SPARK COMPREHENSIVE CLOUD INTEGRATION 1. Replicate support for Google PubSub planned for 2019. 2. Google BQ supported today through GCS or Kafka. Direct load planned for Q2/19. RDS (All) S3 EMR Kinesis Redshift Snowflake Databricks Compose support All ADLS , BLOB HDInsight Event Hubs Azure SQL DW Snowflake Databricks Q2 DB All GCS DataProc Pub Sub BigQuery (2) (1) Q2
- 5. 5© 2019 Attunity 5© 2019 Attunity WHY MULTIPLE CLOUDS? ENTERPRISE MOTIVATIONS TRIGGERS IMPROVE SLAS – PERFORMANCE, DOWNTIME REDUCE OPERATING COSTS HEDGE COMPETITIVE RISK SPECIALIZE FOR ADVANCED ANALYTICS NEW/CHANGED BUSINESS NEEDS INDEPENDENT BU DECISION LEARNING CURVE
- 6. 6© 2019 Attunity 6© 2019 Attunity DECISION TRADE-OFFS SLA PERFORMANCE LOWER COST HEDGED COMPETITIVE RISK SPECIALIZED TOOLS PROS CONS MANAGEMENT OVERHEAD SWITCHING COSTS ADMINISTRATIVE COMPLEXITY
- 7. 7© 2019 Attunity 7© 2019 Attunity MULTI-CLOUD SCENARIOS CLOUD SELECTION CRITERIA DIVERSIFICATION BY INITIATIVE COST REDUCTION CSP CHANGE/ REBALANCING DISASTER RECOVERY BURST TO CLOUD DEV IN CLOUD A PROD IN CLOUD B PRICING BI TOOLS DATA PROCESSING/TRANSFORMATION ON-PREMISES SYSTEM AFFINITY LOCK IN RISK CODING SUPPORT A B
- 8. 8© 2019 Attunity 8© 2019 Attunity ENTERPRISE STRATEGIES AND PRACTICES Carefully define domains and platform selection criteria Take phased approach – TestDev/PROD, mission criticality, risk Assess lock-in risk Ensure security and privacy SLAs with CSP Keep it simple ONGOINGUP FRONT MONITOR ADJUST KEEP DATA MOBILEKEEP DATA MOBILE STREAMLINE DATA FLOW
- 9. 9© 2019 Attunity 9© 2017 Attunity Relocate data and workloads as needed Reduce process variation between end points Accelerate setup and configuration Reduce dependency on ETL developers PLATFORM INTEGRATION AGILE MIGRATION MULTI CLOUD DATA REQUIREMENTS Speed data loading and transformation process Reduce time and effort of creating, updating data stores ANALYTICS READINESS
- 10. 10© 2019 Attunity 10© 2019 Attunity MULTI CLOUD ENVIRONMENTS NEED DATAOPS CODE DATAINFRASTRUCTURETOOLS PEOPLE TECHNOLOGYPROCESS Emerging discipline to build and manage efficient, effective data pipelines Applies DevOps of agility and continuous integration Seeks to improve collaboration between data managers and consumers Source: Gartner Innovation Insight for DataOps, December 2018
- 11. 11© 2019 Attunity 11© 2019 Attunity WHY DATAOPS? CHALLENGES MULTIPLY WITH EACH CLOUD Increasing analytics requirements create complexity and data flow bottlenecks Data consumers drive demands that IT cannot meet with existing processes and technologies Projects are failing due to this friction DATA VOLUME, VARIETY, VELOCITY RISING CHALLENGES NEW PLATFORMS NEW BUSINESS DEMANDS CODING COMPLEXITY “In every pipeline, data must be identified, captured, formatted, tagged, validated, profiled, cleaned, transformed, combined, aggregated, secured, cataloged, governed, moved, queried, visualized, analyzed, and acted upon. Phew!” WAYNE ECKERSON PRESIDENT, ECKERSON GROUP
- 12. 12© 2019 Attunity 12© 2017 Attunity STATE OF THE DATAOPS BUSINESS Source: Diving into DataOps, Eckerson Group, December 2018 Many organizations are just getting started First steps: continuous integration and testing Communication gap persists between data managers and consumers ADOPTION FRAMEWORKEARLY DAYS
- 13. 13© 2019 Attunity 13© 2019 Attunity CLOUD DATAOPS 2. Real-Time Analytics 3. Data Lake Automation 4. DW Automation 5. Metadata & Control MODERN PLATFORMS Big Data Cloud Data Lakes Streaming MODERN ANALYTICS NEED CLOUD DATAOPS MODERN ANALYTICS AI/ML IoT Predictive Real-Time 1. Agile Cloud Migration
- 14. 14© 2019 Attunity 14© 2017 Attunity CLOUD DATAOPS WITH ATTUNITY 1. AGILE CLOUD MIGRATION ZERO DOWNTIME MIGRATIONS RAPID DEPLOYMENT WITH NO AGENTS ON SOURCES 100% AUTOMATED SETUP, EXECUTION AND MONITORING EMPOWERING ARCHITECTS AND DBAS REAL-TIME STREAMING AGENTLESS CDC
- 15. 15© 2019 Attunity 15© 2019 Attunity CLOUD DATAOPS WITH ATTUNITY 1. AGILE CLOUD MIGRATION TARGETSSOURCES ON PREMISES CLOUD Hadoop RDBMSData Warehouse WAN DATA TRANSFER COMPRESSION MULTI-PATHING ENCRYPTION
- 16. 16© 2019 Attunity 16© 2019 Attunity CLOUD DATAOPS WITH ATTUNITY 2. REAL-TIME DATA FOR ANALYTICS SECURE MULTI- STREAMING TO CLOUD TARGETS RAPID DEPLOYMENT WITH NO AGENTS ON SOURCES 100% AUTOMATED SETUP, EXECUTION AND MONITORING LOW-IMPACT CHANGE DATA CAPTURE REAL-TIME DATA STREAMS AGENTLESS CDC
- 17. 17© 2019 Attunity 17© 2017 Attunity STREAMING DATA LAKE PIPELINE AUTOMATION FROM INGEST TO ANALYTICS For data architects and engineers Rapidly deliver real- time and analytics- ready data Remove the time, cost and risk of manual coding Adaptable to new sources, targets, platforms, technologies CLOUD DATAOPS WITH ATTUNITY 3. DATA LAKE AUTOMATION
- 18. 18© 2019 Attunity 18© 2019 Attunity CLOUD DATAOPS WITH ATTUNITY 3. DATA LAKE AUTOMATION STANDARDIZE MERGE FORMAT Full Change History ENRICH SUBSET HDS ODS Snapshot CAPTURE PARTITION Raw Deltas SAP RDBMS DATA WAREHOUSE FILES MAINFRAME Consume ANALYZE MONITOR GENERATE DELIVER REFINE DATA INGESTED VIA CDC INTO CLOUD AND DATA LAKE DATA CONTINUOUSLY UPDATED AND MERGED INTO CHANGE HISTORY PURPOSE-BUILT SUBSETS PROVISIONED FOR ANALYTICS
- 19. 19© 2019 Attunity 19© 2019 Attunity CLOUD DATAOPS WITH ATTUNITY 4. DATA WAREHOUSE AUTOMATION AUTOMATED WORKFLOW Real-Time Extraction Auto Extraction, Loading, Mapping Auto Generated Transformations Change Propagation Auto Design with Best Practices “DWA will accomplish an initial BI implementation up to five times faster than traditional methods”* *TDWI Data Warehouse Automation Course REAL-TIME ODS STAGING EDW MARTS
- 20. 20© 2019 Attunity 20© 2019 Attunity CONTROL AND RECONCILE MODEL VERSIONS ROLL BACK, COMPARE, MERGE, LOCK VERSIONS ACCEPTANCE PRODUCTION DEVELOPMEN T TEST CLOUD DATAOPS WITH ATTUNITY 4. AGILE DATA WAREHOUSE DEVELOPMENT STREAMLINE CREATION OF CUSTOM MODELS, ETL CODE, ETC. EASILY GENERATE SOFTWARE DEPLOYMENT PACKAGES IMPROVE TEAM PRODUCTIVITY AND AGILITY
- 21. 21© 2019 Attunity 21© 2019 Attunity CLOUD DATAOPS WITH ATTUNITY 5. METADATA AND CONTROL ENTERPRISE DASHBOARD VIEWS OPERATIONS ANALYTICS Control tasks and monitor data flow across distributed environments Multiple data centers On premises and cloud REPLICATE SERVER REPLICATE SERVER REPLICATE SERVER Trace data lineage for compliance Historical and real-time reporting Visualize, analyze, improve operations Capacity planning Activity and KPI trends
- 22. 22© 2019 Attunity 22© 2017 Attunity Data modernization initiative with specialized cloud platforms Google for Machine Learning, AWS for Infrastructure, Azure for CRM Leverages automated Attunity data pipeline Specialized platforms for distinct corporate objectives AWS – cost, infrastructure, performance, localization Azure – AI, advanced analytics, new services, microservices Attunity provides single data integration hub Redirecting data from one CSP to another based on partner’s competitive requirements AWS – DevTest, read-only DBs, archiving Google – BI and analytics Attunity provides single data integration hub MAJOR FINSERV FIRM TRAVEL SERVICES PROVIDER FORTUNE 100 FOOD CO. MULTI CLOUD CASE STUDIES
- 23. Thank You LEARN MORE AT www.Attunity.com SEE MY ARTICLES AT https://www.eckerson.com/blogs/decoding-data-software
- 24. 24© 2019 Attunity ATTUNITY – MODERNIZE AND AUTOMATE DATA INTEGRATION MAINFRAME SAP SAAS APPS FILES DATA WAREHOUSE RDBMS STREAMING DATA PIPELINE AUTOMATION DESIGN & MANAGE GENERATE DELIVER REFINE/MERGE change stream To cloud, lakes for analytic use DATA WAREHOUSES (ON-PREMISES & CLOUD) Azure SQL DW Amazon Redshift MODEL OTHER… OTHER… DATA LAKES, STREAMING (ON-PREMISES & CLOUD) CONFORM DATABASES (ON-PREMISES & CLOUD) COMMIT Azure SQL DB OTHER… Amazon RDS
Editor's Notes
- DataOps is a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization. The goal of DataOps is to deliver value faster by creating predictable delivery and change management of data, data models and related artifacts. DataOps uses technology to automate the design, deployment and management of data delivery with the appropriate levels of governance and metadata to improve the use and value of data in a dynamic environment.
- Market evolution aligns with our uniqueness
- The Attunity solution generates real time data streams, at scale and with low impact, from heterogeneous sources that include production databases, data warehouses, SAP and mainframe systems, as well as flat files and SaaS applications. We deliver data to all major target platforms, then refine it for analytics. Our solution enables three data delivery options. In the first option, we replicate committed transactional data, either in bulk or via real-time change data capture (CDC), to relational databases, ranging from Oracle to SQL Server to PostgreSQL, on premises or in the cloud. We also can replicate to or from flat files. This enables use cases such as migrations, reporting and analytics. The second scenario supports data warehouse modernization initiatives. We deliver the transactional data streams to modern data warehouses such as Snowflake and Azure SQL DW, using automated data delivery and modelling methods to enable your reporting and analytics activities. We automate model creation, data warehouse creation, data mart creation, table creation, data instantiation and source/target mappings. You can manage all these tasks as an integrated workflow. Third, we take relational transaction streams and stitch them together into a common format that can be readily consumed for analytics on Big Data platforms such as Amazon EMR, Azure HDInsight and Google Dataproc. Our solution automatically creates, loads and updates data stores, and accelerates dataset readiness for analytics. You can then process the datasets we refine alongside non-relational and unstructured data from other sources. Because all these capabilities are native to the Attunity data integration solution, you are able to prepare data for analytics with fewer software components. [Internal note: we support structured and relational data. We do not support unstructured and/or non-relational data.]