Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Migrating_Large_Scale_Data_Sets_to_the_Cloud

•
•
Amazon Web Services
Amazon Web Services

AWS now offers the simple services of data migration at a petabyte scale. You can easily move large volumes of data from onsite to the cloud. You can also quickly get started with the cloud as a backup target using data transfer services such as AWS Snowball AWS Snowball Edge or AWS Storage Gateway. Learn about the available data migration options and which one is the right fit for your requirements. We discuss customer use cases and review the different applications they used with our data migration services to cut their IT expenditures and management time on hardware and backup solutions.

1 of 36
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. H O N G K O N G 10.18.19
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. H O N G K O N G 10.18.19 Migrating Large Scale Data Sets to the Cloud Clifford Duke Solutions Architect Amazon Web Services
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migrating data: Five key questions 1) What kind of data is it, and where is it going? 2) One time or continuous movement? 3) One way or bi-directional? 4) How much data and time do you have? 5) Bandwidth constraints?
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Security and management Complete set of data building blocks Data movement OnlineOffline Amazon Macie Amazon Quicksight AWS Lambda AWS CloudFormation AWS Trail AWS CloudWatch AWS IAM AWS KMS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon S3 Transfer Acceleration Change your endpoint, not your code Leverages 180 global edge locations Optimized protocols No firewall exceptions No client software required S3 Bucket AWS Edge Location Uploader Optimized Throughput!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Getting started Enable Amazon S3 Transfer Acceleration for their bucket DNS resolution <bucket>.s3-accelerate.amazonaws.com resolves to the nearest POP location for the client Uses the client IP and Amazon CloudFront latency based routing Customers update application/destination URI to: <bucket-name>.s3-accelerate.amazonaws.com File is uploaded to: <bucket-name>
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. How fast is Amazon S3 Transfer Acceleration? Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these edge locations to a bucket in Singapore Public InternetS3 Transfer Acceleration On average, we have seen 171% improvement over regular Amazon S3 when uploading over long distances
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use case: Media upload ” “ S3 transfer acceleration reduces the average time it takes for us to ingest videos from our global user base by almost half. This gives our customers the ability to edit and share videos sooner where speed is a critical factor. - Brian Kaiser, CTO Typical Friday during football season: more than 35 hours of video every minute is uploaded S3-TA = >20% increase in upload and encoding speeds
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Storage Gateway On-premises access to virtually unlimited cloud storage Customer premises AWS Cloud Amazon S3 S3 Glacier S3 Glacier Deep Archive Amazon Elastic Block Store (EBS) Integrated with IAM, KMS, CloudTrail, CloudWatch services Appliance Configuration: VMware, Hyper-V, EC2, Hardware Files (NFS/SMB) Volumes (iSCSI) Tapes (iSCSI VTL) AWS Backup HTTPS Gateway Appliance Storage Gateway Managed Service
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Where does Storage Gateway store data? ap-southeast-1 Storage Gateway Service Activated Region Data Center Physical Location HTTPS iSCSI
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Gateway family File Gateway Store and access objects in Amazon S3 from file-based applications with local caching Volume Gateway Block storage on-premises backed by cloud storage with local caching, Amazon EBS snapshots, and clones, integrated with AWS Backup Tape Gateway Drop-in replacement for physical tape infrastructure backed by cloud storage with local caching Windows & Linux apps using S3 SAN-like with cloud recovery Easily switch tape backups to AWS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. File Gateway overview and use cases Store and access objects in Amazon S3 from file-based applications with local caching Backup on-premises data to the cloud Shift on-premises storage to cloud-backed file shares Low latency access to data in AWS for on-premises applications Use cases On-Premises AWS Cloud NFS & SMB File GatewayApplication Amazon S3 HTTPS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Connect using NFS v3/v4 or SMB v2/v3 protocols • Files stored as native S3 objects • Metadata is preserved as object user metadata • SMB ACLs—Windows Access Control Entries for up to 10 AD users and groups Fully managed local cache • Read-through, write-back, LRU managed, up to 16TB Optimized data transfers • Uploads only changes, downloads retrieve file parts needed • Up to 4 Gbps writes Notifications through Amazon CloudWatch (e.g., upload complete) Refresh cache by prefix • Optimizes content distribution workloads Object-level encryption with SSE-S3 or SSE-KMS S3 object lock support • WORM storage for on-premises file-based applications File Gateway capabilities
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tape gateway overview and use cases Present cloud-backed virtual tapes to on-premises backup applications Tape-based backups and archives to cloudUse cases (S3 Glacier Deep Archive) OR (S3 Glacier) Tape library (Amazon S3) Tape Shelf On-Premises AWS Cloud iSCSI VTL Tape Gateway HTTPS Application Storage Gateway services
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Emulates physical tape library through iSCSI-VTL protocol Compatible with all leading backup software Fully managed local cache for recent backups • Read-through, write-back, LRU managed Virtual tapes stored in Amazon S3 Ejected virtual tapes archived as read-only in S3 Glacier or S3 Glacier Deep Archive Move tapes in S3 Glacier to S3 Glacier Deep Archive Easy retrieval of archived tapes to virtual tape library • Retrieve in 3–5 hours from S3 Glacier and within 12 hours from Deep Archive Configurable encryption SSE-S3 or SSE-KMS Performance • Client writes up to 2.3 Gbps, downloads up to 0.6 Gbps Tape Gateway capabilities
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Volume Gateway overview and use cases Present block storage on-premises backed by cloud storage Provide cloud-backed block storage to on-premises applications Backup on-premises data to the cloud Use cases On-Premises AWS Cloud iSCSI Volume GatewayApplication HTTPS Storage Gateway service Amazon EBS snapshots AWS Backup
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Presents block storage over iSCSI Volumes stored in AWS reducing on-premises SAN footprint Thin-provisioned (cached) or local (stored) volume types Fully managed local cache Read-through, write-back, LRU managed Configurable encryption with SSE-S3 or SSE-KMS Volume snapshots stored in Amazon EBS Manage backups of volumes through AWS Backup Move volumes between gateways using attach-detach feature Volume Gateway capabilities
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Gateway is available on a variety of form factors WhereWhen
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hybrid storage with Storage Gateway On-premises In-cloud Protocols Applications Management Networking Integrates with Amazon S3, Amazon EBS, and Amazon Glacier 1. Standard storage protocols Notifications to enable automated data pipelines in the cloud 2. Low-latency access via fully-managed cache Manage through the AWS console as another AWS Service 3. Multiple hardware and software deployment options Enables native AWS capabilities such as storage classes and replication 4. Optimized data transfers
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Problem Stacks of disk arrays on-premises were expensive and required a lot of space Complex architecture and cache hierarchy Many readers via NFS Solution File gateway for local access to cloud data AWS DataSync to transfer bulk data and active datasets to cloud Active/active multi-region and versioning with lifecycles Outcome $1M bandwidth cost savings Saved ~85% on storage, per location Storage engineers focus on high-value activities Shift on-premises storage to cloud-backed file shares
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS DataSync The speed and reliability of network acceleration software with the cost-effectiveness of open source tools Agent deployed on VMware or EC2 for efficient access to local NFS server Highly parallel transfers using optimized network protocol Optimized to read and writes to S3 and EFS using IAM/VPCE Fully-managed service scales to send or receive data from agent On-Premises Region AWS Storage Resources Amazon S3 bucket Amazon Elastic File System AWS DataSyncShared file system AWS DataSync agent NFS TLS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. As simple as 1-2-3... Deploy on-premises agent and connect to your AWS account Create a task by configuring storage locations and options Start the task and monitor progress 1 2 3 VMware ESXi 4 vCPU 32 GB memory On-premises NFS file systems, S3 buckets, or EFS file systems Monitor in the console or through Amazon CloudWatch
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Snow Family Petabyte-scale data migration Compute and storage for hybrid/edge workloads Exabyte-scale data migration AWS Snowball AWS Snowball Edge AWS Snowmobile
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Snowball Edge Petabyte-scale hybrid device with onboard compute and storage • 100 TB local storage • Local compute via Amazon EC2 instances or AWS Lambda functions • 10GBase-T, 10/25 Gb SFP28, and 40 Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable E-ink shipping label
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowball Edge key features Amazon S3-compatible endpoint File interface (NFS) Clustering Run AWS Lambda functions Faster data transfer Encryption
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. How Snowball moves data into and out of AWS Job created Job completed In transit to you Delivered to you Delivered to AWS At AWS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowball Edge: Remote data collection Customer example: Oregon State University Collect and analyze oceanic and coastal images (60 TB/week) • Environmental and ocean ecosystem research • Prior to Snowball, used many small HDDs—took weeks to months to upload • $4MM+ in infrastructure investment—expensive and inefficient “Snowball lets us migrate TBs of data in days at a fraction of the cost”
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowmobile: Seeing a better world DigitalGlobe takes Satellite imagery of the Earth 100PB image library = 6 billion square kilometers 1PB new image every year Architecture before Snowmobile: Stored data in their own data center Needed elastic compute power to retrieve and analyze images Wanted to move data to the cloud, but no feasible solution Snowmobile lets DigitalGlobe migrate 100PB of data to the cloud
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowmobile from Space
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. When to use AWS Snow* Cloud migration Disaster recovery Data center decommission Remote data collection
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Partner Network (APN): Migration & storages Backup & restore Archive Primary storage BC/DRData migration
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary
Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Clifford Duke

More Related Content

Migrating_Large_Scale_Data_Sets_to_the_Cloud

  • 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. H O N G K O N G 10.18.19
  • 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. H O N G K O N G 10.18.19 Migrating Large Scale Data Sets to the Cloud Clifford Duke Solutions Architect Amazon Web Services
  • 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Migrating data: Five key questions 1) What kind of data is it, and where is it going? 2) One time or continuous movement? 3) One way or bi-directional? 4) How much data and time do you have? 5) Bandwidth constraints?
  • 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Security and management Complete set of data building blocks Data movement OnlineOffline Amazon Macie Amazon Quicksight AWS Lambda AWS CloudFormation AWS Trail AWS CloudWatch AWS IAM AWS KMS
  • 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon S3 Transfer Acceleration Change your endpoint, not your code Leverages 180 global edge locations Optimized protocols No firewall exceptions No client software required S3 Bucket AWS Edge Location Uploader Optimized Throughput!
  • 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Getting started Enable Amazon S3 Transfer Acceleration for their bucket DNS resolution <bucket>.s3-accelerate.amazonaws.com resolves to the nearest POP location for the client Uses the client IP and Amazon CloudFront latency based routing Customers update application/destination URI to: <bucket-name>.s3-accelerate.amazonaws.com File is uploaded to: <bucket-name>
  • 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. How fast is Amazon S3 Transfer Acceleration? Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these edge locations to a bucket in Singapore Public InternetS3 Transfer Acceleration On average, we have seen 171% improvement over regular Amazon S3 when uploading over long distances
  • 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use case: Media upload ” “ S3 transfer acceleration reduces the average time it takes for us to ingest videos from our global user base by almost half. This gives our customers the ability to edit and share videos sooner where speed is a critical factor. - Brian Kaiser, CTO Typical Friday during football season: more than 35 hours of video every minute is uploaded S3-TA = >20% increase in upload and encoding speeds
  • 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Storage Gateway On-premises access to virtually unlimited cloud storage Customer premises AWS Cloud Amazon S3 S3 Glacier S3 Glacier Deep Archive Amazon Elastic Block Store (EBS) Integrated with IAM, KMS, CloudTrail, CloudWatch services Appliance Configuration: VMware, Hyper-V, EC2, Hardware Files (NFS/SMB) Volumes (iSCSI) Tapes (iSCSI VTL) AWS Backup HTTPS Gateway Appliance Storage Gateway Managed Service
  • 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Where does Storage Gateway store data? ap-southeast-1 Storage Gateway Service Activated Region Data Center Physical Location HTTPS iSCSI
  • 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Gateway family File Gateway Store and access objects in Amazon S3 from file-based applications with local caching Volume Gateway Block storage on-premises backed by cloud storage with local caching, Amazon EBS snapshots, and clones, integrated with AWS Backup Tape Gateway Drop-in replacement for physical tape infrastructure backed by cloud storage with local caching Windows & Linux apps using S3 SAN-like with cloud recovery Easily switch tape backups to AWS
  • 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. File Gateway overview and use cases Store and access objects in Amazon S3 from file-based applications with local caching Backup on-premises data to the cloud Shift on-premises storage to cloud-backed file shares Low latency access to data in AWS for on-premises applications Use cases On-Premises AWS Cloud NFS & SMB File GatewayApplication Amazon S3 HTTPS
  • 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Connect using NFS v3/v4 or SMB v2/v3 protocols • Files stored as native S3 objects • Metadata is preserved as object user metadata • SMB ACLs—Windows Access Control Entries for up to 10 AD users and groups Fully managed local cache • Read-through, write-back, LRU managed, up to 16TB Optimized data transfers • Uploads only changes, downloads retrieve file parts needed • Up to 4 Gbps writes Notifications through Amazon CloudWatch (e.g., upload complete) Refresh cache by prefix • Optimizes content distribution workloads Object-level encryption with SSE-S3 or SSE-KMS S3 object lock support • WORM storage for on-premises file-based applications File Gateway capabilities
  • 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tape gateway overview and use cases Present cloud-backed virtual tapes to on-premises backup applications Tape-based backups and archives to cloudUse cases (S3 Glacier Deep Archive) OR (S3 Glacier) Tape library (Amazon S3) Tape Shelf On-Premises AWS Cloud iSCSI VTL Tape Gateway HTTPS Application Storage Gateway services
  • 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Emulates physical tape library through iSCSI-VTL protocol Compatible with all leading backup software Fully managed local cache for recent backups • Read-through, write-back, LRU managed Virtual tapes stored in Amazon S3 Ejected virtual tapes archived as read-only in S3 Glacier or S3 Glacier Deep Archive Move tapes in S3 Glacier to S3 Glacier Deep Archive Easy retrieval of archived tapes to virtual tape library • Retrieve in 3–5 hours from S3 Glacier and within 12 hours from Deep Archive Configurable encryption SSE-S3 or SSE-KMS Performance • Client writes up to 2.3 Gbps, downloads up to 0.6 Gbps Tape Gateway capabilities
  • 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Volume Gateway overview and use cases Present block storage on-premises backed by cloud storage Provide cloud-backed block storage to on-premises applications Backup on-premises data to the cloud Use cases On-Premises AWS Cloud iSCSI Volume GatewayApplication HTTPS Storage Gateway service Amazon EBS snapshots AWS Backup
  • 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Presents block storage over iSCSI Volumes stored in AWS reducing on-premises SAN footprint Thin-provisioned (cached) or local (stored) volume types Fully managed local cache Read-through, write-back, LRU managed Configurable encryption with SSE-S3 or SSE-KMS Volume snapshots stored in Amazon EBS Manage backups of volumes through AWS Backup Move volumes between gateways using attach-detach feature Volume Gateway capabilities
  • 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Gateway is available on a variety of form factors WhereWhen
  • 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hybrid storage with Storage Gateway On-premises In-cloud Protocols Applications Management Networking Integrates with Amazon S3, Amazon EBS, and Amazon Glacier 1. Standard storage protocols Notifications to enable automated data pipelines in the cloud 2. Low-latency access via fully-managed cache Manage through the AWS console as another AWS Service 3. Multiple hardware and software deployment options Enables native AWS capabilities such as storage classes and replication 4. Optimized data transfers
  • 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Problem Stacks of disk arrays on-premises were expensive and required a lot of space Complex architecture and cache hierarchy Many readers via NFS Solution File gateway for local access to cloud data AWS DataSync to transfer bulk data and active datasets to cloud Active/active multi-region and versioning with lifecycles Outcome $1M bandwidth cost savings Saved ~85% on storage, per location Storage engineers focus on high-value activities Shift on-premises storage to cloud-backed file shares
  • 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS DataSync The speed and reliability of network acceleration software with the cost-effectiveness of open source tools Agent deployed on VMware or EC2 for efficient access to local NFS server Highly parallel transfers using optimized network protocol Optimized to read and writes to S3 and EFS using IAM/VPCE Fully-managed service scales to send or receive data from agent On-Premises Region AWS Storage Resources Amazon S3 bucket Amazon Elastic File System AWS DataSyncShared file system AWS DataSync agent NFS TLS
  • 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. As simple as 1-2-3... Deploy on-premises agent and connect to your AWS account Create a task by configuring storage locations and options Start the task and monitor progress 1 2 3 VMware ESXi 4 vCPU 32 GB memory On-premises NFS file systems, S3 buckets, or EFS file systems Monitor in the console or through Amazon CloudWatch
  • 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Snow Family Petabyte-scale data migration Compute and storage for hybrid/edge workloads Exabyte-scale data migration AWS Snowball AWS Snowball Edge AWS Snowmobile
  • 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Snowball Edge Petabyte-scale hybrid device with onboard compute and storage • 100 TB local storage • Local compute via Amazon EC2 instances or AWS Lambda functions • 10GBase-T, 10/25 Gb SFP28, and 40 Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable E-ink shipping label
  • 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowball Edge key features Amazon S3-compatible endpoint File interface (NFS) Clustering Run AWS Lambda functions Faster data transfer Encryption
  • 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. How Snowball moves data into and out of AWS Job created Job completed In transit to you Delivered to you Delivered to AWS At AWS
  • 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowball Edge: Remote data collection Customer example: Oregon State University Collect and analyze oceanic and coastal images (60 TB/week) • Environmental and ocean ecosystem research • Prior to Snowball, used many small HDDs—took weeks to months to upload • $4MM+ in infrastructure investment—expensive and inefficient “Snowball lets us migrate TBs of data in days at a fraction of the cost”
  • 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowmobile: Seeing a better world DigitalGlobe takes Satellite imagery of the Earth 100PB image library = 6 billion square kilometers 1PB new image every year Architecture before Snowmobile: Stored data in their own data center Needed elastic compute power to retrieve and analyze images Wanted to move data to the cloud, but no feasible solution Snowmobile lets DigitalGlobe migrate 100PB of data to the cloud
  • 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowmobile from Space
  • 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. When to use AWS Snow* Cloud migration Disaster recovery Data center decommission Remote data collection
  • 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Partner Network (APN): Migration & storages Backup & restore Archive Primary storage BC/DRData migration
  • 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary
  • 36. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Clifford Duke