Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Data Migration Best Practices

•
•

by Everett Dolgner, Business Development Manager, AWS AWS offers numerous services to migrate data at a petabyte scale. You can easily move large volumes of data from onsite to the cloud and utilize the cloud as a backup target using data transfer services, such as AWS Snowball, AWS Snowball Edge, or AWS Storage Gateway. Learn about available data migration options and which one is the right fit for your requirements.

1 of 31

1

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Everett Dolgner March 2018 AWS Services for Data Migration

2

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data movement OnlineOffline Data security and management Complete set of data building blocks Amazon EFS Amazon EBS AWS Snow family AWS Storage Gateway Family AWS Direct Connect Amazon EFS File Sync Amazon S3 Transfer Acceleration Storage Partners Amazon Kinesis Data Streams Amazon Kinesis Video Streams Amazon S3 Amazon Glacier AWS KMS AWS IAM AWS CloudWatch AWS CloudTrail AWS Cloud Formation AWS Lambda Amazon Macie AWS QuickSight

3

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Moving to the Cloud Common drivers of moving data to the cloud: Pain points of data migration: While every cloud adoption journey is unique, all require a data migration step, which can be hindered by blockers such as: Cost effectiveness Workforce productivity Business agility Operational resilience Access to innovative services Disruption Speed Costs Channels Cloud compatibility

4

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3EC2 EFS EBS Aurora EMR Amazon Glacier RDSIOT Redshift Migrating Data: 5 Key Questions 1) What kind of data is it, and where is it going? 4) How much data & time do you have? 2) 1-time or continuous movement? 3) 1-way or bi-directional? 5) How about the WAN? Files Block Volumes Databases IOT Streams

5

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Direct Connect Reduce Bandwidth Costs Consistent Network Performance Compatible with all AWS Services Private Connectivity to VPC Elastic Simple • Establish private connectivity between AWS and your data center • Dedicated connection can be partitioned into multiple virtual interfaces • Maintain network separation between public and private environments BENEFITS

6

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Snow* Family

7

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Snow* family AWS Snowball AWS Snowball Edge AWS Snowmobile • Petabyte-scale data migration • Rugged 8.5 G impact case • Rain and dust resistant • Data encryption end-to-end • 80 TB capacity/10 Gb network • Move 250 TB in 1 Week! • Compute and storage for hybrid/edge workloads • 100 TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25 Gb SFP28, and 40 Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable • Exabyte-scale data migration • Up to 100 PB capacity • Data encryption end-to-end • Dedicated security personnel • GPS tracking, alarm monitoring, 24/7 surveillance, and optional additional security

8

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How Snowball moves data into and out of AWS Create a job Connect the Snowball Copy data to the Snowball Your data move to Amazon S3 In transit to you Delivered to you Delivered to AWS At AWS Job created Job completed AWS Snow* family

9

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Snowmobile from Space Picture taken by DigitalGlobe’s WorldView-3 satellite http://blog.digitalglobe.com/industry/digitalglobe-moves-to-the-cloud-with-aws-snowmobile/

10

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EFS File Sync

11

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EFS File Sync Sync data from existing file systems into EFS file systems Fast Up to 5x faster than standard Linux copy tools Secure Encrypted parallel data transfer to AWS Simple Set up and manage easily from the AWS Console File systems from on-premises to EFS DIY in-cloud file systems to EFS EFS file systems between AWS Regions Use EFS File Sync to copy…

12

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EFS File Sync Fast and simple way to copy file systems to Amazon EFS File Server /NAS NFS Sync Agent Customer Premises Region file system EFS file system Amazon EFSHTTPS • Sync Agent deployed as a VM • No data is stored on the Sync Agent • Optimized TCP stack for high-throughput • Data is compressed and encrypted across the network

13

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3 Transfer Acceleration

14

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Transfer Acceleration S3 Bucket AWS Edge Location Uploader Optimized Throughput! Change your endpoint, not your code Leverages 59 global edge locations Optimized protocols No firewall exceptions No client software required

15

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Storage Gateway Family

16

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage Gateway hybrid storage solutions Use standard storage protocols to access AWS storage services AWS Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Files Volumes Tapes On-premises AWSCloud

17

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File gateway On-premises file storage maintained as objects in Amazon S3 Customer Premises • Data stored and retrieved from your Amazon S3 buckets • One-to-one mapping from files to objects • File metadata stored in object metadata • Bucket access managed by IAM role you own and manage • Use Amazon S3 lifecycle policies, versioning, or CRR to manage data Amazon Glacier S3 Standard S3 Standard- Infrequent Access HTTPS NFS v3 / v4.1 Application Server Storage Gateway Storage Gateway VM

18

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GlacierS3 Standard S3- Infrequent Access File Gateway PetroBank Application ServersLTO NAS Active archive migration from LTO Cost-effective storage in AWS with local data access AWS Direct Connect Self service loading of data Reduced time-to-data by days or weeks Storage archive costs reduced by 90% Halliburton Data Center AWS Lambda

19

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tape Gateway • Virtual tape storage in S3 and Glacier accessed via tape gateway • Data compressed in-transit and at-rest • Up to 1 PB total tape storage per gateway, unlimited archive capacity • Supports leading backup applications • **3-5 hour tape retrieval from Amazon Glacier Media Changer Virtual Tapes stored in Amazon S3 Archived Tapes stored in Amazon Glacier Backup Server iSCSI Storage Gateway Service Tape Gateway VM Tape Drive HTTPS Customer Premises Region Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management

20

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Move your tape archive Backup Server Customer Premises Tape Library Offsite Vault Backup Disk Appliance

21

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Move your tape archive Media Changer Virtual Tapes stored in Amazon S3 Archived Tapes stored in Amazon Glacier iSCSI Storage Gateway Service Tape Gateway VM Tape Drive HTTPS RegionBackup Server Customer Premises Tape Library Offsite Vault • Deploy Storage Gateway VTL as an archive target • Old archives can expire, or be recalled and written to SGW • Decommission Tape Library when all tapes are expired Backup Disk Appliance

22

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume gateway • Presents block storage on top of Amazon S3 accessed via the volume gateway • Compression of data in transit and at rest • Back up on-premises volumes to EBS snapshots • Create on-premises volumes from EBS snapshots • Up to 1 PB of total volume storage per gateway Application Server Amazon EBS snapshots Storage Gateway Service Volume Gateway VM Volume Stored in Amazon S3 HTTPSiSCSI Customer Premises Region On-premises volume storage backed by Amazon S3 with EBS snapshots

23

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume stored volume Amazon EBS snapshots Customer Premises iSCSI Storage Gateway VM Region stored volumeApp Server complete volume stored in the cloud and on-premises Amazon CloudWatch Storage Gateway Service Supporting storage can be SAN, VMDK, VHD cache upload buffer

24

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes Customer Premises Region App Server Existing Volume Existing Volume SAN, VMDK, VHD based volume

25

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes volume Customer Premises Region App Server Storage Gateway Service stored volume iSCSI Initial Sync HTTPS Stored Volume Mode Storage Gateway VM Attach volume to GW Disconnect volume from app server and connect it to GW (data in place) Accessible over iSCSI

26

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes volume Amazon EBS snapshots Customer Premises Region App Server Storage Gateway Service stored volume iSCSI HTTPS Volume Stored Mode Storage Gateway VM Create EBS snapshot

27

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes volume Amazon EBS snapshots Customer Premises Region App Server stored volume iSCSI HTTPS Volume Stored Mode Storage Gateway VM EC2 instance Windows Server DATAStorage Gateway Service Attach EBS Snapshot to EC2 instance

28

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes Region EC2 instance Windows Server DATA Customer Premises

29

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Partner Network

30

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Partner Network – migration & storage Backup & Restore Archive Primary Storage BC/DRData Migration

31

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage Training For Enterprise Storage Engineers • Learn how to architect and manage highly available solutions on AWS storage services • Advance toward AWS certifications • Help your organization migrate to the cloud faster Online at www.aws.training • Access 100+ new digital training courses including advanced training on storage • Deep Dives on S3, EFS, and EBS • Migrating and Tiering Storage to AWS (Hybrid Solutions)

More Related Content

Data Migration Best Practices

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Everett Dolgner March 2018 AWS Services for Data Migration
  • 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data movement OnlineOffline Data security and management Complete set of data building blocks Amazon EFS Amazon EBS AWS Snow family AWS Storage Gateway Family AWS Direct Connect Amazon EFS File Sync Amazon S3 Transfer Acceleration Storage Partners Amazon Kinesis Data Streams Amazon Kinesis Video Streams Amazon S3 Amazon Glacier AWS KMS AWS IAM AWS CloudWatch AWS CloudTrail AWS Cloud Formation AWS Lambda Amazon Macie AWS QuickSight
  • 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Moving to the Cloud Common drivers of moving data to the cloud: Pain points of data migration: While every cloud adoption journey is unique, all require a data migration step, which can be hindered by blockers such as: Cost effectiveness Workforce productivity Business agility Operational resilience Access to innovative services Disruption Speed Costs Channels Cloud compatibility
  • 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3EC2 EFS EBS Aurora EMR Amazon Glacier RDSIOT Redshift Migrating Data: 5 Key Questions 1) What kind of data is it, and where is it going? 4) How much data & time do you have? 2) 1-time or continuous movement? 3) 1-way or bi-directional? 5) How about the WAN? Files Block Volumes Databases IOT Streams
  • 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Direct Connect Reduce Bandwidth Costs Consistent Network Performance Compatible with all AWS Services Private Connectivity to VPC Elastic Simple • Establish private connectivity between AWS and your data center • Dedicated connection can be partitioned into multiple virtual interfaces • Maintain network separation between public and private environments BENEFITS
  • 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Snow* Family
  • 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Snow* family AWS Snowball AWS Snowball Edge AWS Snowmobile • Petabyte-scale data migration • Rugged 8.5 G impact case • Rain and dust resistant • Data encryption end-to-end • 80 TB capacity/10 Gb network • Move 250 TB in 1 Week! • Compute and storage for hybrid/edge workloads • 100 TB local storage • Local compute equivalent to an Amazon EC2 m4.4xlarge instance • 10GBase-T, 10/25 Gb SFP28, and 40 Gb QSFP+ copper, and optical networking • Ruggedized and rack-mountable • Exabyte-scale data migration • Up to 100 PB capacity • Data encryption end-to-end • Dedicated security personnel • GPS tracking, alarm monitoring, 24/7 surveillance, and optional additional security
  • 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How Snowball moves data into and out of AWS Create a job Connect the Snowball Copy data to the Snowball Your data move to Amazon S3 In transit to you Delivered to you Delivered to AWS At AWS Job created Job completed AWS Snow* family
  • 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Snowmobile from Space Picture taken by DigitalGlobe’s WorldView-3 satellite http://blog.digitalglobe.com/industry/digitalglobe-moves-to-the-cloud-with-aws-snowmobile/
  • 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EFS File Sync
  • 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EFS File Sync Sync data from existing file systems into EFS file systems Fast Up to 5x faster than standard Linux copy tools Secure Encrypted parallel data transfer to AWS Simple Set up and manage easily from the AWS Console File systems from on-premises to EFS DIY in-cloud file systems to EFS EFS file systems between AWS Regions Use EFS File Sync to copy…
  • 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EFS File Sync Fast and simple way to copy file systems to Amazon EFS File Server /NAS NFS Sync Agent Customer Premises Region file system EFS file system Amazon EFSHTTPS • Sync Agent deployed as a VM • No data is stored on the Sync Agent • Optimized TCP stack for high-throughput • Data is compressed and encrypted across the network
  • 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3 Transfer Acceleration
  • 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Transfer Acceleration S3 Bucket AWS Edge Location Uploader Optimized Throughput! Change your endpoint, not your code Leverages 59 global edge locations Optimized protocols No firewall exceptions No client software required
  • 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Storage Gateway Family
  • 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage Gateway hybrid storage solutions Use standard storage protocols to access AWS storage services AWS Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Files Volumes Tapes On-premises AWSCloud
  • 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File gateway On-premises file storage maintained as objects in Amazon S3 Customer Premises • Data stored and retrieved from your Amazon S3 buckets • One-to-one mapping from files to objects • File metadata stored in object metadata • Bucket access managed by IAM role you own and manage • Use Amazon S3 lifecycle policies, versioning, or CRR to manage data Amazon Glacier S3 Standard S3 Standard- Infrequent Access HTTPS NFS v3 / v4.1 Application Server Storage Gateway Storage Gateway VM
  • 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GlacierS3 Standard S3- Infrequent Access File Gateway PetroBank Application ServersLTO NAS Active archive migration from LTO Cost-effective storage in AWS with local data access AWS Direct Connect Self service loading of data Reduced time-to-data by days or weeks Storage archive costs reduced by 90% Halliburton Data Center AWS Lambda
  • 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Tape Gateway • Virtual tape storage in S3 and Glacier accessed via tape gateway • Data compressed in-transit and at-rest • Up to 1 PB total tape storage per gateway, unlimited archive capacity • Supports leading backup applications • **3-5 hour tape retrieval from Amazon Glacier Media Changer Virtual Tapes stored in Amazon S3 Archived Tapes stored in Amazon Glacier Backup Server iSCSI Storage Gateway Service Tape Gateway VM Tape Drive HTTPS Customer Premises Region Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management
  • 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Move your tape archive Backup Server Customer Premises Tape Library Offsite Vault Backup Disk Appliance
  • 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Move your tape archive Media Changer Virtual Tapes stored in Amazon S3 Archived Tapes stored in Amazon Glacier iSCSI Storage Gateway Service Tape Gateway VM Tape Drive HTTPS RegionBackup Server Customer Premises Tape Library Offsite Vault • Deploy Storage Gateway VTL as an archive target • Old archives can expire, or be recalled and written to SGW • Decommission Tape Library when all tapes are expired Backup Disk Appliance
  • 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume gateway • Presents block storage on top of Amazon S3 accessed via the volume gateway • Compression of data in transit and at rest • Back up on-premises volumes to EBS snapshots • Create on-premises volumes from EBS snapshots • Up to 1 PB of total volume storage per gateway Application Server Amazon EBS snapshots Storage Gateway Service Volume Gateway VM Volume Stored in Amazon S3 HTTPSiSCSI Customer Premises Region On-premises volume storage backed by Amazon S3 with EBS snapshots
  • 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Volume stored volume Amazon EBS snapshots Customer Premises iSCSI Storage Gateway VM Region stored volumeApp Server complete volume stored in the cloud and on-premises Amazon CloudWatch Storage Gateway Service Supporting storage can be SAN, VMDK, VHD cache upload buffer
  • 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes Customer Premises Region App Server Existing Volume Existing Volume SAN, VMDK, VHD based volume
  • 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes volume Customer Premises Region App Server Storage Gateway Service stored volume iSCSI Initial Sync HTTPS Stored Volume Mode Storage Gateway VM Attach volume to GW Disconnect volume from app server and connect it to GW (data in place) Accessible over iSCSI
  • 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes volume Amazon EBS snapshots Customer Premises Region App Server Storage Gateway Service stored volume iSCSI HTTPS Volume Stored Mode Storage Gateway VM Create EBS snapshot
  • 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes volume Amazon EBS snapshots Customer Premises Region App Server stored volume iSCSI HTTPS Volume Stored Mode Storage Gateway VM EC2 instance Windows Server DATAStorage Gateway Service Attach EBS Snapshot to EC2 instance
  • 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Migration with Storage Gateway volumes Region EC2 instance Windows Server DATA Customer Premises
  • 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Partner Network
  • 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Partner Network – migration & storage Backup & Restore Archive Primary Storage BC/DRData Migration
  • 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage Training For Enterprise Storage Engineers • Learn how to architect and manage highly available solutions on AWS storage services • Advance toward AWS certifications • Help your organization migrate to the cloud faster Online at www.aws.training • Access 100+ new digital training courses including advanced training on storage • Deep Dives on S3, EFS, and EBS • Migrating and Tiering Storage to AWS (Hybrid Solutions)