AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, and Data Transfer Services

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
AWS Data Transfer Services
Everett Dolgner
Storage BD
Pop-up Loft
AWS Data Transfer Services
Everett Dolgner
Storage BD

Cloud Data Migration
Direct ConnectSnow* data
transport
family
3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
The AWS Storage Portfolio
Object
Amazon GlacierAmazon S3
Block
Amazon EBS
(persistent)
Amazon EC2
Instance Store
(ephemeral)
File
Amazon EFS

The Snow Family:
large batches or “edge” scenarios

AWS Snow Family
Snowball Snowball Edge Snowmobile
Petabyte-scale data
migration
Compute & Storage for
Hybrid/Edge workloads
Exabyte-scale data
migration

AWS Snowball
Petabyte-scale data transport
• Rugged 8.5G impact case
• Rain and dust resistant
• Data encryption end-to-end
• 80 TB capacity/10G network
E-ink shipping
label

AWS Snowball Edge
Petabyte-scale hybrid device with onboard compute and storage
• 100 TB local storage
• Local compute equivalent to an Amazon
EC2 m4.4xlarge instance
• 10GBase-T, 10/25Gb SFP28, and 40Gb
QSFP+ copper, and optical networking
• Ruggedized and rack-mountable

Snowball Edge key features
S3-compatible endpoint
File interface (NFS)
Clustering
Run AWS Lambda functions
Faster data transfer
Encryption

How Snowball moves data into and out of AWS
Create
a job
Connect the
Snowball
Copy data to
the Snowball
Your data moved
to Amazon S3
In transit to you Delivered to you Delivered to AWS At AWS
Job created Job completed

Snowball Edge: Remote Data Collection
Customer Example: Oregon State University
• Collect and analyze oceanic and coastal images (60TB/week)
• Environmental and ocean ecosystem research
• Prior to Snowball, used many small HDDs – took weeks to months to upload
• $4MM+ in infrastructure investment -- Expensive and inefficient
• Customer Quote: “Snowball lets us migrate TBs of data in days at a fraction of the
cost”

Snowmobile case study: DigitalGlobe
• Use case: Seeing a better world
• DigitalGlobe takes Satellite imagery of the Earth
• 100PB image library = 6 billion square kilometers
• 1PB new image every year
• Architecture before Snowmobile:
• Stored data in their own data center
• Needed elastic compute power to retrieve and analyze images
• Wanted to move data to the cloud, but no feasible solution
• Snowmobile lets DigitalGlobe migrate 100PB of data to the cloud

Snowmobile from Space
Picture taken by DigitalGlobe’s WorldView-3 satellite
http://blog.digitalglobe.com/industry/digitalglobe-moves-to-the-cloud-with-aws-snowmobile/

When to use AWS Snow*
Cloud
Migration
Disaster
Recovery
Data Center
Decommission
Content
Distribution

S3 Transfer Acceleration:
moving large files over long distances

Amazon S3 Transfer Acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Change your endpoint, not your code
Leverages 59 global edge locations
Optimized protocols
No firewall exceptions
No client software required

Getting Started
• Enable Amazon S3 Transfer Acceleration for their bucket
• DNS resolution
– <bucket>.s3-accelerate.amazonaws.com resolves to
the nearest POP location for the client
– uses the client IP and CloudFront latency based routing
• Customers update application / destination URI to:
• <bucket-name>.s3-accelerate.amazonaws.com
• File is uploaded to:
• <bucket-name>
S3 Management
Console, AWS SDK,
AWS CLI

Rio De
Janeiro
Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore
Time [hrs]
500 GB upload from these edge locations to a bucket in Singapore
Public Internet
How fast is S3 Transfer Acceleration?
S3 Transfer Acceleration
On average, we have seen 171%
improvement over regular S3 when
uploading over long distances

Use case: Media upload
”
“ S3 transfer acceleration reduces the average time it takes for us to
ingest videos from our global user base by almost half. This gives
our customers the ability to edit and share videos sooner where
speed is a critical factor. - Brian Kaiser, CTO
Typical Friday during football season – over 35 hours of video
every minute is uploaded
Data in: User > S3-TA > S3 > Transcode > Redshift
Data out: S3 > CloudFront > User
S3-TA = >20% increase in upload and encoding speeds

Storage Gateway:
augmenting existing on-premises storage with AWS

Storage Gateway hybrid storage solutions
Use standard storage protocols to access AWS storage services
AWS Storage
Gateway
Amazon EBS
snapshots
Amazon S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Files
Volumes
Tapes
On-premises
AWS Cloud

Where does Storage Gateway store data?
us-east-1
Storage Gateway Service
Data Center
iSCSI
HTTPS

File Gateway
On-premises file storage maintained as objects in Amazon S3
Customer Premises
Data stored and retrieved from your Amazon S3 buckets
One-to-one mapping from files-to-objects
File metadata stored in object metadata
Bucket access managed by IAM role you own and manage
Use S3 Lifecycle Policies, versioning, or CRR to manage data
GlacierS3 Standard S3 Standard -
Infrequent
Access
HTTPS
NFS
v3 / v4.1
Application
Server
Storage
Gateway
Storage
Gateway VM

File Gateway use cases
Storage
Gateway VM
NFS client
RefreshCache
Amazon
EMR
In-cloud workload
S3 Bucket
Am azon
Athena
Amazon
QuickSight
S3 Bucket
Snowball + Gateway
NFS
client
Storage
Gateway VM
RefreshCache
Storage
Gateway VM
S3 BucketRead-only NFS
client
Cross region
replication
File Sharing
Site A
Site B
NFS
client

Enabling cloud workloads
Move data to AWS storage for big data, cloud bursting, or migration
“Storage Gateway has the promise to transform the way we move data into
the cloud. The NFS interface lets us easily integrate data files from
analytical instruments, and the transparent S3 storage lets us easily
connect our cloud-based applications and leverage the powerful storage
capabilities of S3.
With Storage Gateway, we can now unleash the full power of AWS on our
instrument data.”

Application
Server
Volume Gateway
On-premises volume storage backed by Amazon S3 with EBS snapshots
• Block storage in S3 accessed via the volume gateway
• Compression of data in-transit and at-rest
• Backup on-premises volumes to EBS snapshots
• Create on-premises volumes from EBS snapshots
• Up to 1PB of total volume storage per gateway
Amazon EBS
snapshots
Customer Premises
Storage
Gateway
Service
iSCSI HTTPS
Volume
Gateway VM
Volume Stored in
Amazon S3

Application
Server
Volume Gateway with existing data
Customer Premises
SAN Volume

Application
Server
Volume Gateway with existing data
Customer Premises
iSCSI
SAN Volume
Amazon EBS
snapshots
Storage
Gateway
Service
Volume Stored in
Amazon S3
Volume
Gateway VM

Tiering storage into Amazon S3 and EBS
Easily add AWS storage to your on-premises environment
“Storage Gateway is at the core of our disaster recovery and
business continuity (BCM) processes, handling our Co-Lo'd
OLTP and OLAP off site data backups, as well as our in-office
BCM. It works transparently, in a lights out way,
archiving off to a separate AWS account with a simple
grandfather-father-son snapshot plan in place.”

Durable offsite storage
Easily store important data in a durable remote site
“As our business expanded, it became clear that using on-
premises backup processes to store and recover these records
was massively inefficient, not scalable, and very costly. Using
the AWS Cached Volume Storage Gateway, we can retrieve
these records very quickly.”

Flexible storage
Easily expand storage capacity on-demand
“Storage Gateway allows us to store backups offsite, create
online archives of big files, and easily expand our file servers.
With cached volumes, we have the most used files local, and
all of the files are stored safely in AWS. We use Storage
Gateway whenever we need a quick way to have big storage
available.”

Data center SMB server with SGW backend
SMB hosted onsite, blocks stored durably in Amazon S3
Private Datacenter
us-west-2
Storage
Gateway VM
Storage
Gateway
Volume
Windows
Clients
Windows Server
HTTPSiSCSISMB

High availability Windows Server storage
Local HA with remote DR
us-west-1
Data Center
Windows
Clients Storage
Gateway VMMicrosoft DFS
HTTPSiSCSI
HTTPSiSCSI
SMB
us-east-1
Microsoft DFS
HTTPSiSCSI
us-east-1
us-west-1

Tape Gateway
Virtual tape storage in Amazon S3 and Amazon Glacier with VTL management
Virtual tape storage in S3 and Glacier accessed via tape gateway
Data compressed in-transit and at-rest
Up to 1 PB total tape storage per gateway, unlimited archive capacity
Supports leading backup applications
**3-5 hour tape retrieval from Amazon Glacier
Archived Tapes
stored in
Amazon Glacier
MEDIA
CHANGER
TAPE
DRIVE
Customer Premises
Virtual Tapes
stored in
Amazon S3
Backup
Server
HTTPSiSCSI
Tape
Gateway
Tape
Gateway VM

Backup, archive, and disaster recovery
Cost-effective storage in AWS with local or cloud restore
“Tapes are a headache, prone with hardware
failures, offsite storage costs, and constant
maintenance needs. Storage Gateway provided the
most cost-effective and simple alternative. We
even got disaster recovery by using a bi-coastal
data center.”

GlacierS3 Standard S3-Infrequent
Access
File Gateway
PetroBank
Application
ServersLTO
NAS
Active archive migration from LTO
Cost-effective storage in AWS with local data access
AWS Direct
Connect
Self service loading of data
Reduced time-to-data by days or weeks
Storage archive costs reduced by 90%
Halliburton Data Center
AWS Lambda

EFS with DirectConnect (DX):
working over a dedicated network

Access your EFS file system via AWS Direct Connect
Direct Connect EFS in your Amazon VPCOn-premises Servers

Partner Tiering & Migration Solutions:
working with what’s already there

Storage Partner Solutions
Technology Solutions vetted by the AWS Storage Competency Program
aws.amazon.com/backup-recovery/partner-solutions/ Note: Represents a sample of storage partners
Note: Dell-EMC, IBM and Veritas have solutions and are working towards competency requirements
Backup and RecoveryPrimary Storage Archive BCDR
Solutions that leverage file, block, object,
and streamed data formats as an
extension to on-premises storage
Solutions that leverage Amazon S3 for
durable data backup
Solutions that leverage Amazon Glacier
for durable and cost-effective long-term
data backup
Solutions that utilize AWS to enable
recovery strategies focused on RTO
and RPO requirements

Pop-up Loft

AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, and Data Transfer Services

More Related Content

AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, and Data Transfer Services