AWS Data Transfer Services: Deep Dive - SRV302 - Chicago AWS Summit

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ian Perez Ponce, Sr. Business Development Manager, AWS
Paul Reed, Principal Product Manager, AWS
SRV302
AWS Data Transfer Services: Deep Dive

What’s driving storage relevance?
Artificial
Intelligence
Natural Language
Processing
Internet of
Things
Information
Assets
Data Trust
Frameworks
Unlimited Storage Scale

AWS cloud storage is core
Building cloud-native applications or migrating
existing ones to AWS …
✓ Advanced developer tools
✓ Experienced consulting and support
✓ Methodical migration services
✓ The most data movement services
Gives you unique scale ...
✓ Greatest reliability
✓ Broad security and compliance
✓ Diverse portfolio
✓ Fastest innovation
✓ Most big data & data lake
deployments
✓ Most managed databases
✓ Easiest data warehousing
✓ Singular query-in-place analytics
Yielding bigger insights ...
Helping you innovate faster ...
✓ Artificial Intelligence
✓ Deep Learning / Machine
Learning
✓ IoT
Data matters at
any scale

Data movement
OnlineOffline
Data security
and management
Amazon
EFS
Amazon
EBS
Amazon
S3
Amazon
Glacier
AWS KMS
AWS IAM
AWS CloudWatch
AWS CloudTrail
AWS CloudFormation
AWS Lambda
Amazon Macie
Amazon QuickSight
AWS Snow Family
AWS Storage Gateway
AWS Direct Connect
Amazon EFS File Sync
Amazon S3 Transfer
Acceleration
Third-party applications
Amazon Kinesis Firehose
The broadest range of storage services

AWS storage customers

AWS offers the most ways to move data to / from the cloud
AWS
Direct
Connect
A private
connection
between your data
center, office, or
colocation
environment and
AWS.
AWS Snow
family
(Snowball, Snowball
Edge, Snowmobile)
Secure, physical
transport
appliances that
move up to
exabytes of data
into and out of
AWS.
AWS
Storage
Gateway
Hybrid storage that
seamlessly connects
on-premises
applications to AWS
storage. Ideal for
backup, DR, bursting,
tiering, or migration.
Amazon
Kinesis Data
Firehose
Capture, trans-
form, & load
streaming data
into Amazon S3 for
use with Amazon
business
intelligence and
analytics tools.
Amazon EFS
File
Sync
Up to 5x faster file
transfers than open-
source tools. Ideal
for migrating data
into Amazon EFS or
moving between
cloud file systems.
Amazon S3
Transfer
Acceleration
Up to 300% faster
transfers into and
out of Amazon S3.
Ideal when
working with long
geographic
distances.
APN
competency
partners
Integrations
between third-party
vendors and AWS
services. Ideal for
leveraging existing
software licenses
and skills.
Networks Roads Hybrid

Data migration fundamentals
Cloud storage tier
selection
Data discovery
and preparation
Data validation Data marshalling
Transfer method
selection
Step 1 Step 2 Step 3 Step 4 Step 5

Backup & Restore Archive Primary Storage BC/DRData Migration
AWS Partner Network: Migration & storage

Large-scale offline data transfer
& edge processing
with the AWS Snow family

What is AWS Snowball, and why
did we build it?
AWS Snowball AWS Snowball Edge
Moving large volumes of data over the internet can take
years – we ship secure physical devices to you to transfer
your data at the source before shipping it back for bulk
import to the cloud.
The cloud is not always accessible from remote locations
where connectivity is limited or intermittent – deploy
ruggedized devices at the edge with local storage and
compute capacity to process data without network
dependencies.
Traditional shipping of conventional hard drives is
laborious and error prone – our E-Ink shipping label and
chain of custody tracking simplifies logistics at scale.

AWS Snowball AWS Snowball Edge AWS Snowmobile
• 50 or 80 TB storage capacity
• 10 GE networking
• Data encryption end-to-end
• Rugged 8.5 G impact case
• Rain and dust resistant
• 100 TB storage capacity
• 10/25/40 GE networking
• Rugged 8.5 G impact case
• Rain and dust resistant
• AWS Greengrass support for local
compute, messaging and caching
• Amazon EC2/AMI support for
edge compute
• Exabyte-scale storage in a 45-foot
container
• Dedicated security personnel
• GPS tracking, alarm monitoring,
24/7 surveillance, and optional
additional security
AWS Snow family

Specifications and pricing details
Storage 50 or 80 TB 100 TB 100 PB
Interfaces CLI, S3 SDK S3, NFSv4 NFSv4
Network 10G 10/25/40G 40/100G
Power 200W 400W ~350kW
Compute NA m4.4xl (equivalent) NA
Job Fee (Import) $200 or $250 per job $300 per job $0.005/GB per month
Daily Rate $15/day after 10th day $30/day after 10th day NA
Snowball Snowball Edge Snowmobile
Commercially available in 16 regions.

Do you need the truck?
< 10PB > 10PB

Amazon S3 compatible
SDK adapter
Snowball CLI
tools
Embedded E-Ink
display for shipping
Chain of custody
tracking
Automated small file
batching
Storage density
alternatives
Snowball key features
SOC, PCI, & HIPAA
compliant

Amazon S3 Compatible
Endpoint
NFS File
Interface
Storage and compute
cluster
AWS Greengrass and
Lambda support
Line rate data
transfer
Hardware enabled data
encryption
Snowball Edge key features
EC2 Compute
Instances support

Who’s using the AWS Snow family today?
ISV PartnersCustomers

And for what use cases?
W h o l e s a l e d a t a
c e n t e r m i g r a t i o n
B a c k u p s e e d i n g
A s s i s t e d d a t a b a s e m i g r a t i o n
N A S a p p l i a n c e a r c h i v a l
D a t a l a k e c r e a t i o n
A c t i v e a r c h i v e

Snowball mechanics

Step 1: Job creation
Customer Premises Region
AWS console
Snowball
Service

Step 2: Local data transfer
AWS console
Snowball
Service
Local
Infrastructure
Snowball
Service
S3 bucket
S3 bucket

Step 3: Ingest to Amazon S3 (or other storage tiers)
AWS console
Snowball
Service
Local
Infrastructure
Snowball
Service
S3 bucket
S3 bucket
Amazon EBS
Amazon EFS
Amazon
Glacier
AWS Storage
Gateway

Data path options between Snowball v1 and Snowball
Edge
Workstation Snowball v1
Snowball client
Workstation Snowball v1
(S3 adapter)
Workstation
S3 api
Workstation
Workstation
Amazon S3 API / NFS
Workstation
Snowball
Edge

Edge computing – when migrating data
over distance isn’t an option.

Where do we see edge computing in effect?
Research Industrial Healthcare Transportation
Various industries ranging from research to transportation necessitate edge
computing capabilities to help cope with environments where data generation is
decentralized, data volumes are significant, and network connectivity is either
inaccessible or intermittent.

Transportation /
Automotive
Problem
Auto manufacturers and suppliers operating
autonomous vehicle fleets for research and
development are challenged with the increasing
volume and sophistication of instrument data
being collected.
Solution
Several leading manufacturers and suppliers are
looking to Snowball Edge to not only help with
migrating data to the cloud, but to also process
data at the edge where localized machine
learning and low-latency data analytics can be
performed.
Impact
Petabyte-scale datasets being generated on a
monthly basis no longer have to be migrated to
the cloud in their entirety, reducing both costs
and time-to-value as a result of edge
computing.

Amazon EC2 compute instances on Snowball Edge
What are we announcing
• Support for EC2 sbe1 instances
• Three AMIs available on AWS
Marketplace
• SBE1 instances feature 1.8 GHz
Intel Xeon D processors, up to 24
vCPUs, 32 GiB of memory
• Up to 1 TB disk volumes
• 1- and 3-year discounted pricing
options
Supported EC2 instance sizes
Snowball
Edge
Model vCPU Mem (GiB)
sbe1.small 1 1
sbe1.medium 1 2
sbe1.large 2 4
sbe1.xlarge 4 8
sbe1.2xlarge 8 16
sbe1.4xlarge 16 32

Snowball Edge clustering for scale
Clustering features
• Scale from 5 to 10 nodes in a
Snowball Edge cluster
• Clusters support aggregate
storage and compute with
leaderless nodes
• Increase durability for Amazon S3
object storage on premises
• Easily swap nodes for
maintenance
Scale-out
5 Nodes
80 vCPU
320 GB
225 TB
10 Nodes
160 vCPU
640 GB
550 TB
AWS
Snowman Cluster

How Amazon EC2 instances work on Snowball Edge
Customer AWS Snowball Edge VM Import/ExportPlaces order using Console/CLI/SDK
Provides AMI
Snowball accepts job and provides job id
Provides amis with s-ami.*
Snowball installs
KVM images on device
Snowball asks VM Import/Export
to convert AMI to KVM
Device is shipped to customer
customer can use EC2 commands to manage instances
VM Import/Export
provides KVM images

Amazon EC2 instances on Snowball Edge pricing

Healthcare
Using Snowball Edge to support medical imaging or
optical scanning MRI machines
• Stores the image files as they are captured
• Gives local users and administration systems
immediate access
• Used a cluster of Snowball Edge devices to
stage imaging data for later into Amazon S3
without disrupting systems onsite
“Snowball Edge enables us to extend the
innovative capabilities of HealthSuite,
our cloud-enabled connected health
ecosystem of devices, applications and
digital tools supported by AWS, even
when there is no network support.”
− Dale Wiggins,
Business Leader, HealthSuite digital
platform, Philips

Academic research
Environmental and ocean ecosystem research
• Collect and analyze oceanic and coastal images
• Able to migrate and process 60 TB of data per
week at a fraction of the cost
Before Snowball Edge:
• Transferred data with many small hard drives
• Used to take weeks to months to upload data
prior to processing
• $4MM+ in infrastructure investment
• Expensive and inefficient
“With AWS Snowball Edge, we can now
collect 100 TB of data with no
intermediate steps, and we can also
analyze the images immediately using
the onboard compute capabilities. This
allows us to do deeper analysis, and we
can upload all the raw data to the AWS
Cloud by simply shipping the AWS
Snowball Edge device back.”
− Bob Cowen,
Director of Hatfield Marine Research Center,
Oregon State University

Hybrid cloud storage architectures

AWS Storage Gateway enables a spectrum of hybrid use cases
Backup | DR | Archive
Enterprise data centers
Amazon
EBS
Amazon
S3
Amazon
Glacier
App serversFile servers
Research sites
AWS Storage Gateway
DevicesDatabasesMultimedia content
Analytics | File Services | Machine Learning | Data Processing
Data Distribution | Backup | DR | Archive | Migration
Amazon
EC2
AWS
Lambda
Amazon
CloudFront
Amazon
Athena
Amazon
EMR
Backup serversUsers
Remote offices Small to medium businesses
Amazon
Rekognition
Amazon
MachineLearning

AWS Storage Gateway
Amazon EC2
IAM
Amazon CloudWatch AWSKeyManagementService
(AWSKMS)
AWS CloudTrail
Files
(NFS / SMB)
Volumes
(iSCSI)
Tapes
(iSCSI VTL)
Amazon S3
Amazon Glacier
Amazon EBS
snapshots
AWS RegionYour data center
Storage Gateway
HTTPS
Gateway Service

AWS Storage Gateway family
File gateway
Store and access objects in Amazon S3 from file-based applications
with local caching
Volume gateway
Block storage on-premises backed by cloud storage with local
caching, Amazon EBS snapshots, and clones
Tape gateway
Drop-in replacement for physical tape infrastructure backed by cloud
storage with local caching
Hybrid storage service enabling applications to seamlessly use AWS storage

Standard storage protocols
AWS storage accessible without applications
needing to be modified
Fully managed cache
Frequently used data cached locally
for low-latency access
Durable storage
On-premises application data natively stored in
Amazon S3, Amazon Glacier, and Amazon EBS
snapshots
AWS Storage Gateway key features
Optimized data transfer
Secure upload of changed data and downloads
requested data

File Gateway for hybrid cloud file workloads

File Gateway
Store and access objects in Amazon S3 from file-based applications with local caching
Customer Premises
HTTPS
NFS v3/v4
SMB v2/v3
File Gateway Objects in your
S3 buckets
Application
Server
Per file share options
IAM role
Object storage class
Encryption with AWS KMS
Guess MIME type, requester pays, bucket
owner ACL, etc.
Per S3 bucket options
Restrict access by client IP (NFS) or Active Directory (SMB)
users/groups
POSIX permissions for object-level access*
Read-only/read-write
* Compatible subset of NTFS

Native SQL backup to Amazon S3 via SMB
Corporate data center
Storage Gateway VM
SQL Server
(native SQL agent)
Domain
controller
US-West-1
Amazon
SQS
Amazon
CloudWatch
AWS
Storage
Gateway
Amazon Glacier
Amazon S3 − Infrequent Access
Backup
bucket Expire / delete backup after x years
Share
(SMB)
Local
cache
Lifecycle after 30 days

Amazon
Glacier
Amazon S3
Standard
S3-Infrequent
Access
File Gateway
PetroBank
Application
ServersLTO
NAS
Active archive migration from disk & tape
Cost-effective storage in AWS with local data access
AWS Direct
Connect
Halliburton data center
1
2
Use Snowball to ship data from on-premises offline archives1
2 Online access to all data through gateway
Minimal on-premises storage reduces cost
Time-to-date by reduced by days or weeks

Hybrid file use case: Content distribution
Seattle data center
File gateway
(read-only)
Application
Boston data center
Objects in your S3
bucket
AWS Region
File gateway
1
Application
2
Application in Seattle writes files, which are uploaded to Amazon S3 by gateway1
2 After refresh cache, files are visible to applications in Boston
Local cache improves access performance

Hybrid volumes for recovery & migration
with volume gateway
and Amazon EBS snapshots

Application
server
Amazon EBS
snapshots
Storage GatewayVolume
gateway
appliance
Volume stored
in
Amazon S3
HTTPSiSCSI
Customer premises Region
Volume gateway
Cloud-backed block storage presented on-premises
• Tier snapshots or whole volumes to the cloud to reduce SAN/NAS mgt.
• Flexible recoveries in-cloud or on-premises with snapshots and clones
• Common uses: backup and restore, disaster recovery, data migrations

Application
server
Amazon EBS
snapshots
AWS Storage
Gateway
Volume
gateway
appliance
Volume stored
in
Amazon S3
HTTPSiSCSI
100% of volume stored in
AWS & on-premises
Volume gateway: Stored mode
Low-latency access to all your data with point-in-time backup to the
cloud through Amazon EBS snapshots

Amazon EBS
snapshots
Storage Gateway Volume stored
in
Amazon S3
HTTPSiSCSI
100% of volume
stored in AWS
Volume gateway VM
Virtual
volume
Fully managed cache of
frequently used data
Application
server
Volume gateway: Cached mode
Reduce on-premises storage, caching frequently used data local to your
application, with 100% of your data in the cloud

Migration & data protection w/ snaps & clones
Volume clones
Instant real-time copy of a cached volume
Represents current state of volume stored in AWS
Restore as a Storage Gateway volume
EBS snapshots
Point-in-time backups of a stored or caches volumes
Created on-demand or on a configurable schedule
Restore either as an Amazon EBS or a Storage Gateway volume

1. Restore to your
data center
Migration & data protection with volume gateway
Storage Gateway
3. Recovery to a 2nd
DR site
EC2EBSGateway
volume
1
2
3
2. Migrate to AWS

Canada’s largest biotech firm
• Data sovereignty required local hot files
& tape archives in 10 global offices
• Volume Gateway eliminated 50-hour backup
windows and tape archive systems
• Cut on-premises; storage CAPEX 40%;
reduced RTO from 48 hours to 10 minutes
• Meets cloud strategy while retaining local
ownership and data sovereignty
• Enabled data center exit in next 6–12 months
“It made no sense to keep buying
big disk siloes, especially as we
opened up new global offices, and
now we can recover in the cloud from
a snapshot if we ever had to.”
− Adam Leggett
IT manager
Stemcell’s backup & restore with volume gateway

Migrate tape backup workflows to AWS
with Tape Gateway

Tape gateway: Drop-in replacement for tape backups
• Emulates a tape library. Virtual tapes on Amazon S3 and Amazon Glacier.
• Works with common backup apps, to support existing backup workflows.
• Low-cost: Predictable costs and reduced management.

Tape gateway: How the VTL works
Customer premises
Region
Storage Gateway
Backup server
Upload buffer
Cache
Media
changer
Tape Drive
Tape Drive
Tape Drive
Tape Drive
Tape Drive
Tape Drive
Tape library
(Amazon S3))
Tape shelf
(Amazon Glacier)
• Emulates a physical tape device with a media changer and tape drives
• Scalable: Virtually unlimited tape storage in AWS
• Virtual tapes are written to Amazon S3: Data is in Amazon S3 when tape is in virtual library drive or slot
• “Ejected” virtual tapes are marked read-only and moved to “Tape Shelf” on Amazon Glacier
• Recovery: Retrieve tapes to library (3–5 hours) and read data to same or different gateway
Tape gateway VM
Tape Drive

Submit session feedback
1. Tap the Schedule icon.
2. Select the session you
attended.
3. Tap Session Evaluation to
submit your feedback.

Questions?

Thank you.

AWS Data Transfer Services: Deep Dive - SRV302 - Chicago AWS Summit

Related slideshows

More Related Content

AWS Data Transfer Services: Deep Dive - SRV302 - Chicago AWS Summit