Deep Dive on Amazon Elastic File System (Amazon EFS) (STG301-R1) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep Dive
on Amazon Elastic File System
Duncan Lennox
Director, Product Management
Amazon Web Services, Inc.
S T G 3 0 1 - R
Vince Carreon
Senior Product Manager
Amazon Web Services, Inc.

Agenda
AWS storage overview
Introduction to Amazon Elastic File System (Amazon EFS)
Deep dive
Customer use cases
Best practices
Summary and Q&A

Magic Quadrant
Magic Quadrant for
Public Cloud Storage
Services, Worldwide – 2018
Positioned furthest for
completeness of vision
and highest for ability
to execute in each report
since inception in 2014
Magic Quadrant for Public Cloud Storage Services, July 2018 – Raj Bala, Julia Palmer
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the
context of the entire document. The Gartner document is available upon request from Amazon Web Services.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise
technology users to select only those vendors with the highest ratings or other designation. Gartner research
publications consist of the opinions of Gartner’s research organization and should not be construed as statements
of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any
warranties of merchantability or fitness for a particular purpose.

Portfolio breadth and depth
Amazon
EFS
AWS Storage
Gateway
Family
Amazon S3
Amazon
EBS
Amazon
EC2

More choice—Best fit
Object storage
S3 Standard
S3 Glacier Deep Archive
S3 Glacier
S3 Intelligent-Tiering
S3 One Zone-IA
S3 Standard-IA
Block storage
Provisioned IOPS SSD
Cold HDD
Throughput-Optimized HDD
NEW!
COMING
SOON!
File storage
EFS Standard
EFS Infrequent Access
COMING
SOON!
Elastic
Amazon EFS
AWS Storage
Gateway Family
Amazon S3
NEW!
Amazon FSx
for Lustre
Amazon FSx
for Windows
File Server
NEW!
Amazon
EBS
Amazon
EC2

Why does network file storage matter?
Large majority of data is unstructured files
Large percentage of applications operate on file and file systems
Re-platforming on AWS is simplified using AWS native services
AWS services enable existing applications and IT environments

Fully managed cloud file systems
Amazon EFS Amazon FSx for
Windows File Server
Amazon FSx
for Lustre
LINUX-BASED WORKLOADS WINDOWS-BASED WORKLOADS
Fully managed Windows file servers
for business applications
Fully managed cloud-native file system
for Linux-based applications
Fully managed Lustre file system for
compute-intensive workloads
COMPUTE-INTENSIVE WORKLOADS
AWS provides file system options that help you easily address
the diverse needs of your file-based applications and workloads

Before Amazon EFS… DIY shared file storage
VPC
AZ-aAZ-b
File server
File server Storage
Volumes
Clients Storage
Volumes

Key benefits of Amazon EFS
Simple

ElasticSimple

ElasticSimple Scalable

ElasticSimple Scalable
Highly available and durable

Amazon EFS 2018 year in review
March
• US West (N.
California)
April
• Encryption of
data in transit
May
• Asia Pacific
(Seoul)
• HIPAA
• GDPR
July
• Asia Pacific (Tokyo)
• Provisioned Throughput
• PCI-DSS
August
• Asia Pacific
(Singapore)
October
• Access EFS over AWS
VPN
November
• 512 Locks per file (was 87)
• Multi-VPC access
• 1000 file systems per
account (was 125)
re:Invent
2017
Today
NEW!
NEW!

Deep dive
Elastic and
scalable
Highly available
& durable
Shared
access
Performance
modes
Throughput
modes
Security &
new
Integrations
Cost-effective

Elastic & scalable
Petabyte scalable
file system
Grow and shrink
on demand
No provisioning
necessary
Performance scales
as capacity grows
Mount thousands
of instances with
concurrent access to
a single file system

Available & durable
Highly available and durable
(designed for 11 9’s of
durability)
Distributed
architecture
Multi-AZ Strong consistency Data redundantly stored
within and across AZs
for durability

Shared access
VPC
Amazon EFS
File System
NFS Clients

Shared access
VPC
Amazon EFS
File System
On Premises
Servers
AWS Direct
Connect
NFS Clients

Shared access
VPC
Amazon EFS
File System
VPC
Intra-region
VPC Peering
Other AWS Regions
On Premises
Servers
AWS Direct
Connect
AWS VPN
NFS Clients
NFS Clients

Shared access
VPC
Amazon EFS
File System
VPC
VPC VPC
Inter-region
VPC Peering Intra-region
VPC Peering
Other AWS Regions
On Premises
Servers
AWS Direct
Connect
AWS VPN
Transit
Gateway
NFS Clients
NFS Clients
NFS Clients
NFS Clients

Shared access
Amazon EFS
File System
NFS Clients
Account B
Shared VPC
NFS Clients
Account A
NFS Clients
Account C
Amazon EFS available across accounts in Shared VPCs

New EFS file system limit
Increased
1000 File Systems Per
Account (10x
increase)
Added
400 Mount Targets
Per VPC

Performance modes

Performance modes
General purpose
(default)
Recommended for the
majority of workloads

Performance modes
General purpose
(default)
Recommended for the
Max I/O
Recommended for
scale out workloads

Performance modes
General purpose
(default)
Recommended for the
Max I/O
Recommended for
scale out workloads
What it is for
Latency-sensitive applications
and general-purpose workloads
Large-scale and
data-heavy applications
Advantages
Lowest latencies
for file operations
Virtually unlimited ability to
scale out throughput/IOPS
Trade-offs
Limit of 7k ops/sec Slightly higher latencies
When to use
Best choice for most
workloads
Consider for large
scale-out workloads

Throughput modes

Throughput modes
Bursting throughput
(default)
Recommended for the

Throughput modes
Bursting throughput
(default)
Recommended for the
Provisioned
throughput
Recommended for higher
throughput to storage
ratio workloads

Throughput modes
Bursting throughput
(default)
Recommended for the
Provisioned
throughput
Recommended for higher
throughput to storage
ratio workloads
What it is for
Advantages
Auto-scaling throughput User-defined throughput
Trade-offs
Fixed throughput to
storage ratio
Separate
throughput charge
When to use
Best choice for most
workloads
Ingest or higher throughput
to storage ratio
Varying throughput
workloads
Higher-consistent
throughput workloads

Provisioned throughput mode

Provisioned
throughput mode
Independent
throughput
Provision throughput
independent of
data stored

Provisioned
throughput mode
Independent
throughput
independent of
data stored
Increase
As often as you need

Provisioned
throughput mode
Independent
throughput
independent of
data stored
Increase
As often as you need
Switch or
decrease
Once every 24+ hours

Coming soon!
New storage class for Amazon Elastic
File System reduces cost by up to 85%
Introducing Amazon
Elastic File System
Infrequent Access
No changes to existing
applications using EFS
Cost savings
up to 85%
Automated lifecycle
management

Amazon Elastic File
System Infrequent Access
How it works
Create your
file system
Enable
Lifecycle
Management
Files added and not
accessed for 30 days
are moved to EFS IA
Coming soon!

Security model & compliance

Security model &
compliance
Control
network traffic
using Amazon VPC
security groups and
network ACLs

Security model &
compliance
Control
network traffic
using Amazon VPC
security groups and
network ACLs
Control file and
directory access
using POSIX
permissions

Security model &
compliance
Control
network traffic
using Amazon VPC
security groups and
network ACLs
Control file and
directory access
using POSIX
permissions
Control administrative
access (API access)
using AWS IAM
(action-level and
resource-level permissions)

Security model &
compliance
Control
network traffic
using Amazon VPC
security groups and
network ACLs
Control file and
directory access
using POSIX
permissions
access (API access)
using AWS IAM
(action-level and
Encrypt data
at rest and in
transit

Security model &
compliance
Control
network traffic
using Amazon VPC
security groups and
network ACLs
Control file and
directory access
using POSIX
permissions
access (API access)
using AWS IAM
(action-level and
Encrypt data
at rest and in
transit
Achieve
compliance
HIPAA
GDPR
PCI-DSS
SOC
ISO

AWS DataSync
AWS
Online transfer service that simplifies, automates, and
accelerates moving data between on-premises storage and AWS
Combines the speed and reliability of network acceleration
software with the cost-effectiveness of open source tools
FAST DATA
TRANSFER
EASY
TO USE
SECURE AND
RELIABLE
CLOUD
INTEGRATED
COST-EFFECTIVE

Amazon EFS economics
No minimum
commitments
or upfront fees
No need to provision
storage in advance
No other fees
or charges

Bursting throughput
mode pricing
Storage price
$0.30/GB-month*
Includes 50 KiB/s throughput per GiB of storage
Single pricing dimension
* US N. Virginia pricing

Bursting throughput
mode pricing
Storage price
$0.30/GB-month*
Two pricing dimensions

Bursting throughput
mode pricing
Storage price
$0.30/GB-month*
Two pricing dimensions
Throughput price
$6.00/MiBps-month*
Above 50 KiB/s throughput per GiB of storage

Storage Classes and
lifecycle management
Standard storage price
$0.30/GB-month*
Infrequent Access storage and access price
$0.045/GB-month*
$0.01/GB for access*

Before Amazon EFS…
DIY file storage costs
VPC
AZ-aAZ-b
Amazon EC2
instance costs
Inter-AZ data
transfer costs
Amazon EBS
volume costs
File server
File server Storage
Volumes
Clients Storage
Volumes

TCO example
Amazon EFS
Amazon EFS cost:
Standard storage class (100 GB * $0.30/GB-month*): $30 per month
Infrequent Access storage class (400 GB * $0.045/GB-month*): $18 per month
Infrequent Access data access (400 GB*1*$0.01/GB*) = $4 per month
TOTAL: $52 per month
DIY - provisioning 600 GB of Amazon EBS (i.e., ~85% utilization):
Storage (2x 120 GB EBS gp2 volumes): $24 per month
Storage (2x 480 GB EBS sc1 volumes): $24 per month
Compute (2x m5.xlarge instances): $281 per month
Inter-AZ data transfer costs (est.): $130 per month
TOTAL $459 per month

TCO example
Amazon EFS
Amazon EFS cost:
Standard storage class (100 GB * $0.30/GB-month*): $30 per month
Infrequent Access storage class (400 GB * $0.045/GB-month*): $18 per month
Infrequent Access data access (400 GB*1*$0.01/GB*) = $4 per month
TOTAL: $52 per month
DIY - provisioning 600 GB of Amazon EBS (i.e., ~85% utilization):
Storage (2x 120 GB EBS gp2 volumes): $24 per month
Storage (2x 480 GB EBS sc1 volumes): $24 per month
Compute (2x m5.xlarge instances): $281 per month
Inter-AZ data transfer costs (est.): $130 per month
TOTAL $459 per month
Amazon
EFS is
~90% less
than DIY

Where is Amazon EFS
available today?
• US West (Oregon)
• US West (N. California)
• US East (N. Virginia)
• US East (Ohio)
• EU (Ireland)
• EU (Frankfurt)
• Asia Pacific (Sydney)
• Asia Pacific (Seoul)
• Asia Pacific (Tokyo)
• Asia Pacific (Singapore)
Coming soon:
EU (London) and AWS GovCloud (US)

Designed for a wide spectrum of needs
Analytics
Media workflows
Lift and shift Enterprise apps
Web serving
Content management
Database backups
Container storage
Application testing
and development
Scale-out jobs Metadata-intensive jobs
High throughput and parallel I/O Low latency and serial I/O

EFS customers

Amreth Chandrasehar
Principal Architect, Domain
Amreth.Chandrasehar1@t-mobile.com

About T-Mobile
 As America's Un-carrier, T-Mobile US, Inc. is
redefining the way consumers and businesses
buy wireless services through leading product
and service innovation.
 NASDAQ traded public company – TMUS
 Operating two flagship brands: T-Mobile and
Metro by T-Mobile
 Based in Bellevue, Washington

Why EFS?
 Fully managed
 Scalable
 Performance mode—Default, Max I/O
 Throughput mode—Bursting, Provisioned
 Security—Encryption at rest, PCI compliant
 Less expensive compared to DIY

Architectural patterns
EFSwithKubernetes CICD–Jenkins
SlavesonMesosand
K8s
HAandDRforArtifactory TIBCOEMS

EFS use cases and benefits at T-Mobile
Kubernetes
EFS Persistent Volumes, Scalable to 100s of nodes, 70% cost effective than DIY
CICD – Jenkins Slaves on Mesos and K8s
Cache build dependencies, reduces execution time by 10-50%
500+ Short lived Jenkins Slaves per day
Artifactory HA and DR
Leverage Max I/O and Provisioned Throughput modes
~200 TB of data across regions with 50,000+ artifacts
TIBCO EMS
Capability to run highly available EMS on Amazon Elastic Compute Cloud (Amazon EC2)

Architectural patterns—K8s apps using EFS

Architectural patterns—CICD on K8s and Mesos

Architectural patterns—Artifactory HA/DR

Architectural patterns—TIBCO EMS

Throughput test results*
Best practices
• Use multiple threads
• Write to multiple directories in parallel
• Use larger IO size (aggregate IO)
*tested from a single c5.large instance
57.722
16.492 15.460
7.837 7.818

IOPS test results*
Best practices
• Use multiple threads
• Write to multiple directories in parallel
16.219
8.061
0.901

File transfer test results*
Best practices
• Use a multiple-threaded tool
435
329
210
73
42
NEW!
AWS
DataSync

EFS Bursting Throughput
Compute
D U R A T I O N
A V E R A G E
T H R O U G H P U T
1TB of 1MB files
from 1 c5.2xlarge instance
Data ingest
EFS

Compute
Compute
Compute
EFS Provisioned Throughput
Compute
D U R A T I O N
A V E R A G E
T H R O U G H P U T
1TB of 1MB files
from 4 c5.2xlarge instances
Data ingest
2h 53m 105 MB/s
EFS

Best practices—Monitoring

Amazon CloudWatch
DataReadIOBytes
DataWriteIOBytes
MetaDataIOBytes
TotalIOBytes
BurstCreditBalance
PermittedThroughput
ClientConnections
PercentIOLimit*
Amazon
CloudWatch
* General purpose only

Amazon CloudWatch Automatic Dashboards

Best practices
Test with
general purpose
performance mode
Start with bursting
throughput mode
Linux kernel 4.3+ EFS mount helper
(NFSv4.1)
Large IO size
(aggregate IO)
AWS
Multiple
threads
Multiple
instances
Multiple
directories
Monitor
metrics

How to learn more: aws.amazon.com/efs
Feature blogs
Whitepapers
Ref architectures
TCO calculator
10-minute tutorials
Documentation

Amazon EFS tutorials
https://github.com/aws-samples/amazon-efs-tutorial

Thank you!
Duncan Lennox
Director, Product Management
dllennox@amazon.com
Vince Carreon
Senior Product Manager
vcarreon@amazon.com

Deep Dive on Amazon Elastic File System (Amazon EFS) (STG301-R1) - AWS re:Invent 2018

More Related Content

Deep Dive on Amazon Elastic File System (Amazon EFS) (STG301-R1) - AWS re:Invent 2018