Track 5 Session 5_STG03 AWS 檔案儲存服務概觀

Editor's Notes

1. Hi Everyone, Thank you so much for your time! If you are interested in evaluating what it means to build a data lake on AWS, or on your way to building your analytics platform on AWS and just keen to explore some possible architectural patterns, and best practices, this session is for you. My name is Kumar Nachiketa, a storage partner solutions architect at AWS. And today, I am going to discuss and demonstrate ways to build a data lake with Amazon S3 being a core component. I am super excited about sharing this with you.
2. How can you transform faster with AWS file storage solutions? We see three common patterns on why customers choose AWS to transform their organization. In summary, customers are looking to transform their IT infrastructure, transform their application architectures, or transform their business results.
3. EFS: Easily shared between multiple applications, instances, and on-premises servers simultaneously Achieve petabyte scale from a distributed design that avoids the constraints imposed by traditional file servers FSx for Windows: Built on Windows Server with native support for Windows file system features you use today SSD storage for high throughput, IOPS, and sub-millisecond latencies FSx for Lustre: Built on the highly popular, open source parallel file system Lustre Process data at hundreds of GB of throughput per second, millions of IOPs, and sub-millisecond latencies. For scratch/short-lived or long-lived/self-healing workloads (recently introduced at re:Invent).
4. Probably something that almost everyone in the audience can relate to is that managing file servers is a lot of work. Before the cloud, [Manage hardware] Well, now we have the cloud. If managing file servers on AWS, you don’t need to worry about the hardware piece, but you still need to manage the software…
5. Let’s start by talking about what is Amazon Elastic File System (or Amazon EFS). Amazon EFS is Network File System evolved. It is a fully-managed shared file system, that is cloud native, highly reliable, and cost optimized. Cloud native – simple, elastic, scalable, and integrated with cloud services like ECS, EKS, and SageMaker Highly reliable – highly available and durable, secure, global footprint, 99.9% SLA Cost optimized – no provisioning, no commitments, and built-in lifecycle management to optimize between an SSD-performance class and an infrequent access class that is 92% lower cost Cloud Native Grow & shrink on demand No need to provision and manage infrastructure & capacity Pay as you go, payonly for what you use Shared access from on-premises, inter region, and cloud native applications Integrated with various AWS computing models Access concurrently from thousands of Amazon EC2 instances Attach to containers launched by both Amazon ECS and EKS Use with Amazon SageMaker notebooks Highly Reliable/ Scalable Stores data across three availability zones for high availability and durability Amazon EFS isavailable in 19 regions New regions recently added: Bahrain, Sao Paulo, Stockholm, Hong Kong Grow up to petabytes Performance modes for low latencies and maximum I/O Throughput that scales with storage Provisioned throughput available Cost-optimized And that’s exactly what we did. Amazon EFS Infrequent Access or EFS IA is a new storage class that can save customers up to 92% compared to the pre-existent EFS storage class, which we’re now calling EFS Standard. Customers can use EFS IA to cost-effectively store larger amounts of data in their file systems, and expand their use of EFS to an ever wider set of applications. Using EFS IA is super easy. It doesn’t require any changes to existing applications, as EFS provides a single file system namespace that transparently serves files from both storage classes – Standard and IA. To get data into EFS IA, you enable a the Lifecycle Management capability for your file system that automatically moves files into IA, and by doing so, you can save up to 92% compared to the EFS Standard storage class. Note: EFS-IA can be enabled for any existing EFS file system simply by selecting the lifecycle management option in the EFS console Before we dive deeper into these core tenets, let’s talk about why Amazon EFS, lets talk about it’s use cases.
6. Cloud Native Grow & shrink on demand No need to provision and manage infrastructure & capacity Pay as you go, payonly for what you use Shared access from on-premises, inter region, and cloud native applications Integrated with various AWS computing models Access concurrently from thousands of Amazon EC2 instances Attach to containers launched by both Amazon ECS and EKS Use with Amazon SageMaker notebooks
7. Highly Reliable/ Scalable Stores data across three availability zones for high availability and durability Amazon EFS isavailable in 19 regions New regions recently added: Bahrain, Sao Paulo, Stockholm, Hong Kong Grow up to petabytes Performance modes for low latencies and maximum I/O Throughput that scales with storage Provisioned throughput available
8. COST OPTIMIZD And that’s exactly what we did. Amazon EFS Infrequent Access or EFS IA is a new storage class that can save customers up to 92% compared to the pre-existent EFS storage class, which we’re now calling EFS Standard. Customers can use EFS IA to cost-effectively store larger amounts of data in their file systems, and expand their use of EFS to an ever wider set of applications. Using EFS IA is super easy. It doesn’t require any changes to existing applications, as EFS provides a single file system namespace that transparently serves files from both storage classes – Standard and IA. To get data into EFS IA, you enable a the Lifecycle Management capability for your file system that automatically moves files into IA, and by doing so, you can save up to 92% compared to the EFS Standard storage class. Note: EFS-IA can be enabled for any existing EFS file system simply by selecting the lifecycle management option in the EFS console Before we dive deeper into these core tenets, let’s talk about why Amazon EFS, lets talk about it’s use cases.
9. We also designed EFS to serve the vast majority of file-based workloads, covering a wide spectrum of performance needs. This spectrum runs from highly parallelized, scale out jobs – those workloads that require the highest possible throughput – things like big data applications, and media workflows, To single-threaded, latency-sensitive workloads, and everything in between.
10. So what is it? FSx for Windows File Server provides fully managed, native Windows file systems… …with a service that is deeply integrated with AWS
11. On April 8th we announced native integration between ECS and EFS, including Fargate. Compared to what you could do before, there are three main differences. First, using EFS with ECS is much simpler: you configure EFS inside your ECS task definition, and don’t worry at all about what happens under the hood. Next, for the first time you can use EFS from AWS Fargate, which means you now have persistence for serverless containers. Last, we’ve added new security capabilities, namely the ability to use IAM to authorize access to your file system based on your task role, and control access to data using EFS Access Points. Now when you look at overall benefits of ECS and EFS working together, in addition to the simplicity, serverless, and security there are additional benefits. Availability and Durability: Both EFS and ECS are regional services, which means they run across all availability zones in a region. With EFS, data is durable across at least 3 availability zones, and available from all zones. With ECS, clusters span as many AZs as you configure. The end result is your application can be scheduled across multiple AZs and share data as if they’re local to each other. Elasticity: this should be of no surprise given that Elastic Container Service and Elastic File System have elastic in the name. This fact means that as your application needs to scale-out, this combination of services instantly provides additional compute and storage capacity. This means you always pay for what you use, and you don’t need to forecast usage, overprovision, or get slowed down when you run out of capacity.
12. [Our 1st file system launched in 2016 was Amazon Elastic File System (EFS). Designed to provide a cloud-scale file system for the vast majority of Linux-based workloads. Today EFS serves 100,000’s of customers in all AWS regions.] We built EFS to be cloud-scale (Elastic), simple (set and forget), cost-effective, performant. On April 1 we announced General Purpose mode file systems support up to 35,000 read operations per second, a 400% increase from the previous limit of 7,000. Maximum write operations are unchanged at 7,000 per second. Link: https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-elastic-file-system-announces-increase-in-read-operations-for-general-purpose-file-systems/ General Purpose mode (GP mode) is the default performance mode for Amazon EFS. It offers the lowest per-operation latency and is the recommended choice for most applications. Amazon EFS also offers the Max I/O performance mode which can scale to higher levels of aggregate throughput and supports over 500,000 operations per second with slightly higher metadata latencies than GP mode.
13. Amazon EFS IA: Storage class for infrequently accessed files for $0.025/GB per month* - assuming 80% of your files are colder for use with Amazon EFS IA, and 20% using EFS Standard, that equates to a BLENDED, or effective price of $0.08/GB-month!! EFS IA -- 92% savings compared to EFS Standard
14. OK, so let’s talk a little bit about how EFS IA works at a high level. First, you create a file system. EFS file systems transparently serve data from both storage classes, providing a common file system namespace, so you don’t have to worry about which of your files are actively used and which are infrequently accessed. Next, you enable EFS Lifecycle Management for your file system. With EFS Lifecycle Management enabled, any files that aren’t accessed per the lifecycle policy (14, 30, 60 and 90 days) are automatically moved into the Infrequent Access storage class. So in having both these capabilities, the IA storage class and Lifecycle Management, we’re eliminating the need to manually manage your data to optimize for cost.
15. Today you can use IAM identity policies to control what EFS administrative APIs a user has access to. IAM Authorization for NFS Clients will allow customers to manage NFS client access and permissions for an EFS file system using AWS IAM. It adds an additional layer of security on top of EFS’s current network-based access controls and provides often-requested security features such as root squashing, read-only access, and the ability to enforce TLS. With IAM authorization for NFS clients, we are adding three actions that allow you to control NFS client access to your file systems, adding ClientMount (permission to mount as read-only), ClientWrite (permission to read and write), and ClientRootAccess (permission to read and write as the root user). For those times you need more granular policies we are adding file system resource policies. These policies make it easy to specify which exact users have read, write, or root access to a particular file system.
16. On the one hand, today’s SSD devices provide more IOPS per dollar, more throughput per gigabyte, and lower latency than today’s HDD devices. On the other hand, continued density improvements in HDD technology drive the cost per gigabyte down, but also reduce the effective throughput per gigabyte. And with HDD, we offer the LOWEST COST WINDOWS FILE STORAGE IN THE CLOUD. And offer other ways th e
17. We also two weeks ago announced data deduplication, removes portions of files that are redundant across your data set, typical savings for general file shares is 50-60%.
18. Call out with HDD $0.025 for MAZ = lowest cost Windows File Storage in the cloud (even before data dedup )
19. Works at the sub-file level Uses post-processing optimization to minimize performance impact Removes duplicated content and compresses common content Use remote management PowerShell CLI on your file system to… Enable/disable Data Deduplication Customize schedule for deduplication jobs Monitor how much savings you’re achieving with deduplication
20. Let’s take a deeper look into two key use cases for Amazon EFS- container storage and scale out workloads and some of the latest features
21. Stateful containers store data in shared storage Scale container instances without reconfiguring the file system No need to monitor or provision storage as your persistent storage grows Supports rapid failover and event based provisioning Proximity to other container and dev/ops services
22. For customers running analytics, EFS can be a great option. EFS is a POSIX compliant file system, so applications that can access data over NFS can utilize EFS with no code modification required. For example, if you are running analytics against an on premises or built in the cloud NFS file store, your application is seamlessly portable to EFS. Additionally, there are many data sources feeding analytics applications that understand how to write to a file system. This may be lab equipment in the healthcare and life sciences world, or machine data in manufacturing. For these applications, having a common interface where data can be written, transformed if needed, then analyzed provides ease of use and flexibility. Additionally, since EFS provides a scalable, decoupled datastore accessible from thousands of EC2 instances simultaneously, processing analytics jobs can be done quickly and efficiently providing businesses with faster time to insights from their data.
23. So FSX Windows file server server supports maximum up-to 64 TB filesystem size. However you can scale out the throughput by using Microsoft's Distributed File System (DFS) Namespaces. You can use DFS Namespaces to group file shares on multiple file systems into one common folder structure (a namespace) that you use to access the entire file dataset. DFS Namespaces can help you to organize and unify access to your file shares across multiple file systems. Also, DFS Namespaces help to scale file data storage beyond what each file system supports (64 TB) for large file datasets—up to hundreds of petabytes.you can scale performance up to tens of gigabytes per second of throughput, with millions of IOPS, across hundreds of petabytes of Windows-based file data.
24. With Multi-AZ file systems, we now support a type of SMB share called a CA share. With CA shares…
25. AWS DataSync is a data transfer service that makes it easy for you to automate moving data between on-premises storage and Amazon S3, Amazon EFS, or Amazon FSx for Windows File Server. DataSync automatically handles many of the tasks related to data transfers that can slow down migrations or burden your IT operations, including running your own instances, handling encryption, managing scripts, network optimization, and data integrity validation. Fast You can use DataSync to transfer data at speeds up to 10 times faster than open source tools. Automated DataSync automates both the management of data transfer processes and the infrastructure required for high-performance, secure data transfer. The service also includes automatic encryption and data validation. All of this minimizes the in-house development and management otherwise needed for fast, reliable, and secure transfers. One-time migrations or on-going transfers The service enables one-time data migrations, recurring data processing workflows, and automated replication for data protection and recovery.  Flexible DataSync supports flexible configurations to suit your specific needs, including bandwidth throttling, copying source permissions and metadata. AWS integrated Getting started with DataSync is easy: Deploy the DataSync agent on-premises, connect it to a file system or storage array, select Amazon EFS, Amazon S3, or Amazon FSx for Windows File Server as your AWS storage, and start moving data. Secure & compliant All of your data is encrypted in transit and at rest, and for each transfer, the service performs integrity checks. These checks ensure that the data written to your destination matches the data read from your source, validating consistency. DataSync also makes sure that your data arrives securely, intact, and ready to use.