In this session, we fill you in about Amazon EFS, including an overview of this recently introduced service, its use cases, and best practices for working with it.
(STG306) EFS: How to store 8 Exabytes & look good doing itAmazon Web Services
In this session we will review the world’s first cloud-scale network attached file system and its targeted use cases. Session attendees will learn about EFS’s benefits, how to identify applications that are appropriate for use with EFS, and details about its performance and security models. The target audience is file system administrators, application developers, and application owners that operate or build file-based applications.
In this session we will review Amazon EFS and how it delivers fully managed, petabyte-scale file storage for Amazon EC2 instances. Large scale and consistent performance make Amazon EFS ideal for web and content serving, enterprise applications, media processing, container storage, and Big Data analytics use cases. Session attendees will learn how to identify appropriate applications for use with Amazon EFS, understand performance details and security models, and hear how established customers are using it in production. The target audience is file system administrators, application developers, and application owners that operate or build file-based applications that require consistent latencies at cloud scale.
AWS re:Invent 2016: Case Study: How Spokeo Improved Web Application Response ...Amazon Web Services
At Spokeo, we are running a fast, big data, and high-traffic website providing people search services. But at our scale, we started to reach limitations to how fast our conventional web stack could do things and concluded that a Ruby on Rails–only solution simply couldn't keep up. In this session, we cover some of the options we had to solve this problem and why we chose Amazon Elastic File System (Amazon EFS) as a central part of our solution with metrics and benchmarking. Using EFS, we were able to take response times down from 250 ms to below 70 ms. We look into the architecture of the solution and lessons we learned along the way. In the end, we find that faster response times are just the beginning of the benefits that we see.
AWS re:Invent 2016: Taking DevOps to the AWS Edge (CTD302)Amazon Web Services
In this session, we dive deep into how you can integrate Amazon CloudFront and related services into your application, be agile in developing and adapting the application, and follow best practices when configuring the services to improve security and performance, all while reducing costs. Attend this session and learn how to avoid needless forwarding of headers and cookies, test your application when making changes to the origin, version your configuration changes, monitor usage and automate security, create templates for new distributions, configure SSL/TLS certificates, and more.
AWS re:Invent 2016: Proactive Security Testing in AWS: From Early Implementat...Amazon Web Services
Attend this session to learn about security testing your applications in AWS. Effective security testing is challenging, but multiple features and services within AWS make security testing easier. This session covers common approaches to testing, including how we think about testing within AWS, how to apply AWS services to your test setup, remediating findings, and automation.
Deploying a Disaster Recovery Site on AWS: Minimal Cost with Maximum EfficiencyAmazon Web Services
In the event of a disaster, you need to be able to recover lost data quickly to ensure business continuity. For critical applications, keeping your time to recover and data loss to a minimum as well as optimizing your overall capital expense can be challenging. This session presents AWS features and services along with Disaster Recovery architectures that you can leverage when building highly available and disaster resilient applications. We will provide recommendations on how to improve your Disaster Recovery plan and discuss example scenarios showing how to recover from a disaster.
Simple, Scalable and Highly Durable NAS in the Cloud – Amazon EFSAmazon Web Services
Come learn how to get started with a scalable file system with a simple interface for use with Amazon EC2 instances in the AWS Cloud. We’ll cover the basics and go through customer use cases to illustrate key features. We will walk through active demos that show you how EFS supports application workflows.
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...Amazon Web Services
Learning Objectives:
• Learn how to use CloudFront dynamic delivery features • See a live demo and learn how to take advantage of Cloud Front newest features
Traditionally, content delivery networks (CDNs) were designed to accelerate static content. Amazon CloudFront supports delivery of an entire website, including dynamic, static, streaming and interactive content using a global network of edge locations. CloudFront integrates with other AWS services that are built to scale massively. Together, the solution can automatically scale to millions of users by leveraging the global reach of CloudFront and the auto scaling capability of AWS platform. In this talk, we introduce you to various design patterns and best practices to build a massively scalable solution using CloudFront. We discuss how this scale can be achieved without compromising on availability, security or cost.
Review of how AWS EC2 storage options have evolved, and making the right selection for your workload. Covering Amazon Elastic Block Storage, EBS and Amazon Elastic File System, EFS.
AWS re:Invent 2016: Amazon Aurora Best Practices: Getting the Best Out of You...Amazon Web Services
Amazon Aurora is a fully managed relational database engine that provides higher performance, availability and durability than previously possible using conventional monolithic database architectures. After launching a year ago, we continued adding many new features and capabilities to Aurora. In this session AWS Aurora experts will discuss the best practices that will help you put these capabilities to the best use. You will also hear from Amazon Aurora customer Intercom on the best practices they adopted for moving live databases with over two billion rows to a new datastore in Amazon Aurora with almost no downtime or lost records.
Intercom was founded to provide a fundamentally new way for Internet businesses to communicate with customers at scale. For growing startups like Intercom, it’s natural for the load on datastores to grow on a weekly basis. The usual solution to this problem is to get a bigger box from AWS. But very soon you reach a point where bigger boat is not an option anymore. You will learn about the benefits of moving to such a datastore, the problems it introduced, and all about the new ability for scaling that was not there before.
AWS re:Invent 2016: Netflix: Container Scheduling, Execution, and Integration...Amazon Web Services
Customers from over all over the world streamed forty-two billion hours of Netflix content last year. Various Netflix batch jobs and an increasing number of service applications use containers for their processing. In this session, Netflix presents a deep dive on the motivations and the technology powering container deployment on top of Amazon Web Services. The session covers our approach to resource management and scheduling with the open source Fenzo library, along with details of how we integrate Docker and Netflix container scheduling running on AWS. We cover the approach we have taken to deliver AWS platform features to containers such as IAM roles, VPCs, security groups, metadata proxies, and user data. We want to take advantage of native AWS container resource management using Amazon ECS to reduce operational responsibilities. We are delivering these integrations in collaboration with the Amazon ECS engineering team. The session also shares some of the results so far, and lessons learned throughout our implementation and operations.
AWS re:Invent 2016: How Netflix Achieves Email Delivery at Global Scale with ...Amazon Web Services
Companies around the world are using Amazon Simple Email Service (Amazon SES) to send millions of emails to their customers every day, and scaling linearly, at cost. In this session, you learn how to use the scalable and reliable infrastructure of Amazon SES. In addition, Netflix talks about their advanced Messaging program, their challenges, how SES helped them with their goals, and how they architected their solution for global scale and deliverability.
Amazon provides several cloud storage options on AWS including S3 for object storage, EFS for file storage accessible by multiple EC2 instances, and EBS for block storage attached to a single EC2 instance. S3 is highly durable, available globally at low cost but doesn't provide file system semantics. EFS delivers file system access across instances but has higher latency and cost than EBS, which provides low latency storage for a single instance via block-level access. Glacier is very low cost archival storage for infrequently accessed data.
Amazon Elastic File System (Amazon EFS) Introduction & DemoAmazon Web Services
Amazon Elastic File System (Amazon EFS) is a file storage service for Amazon Elastic Compute Cloud (Amazon EC2) instances. Amazon EFS is easy to use and provides a simple interface that allows you to create and configure file systems quickly and easily. With Amazon EFS, storage capacity is elastic, growing and shrinking automatically as you add and remove files, so your applications have the storage they need, when they need it.
AWS re:Invent 2016: Case Study: How Videology and Zendesk Modernized Their Bi...Amazon Web Services
The companies Videology and Zendesk both had the same problem—how to rearchitect their big data processing platforms to scale to meet growing demand, while at the same time improving performance, availability, and cost structure? Videology provides a converged advertising solution that is screen-agnostic, ensuring unduplicated reach with the right frequency cadence to achieve guaranteed results. To achieve this, their big data platform ingests, processes, and analyzes a variety of logs. In this talk, Videology discusses how they migrated our platform to use Cloudera on a mix of m4 and r3 instances using the Amazon EBS Streaming Optimized HDD (st1) volume type. Zendesk provides a cloud-based customer support platform that allows quicker and easier interaction between businesses and customers. To deliver this experience, Zendesk runs a large Elasticsearch, Logstash, Kibana stack. The talk also discusses how Zendesk rearchitected their deployment to use m4s and also leverage the EBS Streaming Optimized HDD (st1) volume type. Tips for success will be shared throughout.
Get the Most Out of Amazon EC2: A Deep Dive on Reserved, On-Demand, and Spot ...Amazon Web Services
With Amazon EC2, you have the flexibility to mix-and-match purchasing models to suit your business needs. By combining pay-as-you-go (On-Demand), reserve ahead of time for discounts (Reserved), and high-discount spare capacity (Spot) purchasing models, you can optimize cost, grow your compute capacity and throughput, and enable new types of cloud computing applications. This presentation will guide you on how to achieve high performance and availability at the lowest TCO. We will explore how to best combine EC2's purchasing models across several common applications with immediately actionable takeaways.
Amazon Aurora for the Enterprise - August 2016 Monthly Webinar SeriesAmazon Web Services
Relational databases are a cornerstone of the enterprise IT landscape, powering business-critical applications of many kinds. Though they have been around for a while, current commercial relational databases have lagged behind in innovation. Amazon Aurora, a managed database service built for the cloud, is intended to change that. It fulfils the high-performance, high-availability needs of business-critical applications with an emphasis on cost-effectiveness. In this session, we will look into how Aurora fits the needs of applications built and bought by enterprises to power their business.
Learning Objectives:
• Explore the overall architecture, capabilities, and cost-effectiveness of Aurora and see how it compares to commercial database offerings
• Learn best practices for enterprises adopting Aurora for existing and new workloads, as well as strategies, tools, and techniques for migrating existing databases to Aurora
This presentation talks about how you can optimize your Application Architecture on AWS Cloud and create a Fault Tolerant Architecture that will have Zero Down Time! The best practices for a fault tolerant Web Applicaiton.
Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances for fault tolerance and load distribution. In this session, we go into detail about Elastic Load Balancing's configuration and day-to-day management, as well as its use in conjunction with Auto Scaling. We explain how to make decisions about the service and share best practices and useful tips for success.
SRV401 Deep Dive on Amazon Elastic File System (Amazon EFS)Amazon Web Services
In this session we will review Amazon EFS and how it delivers fully managed, petabyte-scale file storage for Amazon EC2 instances. Large scale and consistent performance make Amazon EFS ideal for web and content serving, enterprise applications, media processing, container storage, and Big Data analytics use cases. Session attendees will learn how to identify appropriate applications for use with Amazon EFS, understand performance details and security models, and hear how established customers are using it in production. The target audience is file system administrators, application developers, and application owners that operate or build file-based applications that require consistent latencies at cloud scale.
Hands-on Labs: Getting Started with AWS - March 2017 AWS Online Tech TalksAmazon Web Services
The document provides information about a webinar on getting started with AWS, including deploying a static website. It outlines the agenda which includes: watching a 15 minute presentation on AWS; watching a 25 minute demo of deploying a static website; and having 45-60 minutes to complete the demo independently. It then details the various sections of the webinar which cover creating an AWS account, enabling security features, using S3 buckets to host the website, configuring permissions, associating a domain name, and using CloudFront for acceleration.
Best Practices for Managing Security Operations in AWS - March 2017 AWS Onlin...Amazon Web Services
To help prevent unexpected access to your AWS resources, it is critical to maintain strong identity and access policies. It is equally important to track and alert on changes to your AWS resources. In this tech talk, you will learn how to use AWS Identity and Access Management (IAM) to control access to your AWS resources and integrate your existing authentication system with AWS IAM. We will cover how you can deploy and control your AWS infrastructure using code templates, including change management policies with AWS CloudFormation. In addition, we will explore different options for managing both your AWS access logs and your Amazon Elastic Compute Cloud (EC2) system logs using Amazon CloudWatch Logs. We also will cover how to use these logs to implement an audit and compliance validation process using services such as AWS Config, AWS CloudTrail, and Amazon Inspector.
Learning Objectives:
• Understand the AWS Shared Responsibility Model.
• Understand AWS account and identity management options and configuration.
• Learn the concept of infrastructure as code and change management using AWS CloudFormation.
• Learn how to audit and log your AWS service usage.
• Learn about AWS services to add automatic compliance checks to your AWS infrastructure.
With AWS, you can choose the right storage service like including Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Block Storage (Amazon EBS) for the right use case. This session shows the range of AWS choices—from object storage to block storage—that are available to you. The sessions will also include specifics about real-world deployments from customers who are using Amazon S3, Amazon EBS, Amazon Glacier, and AWS Storage Gateway.
Reasons to attend:
Learn how to select which storage options to use, based your requirements for cost, access pattern and use case.
Understand why AWS is a perfect platform for the storage of digital assets, data, media and backups.
Discover how Glacier can revolutionize your long term archive management by removing the need for costly and fragile media types.
Hear about customer use cases and a rich partner ecosystem of services built on AWS storage services.
Not just for archiving or compliance use cases, Amazon Glacier accommodates customers simply looking to replace their on-premises long term storage with a cost efficient, durable, cloud option, from which they can easily and quickly access their data when they need to. This session will introduce newly launched features for Amazon Glacier, review the current service feature set, and share the global data center shut down and storage strategy for Sony DADC New Media Solutions (NMS). NMS is Sony’s digital servicing division providing global digital distribution, linear playout and white label OTT/Commerce solutions for clients such as BBC Worldwide, NBCUniversal, Sony Playstation, and Funimation Entertainment.
Hear from Andy Shenkler, NMS’s Chief Technology and Solutions Officer as he talks about the key factors that drove the organization’s decision to move away from tape and go towards the cloud and out of the infrastructure business overall. Learn more about the impact and operational practices inside a world class digital supply chain as they were able to move over 20 petabytes of data, over 1M hours of video, to the cloud and never looked back.
AWS re:Invent 2016: Migrating Your Data Warehouse to Amazon Redshift (DAT202)Amazon Web Services
Amazon Redshift is a fast, simple, cost-effective data warehousing solution, and in this session, we look at the tools and techniques you can use to migrate your existing data warehouse to Amazon Redshift. We will then present a case study on Scholastic’s migration to Amazon Redshift. Scholastic, a large 100-year-old publishing company, was running their business with older, on-premise, data warehousing and analytics solutions, which could not keep up with business needs and were expensive. Scholastic also needed to include new capabilities like streaming data and real time analytics. Scholastic migrated to Amazon Redshift, and achieved agility and faster time to insight while dramatically reducing costs. In this session, Scholastic will discuss how they achieved this, including options considered, technical architecture implemented, results, and lessons learned.
AWS re:Invent 2016: ElastiCache Deep Dive: Best Practices and Usage Patterns ...Amazon Web Services
In this session, we provide a peek behind the scenes to learn about Amazon ElastiCache's design and architecture. See common design patterns with our Redis and Memcached offerings and how customers have used them for in-memory operations to reduce latency and improve application throughput. During this session, we review ElastiCache best practices, design patterns, and anti-patterns.
This document provides an overview of Amazon Web Services storage options, including scalable object storage with Amazon S3, inexpensive archive storage with Amazon Glacier, persistent block storage with Amazon EBS, and a shared file system with Amazon EFS. It discusses the growth of data production across industries and how AWS storage services provide scalable, cost-effective solutions. Key features and use cases are described for each storage service.
(STG312) Amazon Glacier Deep Dive: Cold Data Storage in AWSAmazon Web Services
This session explores some of the key features of Amazon Glacier, including security, durability, and configuration for storing compliance and regulatory data. It covers best practices for managing your cold data, including ingest, retrieval, and security controls. Other topics include: how to optimize storage, upload, and retrieval costs; how to identify the most applicable workloads; and recommended optimizations based on a few sample use cases from a number of industry verticals.
The document provides an introduction to Amazon DynamoDB, a fully managed NoSQL database service. It discusses how DynamoDB provides fast and consistent performance at scale without the need to provision or manage infrastructure. It also demonstrates how to build a serverless web application using DynamoDB along with AWS Lambda and API Gateway.
AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...Amazon Web Services
by Everett Dolgner, Business Development Manager, AWS
AWS offers a suite of tools to help you surmount limitations associated to data migration from on premise to the cloud. Attend this session to learn about moving data by using networks, roads, and AWS technology partners. We will also discuss how to move data into and out of the Cloud in batches, increments, and streams.
Accelerating Application Performance with Amazon ElastiCache (DAT207) | AWS r...Amazon Web Services
Learn how you can use Amazon ElastiCache to easily deploy a Memcached or Redis compatible, in-memory caching system to speed up your application performance. We show you how to use Amazon ElastiCache to improve your application latency and reduce the load on your database servers. We'll also show you how to build a caching layer that is easy to manage and scale as your application grows. During this session, we go over various scenarios and use cases that can benefit by enabling caching, and discuss the features provided by Amazon ElastiCache.
(STG202) AWS Import/Export Snowball: Large-Scale Data Ingest into AWSAmazon Web Services
Moving terabyte and petabyte volumes of data into the cloud can be a challenge for many businesses. Come learn how you can use Snowball, a new AWS feature, to move large-scale (terabyte and petabyte) data to AWS storage services.
AWS re:Invent 2016: Deep Dive on Amazon DynamoDB (DAT304)Amazon Web Services
Explore Amazon DynamoDB capabilities and benefits in detail and learn how to get the most out of your DynamoDB database. We go over best practices for schema design with DynamoDB across multiple use cases, including gaming, AdTech, IoT, and others. We explore designing efficient indexes, scanning, and querying, and go into detail on a number of recently released features, including JSON document support, DynamoDB Streams, and more. We also provide lessons learned from operating DynamoDB at scale, including provisioning DynamoDB for IoT.
AWS Webcast - Archiving in the Cloud - Best Practices for Amazon GlacierAmazon Web Services
Join our webinar to learn more about how to build a cost effective archive application using Amazon Glacier, an extremely low cost, secure, highly durable, and easy to use storage service in the AWS cloud.
We will explain how Amazon Glacier works and walk through some best practices to get the most out of the service
We will also highlight how to choose between Amazon Glacier and Amazon S3’s Glacier storage option.
Learn more: http://aws.amazon.com/glacier/
Migrate your Data Warehouse to Amazon Redshift - September Webinar SeriesAmazon Web Services
- TrueCar migrated their data warehouse from an on-premises Hadoop cluster to Amazon Redshift. They load clickstream, transactions, inventory, and lead data into Redshift for analytics and reporting.
- They use ETL tools like Talend and Hive to process data and load it into HDFS and S3, then load it into Redshift using a custom utility. The data is organized into schemas separating raw, user, and reporting data.
- Best practices for Redshift include designing tables for compression, sort keys, and distribution, managing cluster size and workloads over time, and vacuuming and analyzing tables regularly. TrueCar's migration to Redshift improved performance and reduced costs.
AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...Amazon Web Services
Moving files over long distances, or terabyte and petabyte volumes of data into the cloud can be a sticking point for cloud migrations. Come learn how customers are using the latest Snowball to move large scale data stores to the cloud, while remaining compliant with existing regulations.
Amazon Redshift is a fully managed data warehouse service that makes it fast, simple and cost effective to analyze data using SQL and existing business intelligence tools. The document provides an overview of Amazon Redshift and its benefits including speed, low cost, security, scalability and ease of use. It also provides examples of how various companies use Redshift for big data analytics including analyzing social media firehoses, mobile usage and real-time IoT streaming data.
This document provides an overview and agenda for a presentation on Amazon DynamoDB. It discusses key concepts like tables, data types, partitioning, indexing and scaling in DynamoDB. It also provides best practices and examples for modeling different data scenarios like event logging, product catalogs, messaging apps and multiplayer games.
An overview of the Amazon ElastiCache managed service, with examples of how it can be used to increase performance, lower costs and augment other database services and databases to make things faster, easier and less expensive.
In this session we will review Amazon EFS and how it delivers fully managed, petabyte-scale file storage for Amazon EC2 instances. Large scale and consistent performance make Amazon EFS ideal for web and content serving, enterprise applications, media processing, container storage, and Big Data analytics use cases. Session attendees will learn how to identify appropriate applications for use with Amazon EFS, understand performance details and security models, and hear how established customers are using it in production. The target audience is file system administrators, application developers, and application owners that operate or build file-based applications that require consistent latencies at cloud scale. Learn More: https://aws.amazon.com/government-education/
Deep Dive on Amazon Elastic File System - June 2017 AWS Online Tech TalksAmazon Web Services
Learning Objectives:
- Recognize why and when to use Amazon EFS and the economic benefits versus other solutions
- Understand key technical, performance, and security concepts
- See Amazon EFS in action with live demo
The vast majority of applications and workloads interact with data storage via a file system interface and require file system semantics. As businesses move to the cloud they require storage resources that integrates with their existing applications and tools. In this technical session, we will explore file storage with Amazon Elastic File System (Amazon EFS) and its targeted use cases. Attendees will learn about the Amazon EFS features and benefits, how to identify applications that are appropriate for use with Amazon EFS, and details about its performance and security models. We will highlight and demonstrate how to deploy Amazon EFS in our most common use cases and will share tips for success throughout.
Deep Dive on Elastic File System - February 2017 AWS Online Tech TalksAmazon Web Services
Organizations face significant challenges moving their applications to the cloud when they require a standard file system interface for accessing their cloud data. In this technical session, we will explore the world’s first cloud-scale file system and its targeted use cases. Attendees will learn about the Amazon Elastic File System (EFS) features and benefits, how to identify applications that are appropriate for use with Amazon EFS, and details about its performance and security models. We will highlight and demonstrate how to deploy Amazon EFS in one of our most common use cases and will share tips for success throughout.
Learning Objectives:
• Recognize why and when to use Amazon EFS
• Understand key technical/security concepts
• Learn how to leverage EFS’s performance
• See a demo of EFS in action
• Review EFS’s economics
Amazon Elastic File System (Amazon EFS) is a new fully managed service that makes it easy to set up and scale shared file storage in the AWS cloud. Amazon EFS supports NFS v4 and is designed to be highly available and durable. Amazon EFS can support thousands of concurrent Amazon EC2 client connections with consistent performance, making it ideal for a wide range of use cases, including content repositories, development environments, and home directories, as well as big data applications that require on-demand scaling of file system capacity and performance. In this session, we'll provide an overview of EFS, walk through the creation of a file system, discuss EFS availability and durability properties, and review security features.
Amazon Elastic File System (Amazon EFS)Danilo Poccia
Amazon Elastic File System (Amazon EFS) provides simple, scalable file storage for use with AWS services like EC2. It allows sharing of file systems across multiple EC2 instances and provides simple integration with existing tools. File systems can grow and shrink automatically as storage needs change. EFS uses NFS v4 for access and stores data redundantly across Availability Zones for high availability and durability. Security features include controlling access via security groups, user-level permissions, and IAM policies. Pricing is $0.30 per GB per month with no minimum commitments.
AWS re:Invent 2016: Case Study: How Monsanto Uses Amazon EFS with Their Large...Amazon Web Services
This document discusses how Monsanto uses Amazon EFS for large scale geospatial data sets. It provides an overview of EFS and its key features. It then details how Monsanto moved its geospatial data and analytics to the cloud using EFS, including setting up a GeoServer cluster on EFS. It also discusses how Monsanto built a collaborative analytics platform and production environmental classification engine that run analytics at scale on EFS and EMR. The document concludes with recommendations when using EFS and takeaways.
This document provides an overview and summary of Amazon Web Services (AWS) file storage options, including Amazon Elastic File System (Amazon EFS) and Amazon FSx for Windows File Server. It discusses the key features and use cases of each service. It also provides guidance on choosing the right file storage solution based on an application's needs and examples of how customers can use AWS file storage for specific workloads.
This document provides an overview and summary of Amazon Web Services (AWS) file storage options, including Amazon Elastic File System (Amazon EFS) and Amazon FSx for Windows File Server. It discusses the key features and use cases of each service. It also provides guidance on choosing the right file storage solution based on an application's needs and examples of how customers can use AWS file storage for specific workloads.
Simple, Scalable and Highly Durable NAS in the Cloud - Amazon EFSAmazon Web Services
This document discusses Amazon Elastic File System (EFS), a file storage service in AWS. EFS provides simple, scalable, and durable file storage in the cloud using the NFS protocol. It allows files to be accessed from EC2 instances and on-premises networks. EFS automatically scales capacity up and down as files are added and removed, and is designed to burst throughput to high levels. The document provides information on EFS architecture, performance modes, resources, security, and use cases. It also summarizes hands-on exercises for creating and mounting EFS file systems.
Learn how to get started with a scalable file system with a simple interface for use with Amazon EC2 instances in the AWS Cloud. We’ll cover the basics, go through customer use cases to illustrate key features, and run active demos that show you how EFS supports application workflows.
by Darryl Osborne, Solutions Architect, AWS
Learn how to get started with a scalable file system with a simple interface for use with Amazon EC2 instances in the AWS Cloud. We’ll cover the basics, go through customer use cases to illustrate key features, and run active demos that show you how EFS supports application workflows.
AWS re:Invent 2016: Case Study: How Atlassian Uses Amazon EFS with JIRA to Cu...Amazon Web Services
Atlassian used Amazon EFS with JIRA to cut costs and accelerate performance. Testing showed Amazon EFS provided consistent throughput and capacity as usage increased. Atlassian now uses EFS for their production JIRA Data Center deployment, which benefits from EFS's automatic scaling, high availability, and lower costs compared to on-premises shared storage.
Si stima che i clienti abbiamo in totale 256 EB di file shares in locale. La gestione di questi file systems è onerosa e comporta problematiche sia di budget (CAPEX) che di operation (gestione, scalabilità, data protection). Tipicamente gli apparati NAS locali devono essere sostituiti ogni 3-5 anni, obbligando i clienti a fare un capacity planning pluriennale e richiedendo un progetto a sè stante per la migrazione dati.
Il passaggio al cloud di AWS consente ai clienti di pagare esattamente la quantità di spazio di archiviazione di file di cui hanno bisogno ora, senza costi o vincoli iniziali e ridimensionare la capacità necessaria durante la crescita dei dati senza dover stimare in anticipo di quanto avranno bisogno. Sfruttando soluzioni di file completamente gestite come Amazon FSx per Windows File Server, FSx Backup, i clienti non devono più preoccuparsi del sovraccarico amministrativo di impostazione, protezione, manutenzione e backup della propria infrastruttura di file.
La recente apertura della regione italiana MXP apre a nuovi scenari di hybrid cloud per la parte filesystem/SMB share.
Introduction to AWS EFS and FSx PresentationKnoldus Inc.
Amazon EFS offers scalable storage, reaching petabytes of data without disrupting apps. The service works with thousands of Amazon EC2 instances with consistently low latency. Amazon FSx provides a comprehensive managed file storage service tailored to the needs of AWS customers. Based on the Windows Server foundation, FSx offers compatibility with a wide range of Windows Server platforms, ensuring seamless integration and performance.
Integrating On-premises Enterprise Storage Workloads with AWS (ENT301) | AWS ...Amazon Web Services
AWS gives designers of enterprise storage systems a completely new set of options. Aimed at enterprise storage specialists and managers of cloud-integration teams, this session gives you the tools and perspective to confidently integrate your storage workloads with AWS. We show working use cases, a thorough TCO model, and detailed customer blueprints. Throughout we analyze how data-tiering options measure up to the design criteria that matter most: performance, efficiency, cost, security, and integration.
With AWS, you can choose the right storage service for the right use case. Given the myriad of choices, from object storage to block storage, this session will profile details and examples of some of the choices available to you, with details on real world deployments from customers who are using Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), Amazon Glacier, and AWS Storage Gateway.
The Google File System (GFS) was designed by Google to store massive amounts of data across cheap, unreliable hardware. It uses a single master to coordinate metadata and mutations across multiple chunkservers. Files are divided into fixed-size chunks which are replicated for reliability. The design focuses on supporting huge files that are written once and read through streaming, while tolerating high failure rates through replication and relaxed consistency. GFS has proven successful at meeting Google's storage needs at massive scale.
The document provides an overview of AWS storage services including block storage, shared file systems, and object storage. It begins with an introduction to why AWS is chosen for storage and lists the global AWS infrastructure. It then covers block storage options like Amazon EBS, file storage with Amazon EFS, and object storage with Amazon S3. Specific features of each service are described like durability, availability and pricing. Example use cases are provided for each storage type.
Introduction to Storage on AWS - AWS Summit Cape Town 2017Amazon Web Services
With AWS, you can choose the right storage service for the right use case. This session shows the range of AWS choices that are available to you: Amazon S3, Amazon EBS, Amazon EFS, Amazon Glacier and Cloud Data Migration solutions.
Similar to AWS re:Invent 2016: Deep Dive on Amazon Elastic File System (STG202) (20)
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS.
In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup.
Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti.
Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.
Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi
Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
L’utilizzo dei container è in continua crescita.
Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili.
I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!
In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th.
Event Agenda :
Open banking so far (short recap)
• PSD2, OB UK, OB Australia, OB LATAM, OB Israel
Intro to Open Finance marketplace
• Scope
• Features
• Tech overview and Demo
The role of the Cloud
The Future of APIs
• Complying with regulation
• Monetizing data / APIs
• Business models
• Time to market
One platform for all: a Strategic approach
Q&A
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc.
AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta.
Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity.
AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet.
Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.
Dal riconoscimento facciale al riconoscimento di frodi o difetti di fabbricazione, l'analisi di immagini e video che sfruttano tecniche di intelligenza artificiale, si stanno evolvendo e raffinando a ritmi elevati. In questo webinar esploreremo le possibilità messe a disposizione dai servizi AWS per applicare lo stato dell'arte delle tecniche di computer vision a scenari reali.
Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti.
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti.
Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire.
Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito.
In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.
Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline.
Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.
1) The document discusses building a minimum viable product (MVP) using Amazon Web Services (AWS).
2) It provides an example of an MVP for an omni-channel messenger platform that was built from 2017 to connect ecommerce stores to customers via web chat, Facebook Messenger, WhatsApp, and other channels.
3) The founder discusses how they started with an MVP in 2017 with 200 ecommerce stores in Hong Kong and Taiwan, and have since expanded to over 5000 clients across Southeast Asia using AWS for scaling.
This document discusses pitch decks and fundraising materials. It explains that venture capitalists will typically spend only 3 minutes and 44 seconds reviewing a pitch deck. Therefore, the deck needs to tell a compelling story to grab their attention. It also provides tips on tailoring different types of decks for different purposes, such as creating a concise 1-2 page teaser, a presentation deck for pitching in-person, and a more detailed read-only or fundraising deck. The document stresses the importance of including key information like the problem, solution, product, traction, market size, plans, team, and ask.
This document discusses building serverless web applications using AWS services like API Gateway, Lambda, DynamoDB, S3 and Amplify. It provides an overview of each service and how they can work together to create a scalable, secure and cost-effective serverless application stack without having to manage servers or infrastructure. Key services covered include API Gateway for hosting APIs, Lambda for backend logic, DynamoDB for database needs, S3 for static content, and Amplify for frontend hosting and continuous deployment.
This document provides tips for fundraising from startup founders Roland Yau and Sze Lok Chan. It discusses generating competition to create urgency for investors, fundraising in parallel rather than sequentially, having a clear fundraising narrative focused on what you do and why it's compelling, and prioritizing relationships with people over firms. It also notes how the pandemic has changed fundraising, with examples of deals done virtually during this time. The tips emphasize being fully prepared before fundraising and cultivating connections with investors in advance.
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
This document discusses Amazon's machine learning services for building conversational interfaces and extracting insights from unstructured text and audio. It describes Amazon Lex for creating chatbots, Amazon Comprehend for natural language processing tasks like entity extraction and sentiment analysis, and how they can be used together for applications like intelligent call centers and content analysis. Pre-trained APIs simplify adding machine learning to apps without requiring ML expertise.
Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.
How Social Media Hackers Help You to See Your Wife's Message.pdfHackersList
In the modern digital era, social media platforms have become integral to our daily lives. These platforms, including Facebook, Instagram, WhatsApp, and Snapchat, offer countless ways to connect, share, and communicate.
Performance Budgets for the Real World by Tammy EvertsScyllaDB
Performance budgets have been around for more than ten years. Over those years, we’ve learned a lot about what works, what doesn’t, and what we need to improve. In this session, Tammy revisits old assumptions about performance budgets and offers some new best practices. Topics include:
• Understanding performance budgets vs. performance goals
• Aligning budgets with user experience
• Pros and cons of Core Web Vitals
• How to stay on top of your budgets to fight regressions
Details of description part II: Describing images in practice - Tech Forum 2024BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and transcript: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
The DealBook is our annual overview of the Ukrainian tech investment industry. This edition comprehensively covers the full year 2023 and the first deals of 2024.
GDG Cloud Southlake #34: Neatsun Ziv: Automating AppsecJames Anderson
The lecture titled "Automating AppSec" delves into the critical challenges associated with manual application security (AppSec) processes and outlines strategic approaches for incorporating automation to enhance efficiency, accuracy, and scalability. The lecture is structured to highlight the inherent difficulties in traditional AppSec practices, emphasizing the labor-intensive triage of issues, the complexity of identifying responsible owners for security flaws, and the challenges of implementing security checks within CI/CD pipelines. Furthermore, it provides actionable insights on automating these processes to not only mitigate these pains but also to enable a more proactive and scalable security posture within development cycles.
The Pains of Manual AppSec:
This section will explore the time-consuming and error-prone nature of manually triaging security issues, including the difficulty of prioritizing vulnerabilities based on their actual risk to the organization. It will also discuss the challenges in determining ownership for remediation tasks, a process often complicated by cross-functional teams and microservices architectures. Additionally, the inefficiencies of manual checks within CI/CD gates will be examined, highlighting how they can delay deployments and introduce security risks.
Automating CI/CD Gates:
Here, the focus shifts to the automation of security within the CI/CD pipelines. The lecture will cover methods to seamlessly integrate security tools that automatically scan for vulnerabilities as part of the build process, thereby ensuring that security is a core component of the development lifecycle. Strategies for configuring automated gates that can block or flag builds based on the severity of detected issues will be discussed, ensuring that only secure code progresses through the pipeline.
Triaging Issues with Automation:
This segment addresses how automation can be leveraged to intelligently triage and prioritize security issues. It will cover technologies and methodologies for automatically assessing the context and potential impact of vulnerabilities, facilitating quicker and more accurate decision-making. The use of automated alerting and reporting mechanisms to ensure the right stakeholders are informed in a timely manner will also be discussed.
Identifying Ownership Automatically:
Automating the process of identifying who owns the responsibility for fixing specific security issues is critical for efficient remediation. This part of the lecture will explore tools and practices for mapping vulnerabilities to code owners, leveraging version control and project management tools.
Three Tips to Scale the Shift Left Program:
Finally, the lecture will offer three practical tips for organizations looking to scale their Shift Left security programs. These will include recommendations on fostering a security culture within development teams, employing DevSecOps principles to integrate security throughout the development
INDIAN AIR FORCE FIGHTER PLANES LIST.pdfjackson110191
These fighter aircraft have uses outside of traditional combat situations. They are essential in defending India's territorial integrity, averting dangers, and delivering aid to those in need during natural calamities. Additionally, the IAF improves its interoperability and fortifies international military alliances by working together and conducting joint exercises with other air forces.
Blockchain technology is transforming industries and reshaping the way we conduct business, manage data, and secure transactions. Whether you're new to blockchain or looking to deepen your knowledge, our guidebook, "Blockchain for Dummies", is your ultimate resource.
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsMydbops
This presentation, delivered at the Postgres Bangalore (PGBLR) Meetup-2 on June 29th, 2024, dives deep into connection pooling for PostgreSQL databases. Aakash M, a PostgreSQL Tech Lead at Mydbops, explores the challenges of managing numerous connections and explains how connection pooling optimizes performance and resource utilization.
Key Takeaways:
* Understand why connection pooling is essential for high-traffic applications
* Explore various connection poolers available for PostgreSQL, including pgbouncer
* Learn the configuration options and functionalities of pgbouncer
* Discover best practices for monitoring and troubleshooting connection pooling setups
* Gain insights into real-world use cases and considerations for production environments
This presentation is ideal for:
* Database administrators (DBAs)
* Developers working with PostgreSQL
* DevOps engineers
* Anyone interested in optimizing PostgreSQL performance
Contact info@mydbops.com for PostgreSQL Managed, Consulting and Remote DBA Services
2. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
3. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
4. Data Transfer
Direct
Connect
Snowball 3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
File
Amazon EFS
Block
Amazon EBS
(persistent)
Object
Amazon GlacierAmazon S3 Amazon EC2
Instance Store
(ephemeral)
How EFS fits in to the AWS storage platform
5. Data Transfer
Direct
Connect
Snowball 3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
File
Amazon EFS
Block
Amazon EBS
(persistent)
Object
Amazon GlacierAmazon S3 Amazon EC2
Instance Store
(ephemeral)
How EFS fits in to the AWS storage platform
6. Data Transfer
Direct
Connect
Snowball 3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
File
Amazon EFS
Block
Amazon EBS
(persistent)
Object
Amazon GlacierAmazon S3 Amazon EC2
Instance Store
(ephemeral)
How EFS fits in to the AWS storage platform
7. Data Transfer
Direct
Connect
Snowball 3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
File
Amazon EFS
Block
Amazon EBS
(persistent)
Object
Amazon GlacierAmazon S3 Amazon EC2
Instance Store
(ephemeral)
How EFS fits in to the AWS storage platform
8. We focused on changing the game
Simple Elastic Scalable
1 2 3
Highly durable
Highly available
9. Amazon EFS is simple
• Fully managed
- No hardware, network, file layer
- Create a scalable file system in seconds!
• Seamless integration with existing tools and apps
- NFS v4.1—widespread, open
- Standard file system access semantics
- Works with standard OS file system APIs
• Simple pricing = simple forecasting
1
10. Amazon EFS is elastic
• File systems grow and shrink automatically as
you add and remove files
• No need to provision storage capacity or
performance
• You pay only for the storage space you use,
with no minimum fee
2
11. • File systems can grow to petabytes of
capacity
• Throughput scales automatically as file
systems grow
• Consistent low latencies regardless of file
system size
• Support for thousands of concurrent NFS
connections
Amazon EFS is scalable
3
12. • Every file system object is redundantly
stored across multiple Availability Zones in a
Region
• Designed to sustain Availability Zone offline
conditions
• Superior to traditional NAS availability
models
• Appropriate for production/tier 0 applications
Highly durable and highly available (Multi-AZ)
13. In which Regions can I use EFS today?
• US West (Oregon)
• US East (N. Virginia)
• US East (Ohio)
• EU (Ireland)
More coming soon!
14. Do you need an EFS file system?
If you have an EC2 application or use case that requires a
file system AND
• Requires multi-attach OR
• Multi-AZ availability/durability OR
• GBs/s throughput OR
• Requires automatic scaling (grow/shrink) of storage
15. Operating your own multi-attach file storage on
the cloud is complex and expensive
Use an NFS
server or shared
file layer
Replicate EBS
volumes (1 per
EC2 instance)
Substantial management overhead (sync data, provision
and manage volumes)
Costly (one volume per instance)
Complex to set up and maintain
Scale challenges
HA challenges
Costly (compute + storage)
16. What customers are using EFS for today
Web serving Content management
Analytics
Media and Entertainment
workflows
Workflow management
Home directories
Container storage
Database backups
17. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
18. What is a file system?
• The primary resource in EFS
• Where you store files and directories
• Can create 125 file systems per account
19. What is a mount target?
• To access your file system within
a VPC, you create mount targets
in the VPC
• A mount target is an NFS endpoint
that lives in your VPC
• A mount target has an IP address
and a DNS name you use in your
mount command
• A mount target is highly available
AVAILABILITY ZONE 1
REGION
AVAILABILITY ZONE 2
AVAILABILITY ZONE 3
VPC
EC2
EC2
EC2
EC2
Mount
target
20. How to access a file system from an instance
• You “mount” a file system on an Amazon EC2 instance (standard
command) — the file system appears like a local set of directories
and files
• An NFS v4.1 client is standard on Linux distributions
mount –t nfs4 –o nfsvers=4.1
[file system DNS name]:/
/[user’s target directory]
21. How does it all fit together?
AVAILABILITY ZONE 1
REGION
AVAILABILITY ZONE 2
AVAILABILITY ZONE 3
VPC
EC2
EC2
EC2
EC2
File system
Data can be accessed from any AZ in the Region while maintaining full consistency
22. Several security mechanisms
Control network traffic to and from file systems (mount targets) by
using VPC security groups and network ACLs
Control file and directory access by using POSIX permissions
Control administrative access (API access) to file systems by
using AWS Identity and Access Management (IAM)
EFS supports action-level and resource-level permissions
23. The AWS Management Console, CLI, and SDK each allow
you to perform a variety of management tasks
Create a file system
Create and manage mount targets
Tag a file system
Delete a file system
View details on file systems in your AWS account
24. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
25. Amazon EFS is designed for wide spectrum of
performance needs
High throughput and parallel I/O
Low latency and serial I/O
Genomics
Big data analytics
Scale-out jobs
Home directories
Content management
Web serving
Metadata-intensive
jobs
26. Choose the performance mode best suited to
your workload
Mode What’s it for? Advantages Tradeoffs When to use
General
purpose
(default)
Latency-sensitive
applications and
general-purpose
workloads
Lowest latencies
for file operations
Limit of 7,000 ops/sec Best choice for most
workloads
Max I/O Large-scale and data-
heavy applications
Virtually unlimited
ability to scale out
throughput/IOPS
Slightly higher
latencies
Consider if 10s (or
more) instances
access your file
system concurrently
27. Use the PercentIOLimit CloudWatch metric to determine
if you’re constrained by General Purpose mode
28. Amazon EFS has a distributed data storage design
EC2
EC2
…
EC2
EC2
…
EC2
EC2
…
• File systems distributed across
unconstrained number of servers
• Avoids bottlenecks/constraints of
traditional file servers
• Enables high levels of aggregate
IOPS/throughput
• Data also distributed across
Availability Zones (durability,
availability)
29. How to think about EFS perf relative to EBS
Amazon EFS Amazon EBS PIOPS
Performance
Per-operation
latency
Low, consistent Lowest, consistent
Throughput
scale
Multiple GBs per second Single GB per second
Characteristics
Data availability
/ durability
Stored redundantly across multiple AZs Stored redundantly in a single AZ
Access
1 to 1000s of EC2 instances, from
multiple AZs, concurrently
Single EC2 instance in a single AZ
Use cases
Big Data and analytics, media processing
workflows, content management, web
serving, home directories
Boot volumes, transactional and
NoSQL databases, data warehousing
& ETL
30. An implication of per-operation latency: I/O size
impacts throughput of serialized operations
4 KB 32 KB 256 KB 2 MB 16 MB
I/O size
Throughput
31. How to take advantage of EFS’s distributed architecture:
Parallelize
Parallelize via multiple threads and/or multiple instances
0
5000
10000
15000
20000
25000
30000
0 20 40 60 80 100 120 140 160
IOPS
# of Total Threads
Aggregate IOPS of parallel writes using
10 m4.xlarge instances
32. Use CloudWatch for a number of views of file
system performance
DataReadIOBytes
DataWriteIOBytes
MetadataIOBytes
TotalIOBytes
Measure throughput (‘Sum’ of bytes divided by
seconds in time period) or ops/sec (‘Data
Samples’ divided by seconds in time period)
BurstCreditBalance Monitor your burst credit usage over time to
ensure sufficient throughput capacity
PermittedThroughput Compare to actual throughput to determine
whether you’re being constrained by the burst
model
ClientConnections View the number of clients connected to your
file system
PercentIOLimit Determine whether you’re being constrained by
General Purpose mode (PercentIOLimit at or
near 100%)
33. Recommended kernel version and NFS mount options
Kernel
version
Use Linux kernel 4.0+ (e.g., Amazon Linux 2016.03.0, Ubuntu
15.10 or 16.04)
Mount
options
Mount via NFSv4.1
Specify 1MB read/write buffers (“rsize”/”wsize”)
Ensure operations are asynchronous
Recommend the following mount options:
-o nfsvers=4.1,
rsize=1048576,wsize=1048576,hard,
timeo=600,retrans=2,async
34. Key recommendations
• Test your application!
• Use General Purpose mode for lowest latency, Max-I/O for
scale-out
• Use Linux kernel version 4.0 or newer, mount via NFSv4.1
• To optimize, look for opportunities to:
• Aggregate I/O
• Perform async operations
• Parallelize (demo later)
• Cache (demo later)
• Don’t forget to check your burst credit earn/spend rate when
testing – ensure sufficient amount of storage
35. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action: Copying data (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
44. GNU parallel
• Tool for executing jobs in parallel
• Similar to xargs
• Replace loops in shell scripts
• GNU parallel makes sure output
from the commands is the same
output as you would get if you had
run the commands sequentially
https://www.gnu.org/software/parallel/
For people who live life in the parallel lane
45. Use parallel threads – GNU parallel
# Create destination directory tree from source
find . -type d -print0 | parallel -j $N_THREADS -0
"mkdir -p ${DST_DIR}/{}" > /dev/null 2>&1
# Copy files
find . ! ( -type d ) -print0 | parallel -j
$N_THREADS -0 "cp -f {} ${DST_DIR}/{}"
48. Benchmark different instance types
• Determine the optimal instance size
• What is best? T2, C3, C4, M3, M4,
R3, X?
• Transfer test set of 1000 small files
• Increase thread count from 1-1024
concurrent threads
63. Summary / tl;dr
• Parallelize everything
• Threads
• Instances
• Test, test, test
• Capture & analyze test data
• Less than $5/hr for 300 instances
64. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action: WordPress (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
65. Content management & web serving
Web-based applications for creating
and managing website content.
wikis
blogs
discussion
boards
66. Free and open-source content management system hosted
on a web platform
Web software to create beautiful websites, blogs, or apps
“Free and priceless at the same time” – WordPress.org
CODE IS POETRY
67. 27% of all websites (November 2016) – Web Technology Surveys
Easiest and most popular blogging system in use on the
Web – CMS Usage Statistics
Supporting more than 60 million websites – Forbes
CODE IS POETRY
68. Available as..
• Managed Web Hosting Service
• Software package from WordPress.org installed on self-
provisioned web platform
CODE IS POETRY
72. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
73. Simple and predictable pricing
• With Amazon EFS, you pay only for the storage space you use
No minimum commitments or up-front fees
No need to provision storage in advance
No other fees, charges, or billing dimensions
• EFS price: $0.30/GB-month (US Regions)
74. AVAILABILITY
ZONE 1
REGION
EC2
AVAILABILITY
ZONE 2
AVAILABILITY
ZONE 3
EC2
Compute nodes to
manage 3rd-party
file system layer
EBS
Replicated
storage volumes
EBS
Inter-AZ traffic for
replication
Typical multi-AZ file system setup without EFS
EC2
NFS client
accessing file
system
NFS
75. TCO example
Let’s say you need to store ~500 GB and require high availability and durability
Using a shared file layer on top of EBS, you might provision 600 GB (with ~85% utilization)
and fully replicate the data to a second Availability Zone for availability/durability
Example comparative cost:
Storage (2x 600 GB EBS gp2 volumes): $120 per month
Compute (2x m4.xlarge instances): $350 per month
Inter-AZ data transfer costs (est.): $129 per month
Total $599 per month
EFS cost is (500GB * $0.30/GB-month) = $150 per month, with no additional charges
76. What to expect from this session
Recognize why and when to use Amazon EFS
Understand key technical/security concepts
Learn how to leverage EFS’s performance
See EFS in action (hands-on)
Review EFS’s economics
Discover some of our upcoming feature plans
78. Coming soon: Encryption of data at rest
• Additional layer of protection – helps you meet
your organization’s regulatory/compliance
requirements
• Integrated with AWS KMS
• Encryption/decryption handled transparently
• No extra cost
Coming early 2017
79. Coming soon: Easier mounting
• Single DNS name associated with a file system
• DNS name automatically resolves to mount target in local
Availability Zone
• Simpler mount command
Coming early 2017
mount –t nfs4 –o nfsvers=4.1
[file system DNS name]:/
/[user’s target directory]
80. New DNS name will resolve to local mount target’s IP
address
mount –t nfs4 –o nfsvers=4.1
fs-096f99a0.efs.us-west-2.amazonaws.com:/
/efs
Today’s DNS name
[AZ].[fs-id].efs.[region].amazonaws.com
Future DNS name
[AZ].[fs-id].efs.[region].amazonaws.com
Coming early 2017
81. Four scenarios for working with file data across on-
premises environments and EFS
Bursting
Migration
Move entire data set permanently to EFS
Access the data from applications running on EC2 instances
Move data set temporarily to EFS
Access the data from applications running on EC2 instances
Move data back on premises once processing finishes
Tiering
Store part of data set permanently on EFS, and keep part of data set
on premises
Access the entire data set from applications running on on-premises
servers
Backup and Disaster
Recovery
Maintain copy of entire data set on EFS
Restore the data to on premises storage or (for DR) access the data
from failed-over applications running on EC2 instances
82. Now announcing: Access your EFS file system via
AWS Direct Connect
Direct Connect EFS in your Amazon VPCOn-premises servers
83. Direct Connect support addresses three of the
scenarios
Bursting
Migration
Tiering
Backup / DR
84. Latency of AWS Direct Connect connection impacts
performance
• Added latency can be 10s of milliseconds (propagation delay over long
distances)
• If serializing I/O, latency of each operation directly impacts rate of data transfer
85. As with copying from within EC2, using a script
based on the GNU parallel tool reduces transfer time
0
100
200
300
400
500
600
700
800
900
0 2 4 6 8 10 12 14 16 18
Time
Number of Threads
Total Time to Copy 26200 Files vs Number of
Threads
86. AWS Direct Connect access available today in
three Regions
• US West (Oregon)
• US East (Ohio)
• EU (Ireland)
Coming soon to US East (N. Virginia)