Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Travis Williams, Enterprise Solutions Architect
August 17th 2016
Infrastructure as Code
Introduction to Best Practices on AWS
Learning Objectives
• Choosing the right EC2 instances
• Infrastructure as code
• AWS services that help you manage your infrastructure
as code
• Best practices for managing your AWS infrastructure,
host configuration, and applications
Choosing the Right Amazon EC2 Instance
EC2 Instance types are optimized for different use cases & come in
multiple sizes. This allows you to optimally scale resources to your
workload requirements.
AWS utilizes Intel® Xeon® processors for EC2 Instances providing
customers with high performance and value.
Consider the following when choosing your instances: Core count,
Memory size, Storage size & type, Network performance, & CPU
technologies.
Hurry Up & Go Idle - A larger compute instance can save you time and
money, therefore paying more per hour for a shorter amount of time
can be less expensive.
Get the Intel® Advantage
Intel’s latest 22nm Haswell microarchitecture on new C4 instances,
with custom Intel® Xeon® v3 processors, provides new features:
Haswell microarchitecture has better branch prediction; greater
efficiency at prefetching instructions and data; along with other
improvements that can boost existing applications’ performance by
30% or more.
P state and C state control provides the ability to individually tune each
cores performance and sleep states to improve application
performance.
Intel® AVX2.0 instructions can double the floating-point performance for
compute-intensive workloads over Intel® AVX, and provide additional
instructions useful for compression and encryption.
Intel® Processor Technologies
Intel® AVX – Get dramatically better performance for highly
parallel HPC workloads such as life science engineering, data
mining, financial analysis, or other technical computing
applications. AVX also enhances image, video, and audio
processing.
Intel® AES-NI – Enhance your security with these new
encryption instructions that reduce the performance penalty
associated with encrypting/decrypting data.
Intel® Turbo Boost Technology – Get more computing power
when you need it with performance that adapts to spikes in your
workload with Intel® Turbo Boost Technology 2.0
EC2 Instances with Intel® Technologies
Infrastructure as Code
Background
Moving to the cloud and AWS allows you to provision and
manage infrastructure in new ways:
• Infrastructure can be provisioned in seconds
• Scale can be achieved without complicated capacity
planning
• APIs let you interact with infrastructure using languages
typically used in applications
What is Infrastructure as Code?
A practice in which traditional infrastructure management
techniques are supplemented by or replaced with code-
based tools and software development techniques.
Infrastructure as Code workflow
Code
Version
Control
Code
Review
Integrate Deploy
Infrastructure as Code workflow
Code
Version
Control
Code
Review
Integrate Deploy
Text Editor
Git/SVN/
Perforce
Review
Tools
Syntax
Validation
Tools
AWS
Services
Infrastructure as Code workflow
“It’s all software”
Code
Version
Control
Code
Review
Integrate Deploy
Text Editor
Git/SVN/P
erforce
Review
Tools
Syntax
Validation
Tools
AWS
Services
Application Configuration
AWS Resources
Infrastructure as Code workflow
Operating System and Host Configuration
AWS Resources
Operating System and
Host Configuration
Application Configuration
AWS Resources
Operating System and
Host Configuration
Application Configuration
Infrastructure Resource
Management
AWS Resources
Operating System and
Host Configuration
Application Configuration
Infrastructure Resource
Management
Host Configuration
Management
AWS Resources
Operating System and
Host Configuration
Application Configuration
Infrastructure Resource
Management
Host Configuration
Management
Application Deployment
AWS Resources
Operating System and
Host Configuration
Application Configuration
AWS CloudFormation
AWS OpsWorks
AWS CodeDeploy
AWS Resources
Operating System and
Host Configuration
Application Configuration
AWS CloudFormation
AWS OpsWorks
AWS CodeDeploy
Amazon Virtual Private
Cloud (VPC)
Amazon Elastic Compute
Cloud (EC2)
AWS Identity and Access
Management (IAM)
Amazon Relational Database
Service (RDS)
Amazon Simple Storage
Service (S3)
AWS CodePipeline
…
Windows Registry
Linux Networking
OpenSSH
LDAP
AD Domain Registration
Centralized logging
System Metrics
Deployment agents
Host monitoring
…
Application dependencies
Application configuration
Service registration
Management scripts
Database credentials
…
allOfThis == $Code
AWS CloudFormation
• Create templates that describe
and model AWS infrastructure
• CloudFormation then provisions
AWS resources based on
dependency needs
• Version control/replicate/update
the templates like app code
• Integrates with development,
CI/CD, management tools
• No additional charge to use
Benefits
Templated resource
provisioning
Infrastructure
as code
Declarative
and flexible
Easy to use
CloudFormation concepts and technology
JSON formatted file
Parameter definition
Resource creation
Configuration actions
Framework
Stack creation
Stack updates
Error detection and rollback
Configured AWS resources
Comprehensive service support
Service event aware
Customizable
Template CloudFormation Stack
Anatomy of a CloudFormation template: JSON
Plain Text
Perfect for version control
Can be validated
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "AWS CloudFormation Sample Template
EC2InstanceSample: **WARNING** This template an Amazon EC2 instances.
You will be billed for the AWS resources used if you create a stack
from this template.",
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2 KeyPair to enable SSH
access to the instance",
"Type" : "String"
},
"Environment": {
"Type" : "String",
"Default" : ”Dev",
"AllowedValues" : [”Mgmt", "Dev", ”Staging", "Prod"],
"Description" : "Environment that the instances will run in.”
}
},
"Mappings" : {
"RegionMap" : {
"us-east-1" : { "AMI" : "ami-7f418316" },
"us-west-2" : { "AMI" : "ami-16fd7026" }
}
},
"Conditions" : {
”EnableEBSOptimized" : {"Fn::Equals" : [{"Ref" : " Environment
"}, ”Prod"]},
},
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : { "Ref" : "KeyName" },
"EbsOptimized " : {"Fn::If": [ " EnableEBSOptimized ",
{“true”}, {“false”}]},
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" :
"AWS::Region" }, "AMI" ]},
"UserData" : { "Fn::Base64" : "80" }
}
}
},
"Outputs" : {
"InstanceId" : {
"Description" : "InstanceId of the newly created EC2 instance",
"Value" : { "Ref" : "Ec2Instance" }
},
"PublicDNS" : {
"Description" : "Public DNSName of the newly created EC2
instance",
"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }
}
}
}
Anatomy of a CloudFormation template: JSON
Parameters
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2 KeyPair to enable
SSH access to the instance",
"Type" : "String"
},
"Environment": {
"Type" : "String",
"Default" : ”Dev",
"AllowedValues" : [”Mgmt", "Dev", ”Staging", "Prod"],
"Description" : "Environment that the instances will run
in.”
}
},
Mappings
"Mappings" : {
"RegionMap" : {
"us-east-1" : { "AMI" : "ami-7f418316" },
"us-west-2" : { "AMI" : "ami-16fd7026" }
}
},
Conditionals
"Conditions" : {
”EnableEBSOptimized" : {"Fn::Equals" : [{"Ref" : " Environment "}, ”Prod"]},
},
Resources
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : { "Ref" : "KeyName" },
"EbsOptimized " : {"Fn::If": [ " EnableEBSOptimized ", {“true”}, {“false”}]},
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" },
"AMI" ]},
"UserData" : { "Fn::Base64" : "80" }
}
}
},
Outputs
Outputs" : {
"InstanceId" : {
"Description" : "InstanceId of the newly created EC2 instance",
"Value" : { "Ref" : "Ec2Instance" }
},
"PublicDNS" : {
"Description" : "Public DNSName of the newly created EC2 instance",
"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }
}
}
}
Headers
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "AWS CloudFormation Sample Template
EC2InstanceSample: **WARNING** This template an Amazon EC2
instances. You will be billed for the AWS resources used if you
create a stack from this template.",
Anatomy of a CloudFormation template: JSON
Description of what your stack does, contains, etc
Provision time values that add structured flexibility and customization
Pre-defined conditional case statements
Conditional values set via evaluations of passed references
AWS resource definitions
Resulting attributes of stack resource creation
Headers
Parameters
Mappings
Conditionals
Resources
Outputs
Template components
Bootstrapping applications & handling updates
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : { "Ref" : "KeyName" },
"SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[
"#!/bin/bash -ex","n",
"yum -y install gcc-c++ make","n",
"yum -y install mysql-devel sqlite-devel","n",
"yum -y install ruby-rdoc rubygems ruby-mysql ruby-devel","n",
"gem install --no-ri --no-rdoc rails","n",
"gem install --no-ri --no-rdoc mysql","n",
"gem install --no-ri --no-rdoc sqlite3","n",
"rails new myapp","n",
"cd myapp","n",
"rails server -d","n"]]}}
}
}
Option 1: Use EC2 UserData, which is available as a property of
AWS::EC2::Instance resources
cfn-init
cfn-hup
Option 2: AWS CloudFormation
provides helper scripts for
deployment within your EC2
instances
Metadata Key —
AWS::CloudFormation::Init
Cfn-init reads this metadata key and
installs the packages listed in this key
(e.g., httpd, mysql, and php). Cfn-init
also retrieves and expands files listed
as sources.
Amazon EC2
AWS CloudFormation
cfn-signal
cfn-get-
metadata
Bootstrapping applications & handling updates
Manage a wide range of AWS services & resources
• Amazon EC2
• Amazon EC2 Container Service
• Amazon EC2 Container Registry
• Amazon EC2 Simple Systems Manager
• AWS Lambda (including event sources)
• AWS Elastic Beanstalk
• Auto Scaling (including Spot Fleet)
• Amazon VPC & Managed NAT Gateway
• Elastic Load Balancing
• Amazon Route 53
• Amazon CloudFront
• AWS WAF
• Amazon S3
• Amazon RDS
• Amazon Redshift
• Amazon DynamoDB
• Amazon ElastiCache
• Amazon RDS (including Aurora)
• Amazon Elastic MapReduce
• Amazon Elasticsearch Service
• AWS Data Pipeline
• Amazon IAM (including managed policies)
• Amazon Simple AD / Microsoft AD
• Amazon Kinesis
• Amazon SNS
• Amazon SQS
• AWS CloudTrail
• Amazon CloudWatch
• AWS Config
• AWS Key Management Service
• AWS OpsWorks
• AWS CodeDeploy
• AWS CodePipeline
• Amazon Workspaces
• Amazon GameLift
AWS resource support is always growing. See up to date list here.
Template File
Defining Stack
• The entire infrastructure can
be represented in an AWS
CloudFormation template.
Many stacks & environments from one template
Template File
Defining Stack
• The entire infrastructure can
be represented in an AWS
CloudFormation template.
• Use the version control
system of your choice to
store and track changes to
this template
Git
Perforce
SVN
…
Many stacks & environments from one template
Template File
Defining Stack
• The entire infrastructure can
be represented in an AWS
CloudFormation template.
• Use the version control
system of your choice to
store and track changes to
this template
• Build out multiple
environments, such as for
Development, Test,
Production and even DR
using the same template
Git
Perforce
SVN
…
Dev
Test
Prod
Many stacks & environments from one template
Infrastructure as Code with CloudFormation
Versioning
You track changes within your code
Do it with your infrastructure:
• What is changing?
• Who made that change?
• When was it made?
• Why was it made?(tied to ticket/bug/project systems?)
Testing your template:
• Validate via API/CLI
• $ aws cloudformation validate-template – confirm CF
syntax
• Use something like Jsonlint (http://jsonlint.com/) to find
JSON issues like missing commas, brackets
• Throw this into your testing/continuous integration pipelines
Testing your CloudFormation templates
Visualizing your CloudFormation templates
• AWS
CloudFormation
Designer
• Visualize template
resources
• Modify template with
drag-drop gestures
• Customize sample
templates
Deploying your CloudFormation templates
Deploy & update via console or API/command line
OR
• aws cloudformation create-stack --stack-name
myteststack --template-body
file:////home//local//test//sampletemplate.json --
parameters
ParameterKey=string,ParameterValue=string
But what do we do once your
resources are provisioned and
running?
Your infrastructure needs ongoing management
• Updates/patches?
• New software?
• New configurations?
• New code deploys?
• Pool specific changes?
• Environment specific changes?
• Run commands across all hosts?
• Be on top of all running resources?
Ongoing management requires proper tooling
Some common challenges:
• Changing a vhost configuration on every web server across
multiple environments (dev, stage, prod)
• Installing a package on certain hosts to test out newer versions
• Changing LDAP config on every running Amazon EC2 Linux host
when they are across 25 different CloudFormation templates
We need a tool to interact with
each host that we manage and
make it easier to configure
them
• Configuration management service
for automating operational tasks
using Chef
• Model, control and automate
applications of nearly any scale and
complexity
• Manage Linux and Windows
environments
• Supports both AWS and on-
premises servers
• Launched in 2013
AWS OpsWorks
AWS OpsWorks concepts
A stack represents
the cloud
infrastructure and
applications that
you want to manage
together.
A layer defines how
to setup and
configure a set of
instances and
related resources.
Decide how to
scale: manually,
with 24/7 instances,
or automatically,
with load-based or
time-based
instances.
Then deploy your
app to specific
instances and
customize the
deployment with
Chef recipes.
AWS OpsWorks concepts: instance lifecycle
Setup Configure Deploy Undeploy Shutdown
Agent on each instance understands a set
of commands that are triggered by
OpsWorks. The agent then runs Chef.
OpsWorks agent communication
1. Instance connects with OpsWorks
service to send keep alive heartbeat
and receive lifecycle events
2. OpsWorks sends lifecycle event with
pointer to configuration JSON
(metadata, recipes) in S3 bucket
3. Download configuration JSON
4. Pull cookbooks and other build assets
from your repo
5. Execute recipe
6. Upload Chef log
7. Report Chef run status
EC2
Instance
OpsWorks
Service
“Deploy App”
Your repo,
e.g. GitHub







How OpsWorks bootstraps EC2 instances
Instance is started with IAM role
• UserData passed with instance private key, OpsWorks public key
• Instance downloads and installs OpsWorks agent
Agent connects to instance service, gets run info
• Authenticate instance using instance’s IAM role
• Pick-up configuration JSON from the OpsWorks instance queue
• Decrypt & verify message, run Chef recipes
• Upload Chef log, return Chef run status
Agent polls instance service for more messages
AWS OpsWorks + Chef
OpsWorks uses Chef to configure the software on the
instance
OpsWorks provides many Chef Server functions to users.
• Associate cookbooks with instances
• Dynamic metadata that describes each registered node in the
infrastructure
Supports "Push" Command and Control Client Runs
Support for community cookbooks
Working with Chef and OpsWorks
Similar to CloudFormation templates and application code:
• Mixture of JSON and a Ruby DSL
• Tools exist to do linting and syntax checking
• Versioning
• Built in cookbook versioning
• Some manual/processes scripted abilities
• But still can use source control for versioning
• Use with continuous integration systems just like AWS
CloudFormation templates and the rest of your code
AWS OpsWorks
Deploying applications
Automates code deployments to any instance
Handles the complexity of updating your
applications
Avoid downtime during application deployment
Deploy to Amazon EC2 or on-premise servers,
in any language and on any operating system
Integrates with 3rd party tools and AWS
services
AWS CodeDeploy
AWS CodeDeploy concepts
Application
Revision #1
Revision #2
Revision #3
What to deploy?
Revision #1
How to deploy?
Instance
Instance
Instance
Deployment Group
Auto-Scaling Group
Where to deploy?
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html
• Send application files to one
directory and configuration files to
another
• Set specific permissions on specific
directories & files
• Remove/Add instance to ELB
• Install dependency packages
• Start Apache
• Confirm successful deploy
• More!
permissions:
- object: /var/www/html
pattern: “*.html”
owner: root
group: root
mode: 755
hooks:
ApplicationStop:
- location: scripts/deregister_from_elb.sh
BeforeInstall:
- location: scripts/install_dependencies.sh
ApplicationStart:
- location: scripts/start_httpd.sh
ValidateService:
- location: scripts/test_site.sh
- location: scripts/register_with_elb.sh
How It Works: Package app with Appspec.yml
How It Works: Specify targets
Group instances by:
• Auto Scaling Group
• Amazon EC2 Tag
• On-Premises Tag
Dev Deployment Group
AgentAgent Agent
Prod Deployment Group
AgentAgent Agent
AgentAgent Agent
How It Works: Deploy
• AWS CLI & SDKs
• AWS Console
• AWS CodePipeline & CI/CD Partners
• S3, GitHub
aws deploy create-deployment 
--application-name MyApp 
--deployment-group-name TargetGroup 
--s3-location bucket=MyBucket,key=MyApp.zip
v2 v1 v1 v1 v1 v1 v1 v1
v2 v2 v1 v1 v1 v1 v1 v1
v2 v2 v2 v2 v1 v1 v1 v1
v2 v2 v2 v2 v2 v2 v2 v2
One-at-a-time
Min. healthy hosts = 99%
[Custom]
Min. healthy hosts = 75%
Half-at-a-time
Min. healthy hosts = 50%
All-at-once
Min. healthy hosts = 0
Choose your deployment configuration
Summary
Summary
• Create/update/manage AWS resources and their configuration and
properties with CloudFormation
• You can configure OpsWorks and CodeDeploy via
CloudFormation
• Use OpsWorks for ongoing tweaks to software/configuration of host
based applications and the operating system
• You can configure and deploy CodeDeploy’s agent with
OpsWorks
• Use CodeDeploy to deploy your applications and their configurations
Best practices
• Your CloudFormation templates and Chef cookbooks should
go in separate repositories
• Include appspec.yml file and related scripts in your
application’s code repositories
• Every commit should cause an execution of your continuous
delivery pipeline to lint, validate and/or test
• Use each related service’s CLI/console/APIs to update or
deploy as necessary
AWS Resources
Operating System and
Host Configuration
Application Configuration
AWS CloudFormation
AWS OpsWorks
AWS CodeDeploy
Amazon Virtual Private
Cloud (VPC)
Amazon Elastic Compute
Cloud (EC2)
AWS Identity and Access
Management (IAM)
Amazon Relational Database
Service (RDS)
Amazon Simple Storage
Service (S3)
AWS CodePipeline
…
Windows Registry
Linux Networking
OpenSSH
LDAP
AD Domain Registration
Centralized logging
System Metrics
Deployment agents
Host monitoring
…
Application dependencies
Application configuration
Service registration
Management scripts
Database credentials
…
allOfThis == $Code
Learn More
• AWS CloudFormation
• https://aws.amazon.com/cloudformation/
• https://aws.amazon.com/documentation/cloudformation/
• https://aws.amazon.com/cloudformation/aws-cloudformation-templates/
• AWS OpsWorks
• https://aws.amazon.com/opsworks/
• https://aws.amazon.com/documentation/opsworks/
• https://github.com/aws/opsworks-cookbooks
• AWS CodeDeploy
• https://aws.amazon.com/codedeploy/
• https://aws.amazon.com/documentation/codedeploy/
• https://github.com/awslabs/aws-codedeploy-samples

More Related Content

Managing Your Infrastructure as Code by Travis Williams, Solutions Architect, AWS

  • 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Travis Williams, Enterprise Solutions Architect August 17th 2016 Infrastructure as Code Introduction to Best Practices on AWS
  • 2. Learning Objectives • Choosing the right EC2 instances • Infrastructure as code • AWS services that help you manage your infrastructure as code • Best practices for managing your AWS infrastructure, host configuration, and applications
  • 3. Choosing the Right Amazon EC2 Instance EC2 Instance types are optimized for different use cases & come in multiple sizes. This allows you to optimally scale resources to your workload requirements. AWS utilizes Intel® Xeon® processors for EC2 Instances providing customers with high performance and value. Consider the following when choosing your instances: Core count, Memory size, Storage size & type, Network performance, & CPU technologies. Hurry Up & Go Idle - A larger compute instance can save you time and money, therefore paying more per hour for a shorter amount of time can be less expensive.
  • 4. Get the Intel® Advantage Intel’s latest 22nm Haswell microarchitecture on new C4 instances, with custom Intel® Xeon® v3 processors, provides new features: Haswell microarchitecture has better branch prediction; greater efficiency at prefetching instructions and data; along with other improvements that can boost existing applications’ performance by 30% or more. P state and C state control provides the ability to individually tune each cores performance and sleep states to improve application performance. Intel® AVX2.0 instructions can double the floating-point performance for compute-intensive workloads over Intel® AVX, and provide additional instructions useful for compression and encryption.
  • 5. Intel® Processor Technologies Intel® AVX – Get dramatically better performance for highly parallel HPC workloads such as life science engineering, data mining, financial analysis, or other technical computing applications. AVX also enhances image, video, and audio processing. Intel® AES-NI – Enhance your security with these new encryption instructions that reduce the performance penalty associated with encrypting/decrypting data. Intel® Turbo Boost Technology – Get more computing power when you need it with performance that adapts to spikes in your workload with Intel® Turbo Boost Technology 2.0
  • 6. EC2 Instances with Intel® Technologies
  • 8. Background Moving to the cloud and AWS allows you to provision and manage infrastructure in new ways: • Infrastructure can be provisioned in seconds • Scale can be achieved without complicated capacity planning • APIs let you interact with infrastructure using languages typically used in applications
  • 9. What is Infrastructure as Code? A practice in which traditional infrastructure management techniques are supplemented by or replaced with code- based tools and software development techniques.
  • 10. Infrastructure as Code workflow Code Version Control Code Review Integrate Deploy
  • 11. Infrastructure as Code workflow Code Version Control Code Review Integrate Deploy Text Editor Git/SVN/ Perforce Review Tools Syntax Validation Tools AWS Services
  • 12. Infrastructure as Code workflow “It’s all software” Code Version Control Code Review Integrate Deploy Text Editor Git/SVN/P erforce Review Tools Syntax Validation Tools AWS Services
  • 13. Application Configuration AWS Resources Infrastructure as Code workflow Operating System and Host Configuration
  • 14. AWS Resources Operating System and Host Configuration Application Configuration
  • 15. AWS Resources Operating System and Host Configuration Application Configuration Infrastructure Resource Management
  • 16. AWS Resources Operating System and Host Configuration Application Configuration Infrastructure Resource Management Host Configuration Management
  • 17. AWS Resources Operating System and Host Configuration Application Configuration Infrastructure Resource Management Host Configuration Management Application Deployment
  • 18. AWS Resources Operating System and Host Configuration Application Configuration AWS CloudFormation AWS OpsWorks AWS CodeDeploy
  • 19. AWS Resources Operating System and Host Configuration Application Configuration AWS CloudFormation AWS OpsWorks AWS CodeDeploy Amazon Virtual Private Cloud (VPC) Amazon Elastic Compute Cloud (EC2) AWS Identity and Access Management (IAM) Amazon Relational Database Service (RDS) Amazon Simple Storage Service (S3) AWS CodePipeline … Windows Registry Linux Networking OpenSSH LDAP AD Domain Registration Centralized logging System Metrics Deployment agents Host monitoring … Application dependencies Application configuration Service registration Management scripts Database credentials …
  • 21. AWS CloudFormation • Create templates that describe and model AWS infrastructure • CloudFormation then provisions AWS resources based on dependency needs • Version control/replicate/update the templates like app code • Integrates with development, CI/CD, management tools • No additional charge to use
  • 23. CloudFormation concepts and technology JSON formatted file Parameter definition Resource creation Configuration actions Framework Stack creation Stack updates Error detection and rollback Configured AWS resources Comprehensive service support Service event aware Customizable Template CloudFormation Stack
  • 24. Anatomy of a CloudFormation template: JSON Plain Text Perfect for version control Can be validated
  • 25. { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "AWS CloudFormation Sample Template EC2InstanceSample: **WARNING** This template an Amazon EC2 instances. You will be billed for the AWS resources used if you create a stack from this template.", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" }, "Environment": { "Type" : "String", "Default" : ”Dev", "AllowedValues" : [”Mgmt", "Dev", ”Staging", "Prod"], "Description" : "Environment that the instances will run in.” } }, "Mappings" : { "RegionMap" : { "us-east-1" : { "AMI" : "ami-7f418316" }, "us-west-2" : { "AMI" : "ami-16fd7026" } } }, "Conditions" : { ”EnableEBSOptimized" : {"Fn::Equals" : [{"Ref" : " Environment "}, ”Prod"]}, }, "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "EbsOptimized " : {"Fn::If": [ " EnableEBSOptimized ", {“true”}, {“false”}]}, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : "80" } } } }, "Outputs" : { "InstanceId" : { "Description" : "InstanceId of the newly created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } }, "PublicDNS" : { "Description" : "Public DNSName of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] } } } } Anatomy of a CloudFormation template: JSON
  • 26. Parameters "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" }, "Environment": { "Type" : "String", "Default" : ”Dev", "AllowedValues" : [”Mgmt", "Dev", ”Staging", "Prod"], "Description" : "Environment that the instances will run in.” } }, Mappings "Mappings" : { "RegionMap" : { "us-east-1" : { "AMI" : "ami-7f418316" }, "us-west-2" : { "AMI" : "ami-16fd7026" } } }, Conditionals "Conditions" : { ”EnableEBSOptimized" : {"Fn::Equals" : [{"Ref" : " Environment "}, ”Prod"]}, }, Resources "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "EbsOptimized " : {"Fn::If": [ " EnableEBSOptimized ", {“true”}, {“false”}]}, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : "80" } } } }, Outputs Outputs" : { "InstanceId" : { "Description" : "InstanceId of the newly created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } }, "PublicDNS" : { "Description" : "Public DNSName of the newly created EC2 instance", "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] } } } } Headers { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "AWS CloudFormation Sample Template EC2InstanceSample: **WARNING** This template an Amazon EC2 instances. You will be billed for the AWS resources used if you create a stack from this template.", Anatomy of a CloudFormation template: JSON
  • 27. Description of what your stack does, contains, etc Provision time values that add structured flexibility and customization Pre-defined conditional case statements Conditional values set via evaluations of passed references AWS resource definitions Resulting attributes of stack resource creation Headers Parameters Mappings Conditionals Resources Outputs Template components
  • 28. Bootstrapping applications & handling updates "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ], "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[ "#!/bin/bash -ex","n", "yum -y install gcc-c++ make","n", "yum -y install mysql-devel sqlite-devel","n", "yum -y install ruby-rdoc rubygems ruby-mysql ruby-devel","n", "gem install --no-ri --no-rdoc rails","n", "gem install --no-ri --no-rdoc mysql","n", "gem install --no-ri --no-rdoc sqlite3","n", "rails new myapp","n", "cd myapp","n", "rails server -d","n"]]}} } } Option 1: Use EC2 UserData, which is available as a property of AWS::EC2::Instance resources
  • 29. cfn-init cfn-hup Option 2: AWS CloudFormation provides helper scripts for deployment within your EC2 instances Metadata Key — AWS::CloudFormation::Init Cfn-init reads this metadata key and installs the packages listed in this key (e.g., httpd, mysql, and php). Cfn-init also retrieves and expands files listed as sources. Amazon EC2 AWS CloudFormation cfn-signal cfn-get- metadata Bootstrapping applications & handling updates
  • 30. Manage a wide range of AWS services & resources • Amazon EC2 • Amazon EC2 Container Service • Amazon EC2 Container Registry • Amazon EC2 Simple Systems Manager • AWS Lambda (including event sources) • AWS Elastic Beanstalk • Auto Scaling (including Spot Fleet) • Amazon VPC & Managed NAT Gateway • Elastic Load Balancing • Amazon Route 53 • Amazon CloudFront • AWS WAF • Amazon S3 • Amazon RDS • Amazon Redshift • Amazon DynamoDB • Amazon ElastiCache • Amazon RDS (including Aurora) • Amazon Elastic MapReduce • Amazon Elasticsearch Service • AWS Data Pipeline • Amazon IAM (including managed policies) • Amazon Simple AD / Microsoft AD • Amazon Kinesis • Amazon SNS • Amazon SQS • AWS CloudTrail • Amazon CloudWatch • AWS Config • AWS Key Management Service • AWS OpsWorks • AWS CodeDeploy • AWS CodePipeline • Amazon Workspaces • Amazon GameLift AWS resource support is always growing. See up to date list here.
  • 31. Template File Defining Stack • The entire infrastructure can be represented in an AWS CloudFormation template. Many stacks & environments from one template
  • 32. Template File Defining Stack • The entire infrastructure can be represented in an AWS CloudFormation template. • Use the version control system of your choice to store and track changes to this template Git Perforce SVN … Many stacks & environments from one template
  • 33. Template File Defining Stack • The entire infrastructure can be represented in an AWS CloudFormation template. • Use the version control system of your choice to store and track changes to this template • Build out multiple environments, such as for Development, Test, Production and even DR using the same template Git Perforce SVN … Dev Test Prod Many stacks & environments from one template
  • 34. Infrastructure as Code with CloudFormation Versioning You track changes within your code Do it with your infrastructure: • What is changing? • Who made that change? • When was it made? • Why was it made?(tied to ticket/bug/project systems?)
  • 35. Testing your template: • Validate via API/CLI • $ aws cloudformation validate-template – confirm CF syntax • Use something like Jsonlint (http://jsonlint.com/) to find JSON issues like missing commas, brackets • Throw this into your testing/continuous integration pipelines Testing your CloudFormation templates
  • 36. Visualizing your CloudFormation templates • AWS CloudFormation Designer • Visualize template resources • Modify template with drag-drop gestures • Customize sample templates
  • 37. Deploying your CloudFormation templates Deploy & update via console or API/command line OR • aws cloudformation create-stack --stack-name myteststack --template-body file:////home//local//test//sampletemplate.json -- parameters ParameterKey=string,ParameterValue=string
  • 38. But what do we do once your resources are provisioned and running?
  • 39. Your infrastructure needs ongoing management • Updates/patches? • New software? • New configurations? • New code deploys? • Pool specific changes? • Environment specific changes? • Run commands across all hosts? • Be on top of all running resources?
  • 40. Ongoing management requires proper tooling Some common challenges: • Changing a vhost configuration on every web server across multiple environments (dev, stage, prod) • Installing a package on certain hosts to test out newer versions • Changing LDAP config on every running Amazon EC2 Linux host when they are across 25 different CloudFormation templates
  • 41. We need a tool to interact with each host that we manage and make it easier to configure them
  • 42. • Configuration management service for automating operational tasks using Chef • Model, control and automate applications of nearly any scale and complexity • Manage Linux and Windows environments • Supports both AWS and on- premises servers • Launched in 2013 AWS OpsWorks
  • 43. AWS OpsWorks concepts A stack represents the cloud infrastructure and applications that you want to manage together. A layer defines how to setup and configure a set of instances and related resources. Decide how to scale: manually, with 24/7 instances, or automatically, with load-based or time-based instances. Then deploy your app to specific instances and customize the deployment with Chef recipes.
  • 44. AWS OpsWorks concepts: instance lifecycle Setup Configure Deploy Undeploy Shutdown Agent on each instance understands a set of commands that are triggered by OpsWorks. The agent then runs Chef.
  • 45. OpsWorks agent communication 1. Instance connects with OpsWorks service to send keep alive heartbeat and receive lifecycle events 2. OpsWorks sends lifecycle event with pointer to configuration JSON (metadata, recipes) in S3 bucket 3. Download configuration JSON 4. Pull cookbooks and other build assets from your repo 5. Execute recipe 6. Upload Chef log 7. Report Chef run status EC2 Instance OpsWorks Service “Deploy App” Your repo, e.g. GitHub       
  • 46. How OpsWorks bootstraps EC2 instances Instance is started with IAM role • UserData passed with instance private key, OpsWorks public key • Instance downloads and installs OpsWorks agent Agent connects to instance service, gets run info • Authenticate instance using instance’s IAM role • Pick-up configuration JSON from the OpsWorks instance queue • Decrypt & verify message, run Chef recipes • Upload Chef log, return Chef run status Agent polls instance service for more messages
  • 47. AWS OpsWorks + Chef OpsWorks uses Chef to configure the software on the instance OpsWorks provides many Chef Server functions to users. • Associate cookbooks with instances • Dynamic metadata that describes each registered node in the infrastructure Supports "Push" Command and Control Client Runs Support for community cookbooks
  • 48. Working with Chef and OpsWorks Similar to CloudFormation templates and application code: • Mixture of JSON and a Ruby DSL • Tools exist to do linting and syntax checking • Versioning • Built in cookbook versioning • Some manual/processes scripted abilities • But still can use source control for versioning • Use with continuous integration systems just like AWS CloudFormation templates and the rest of your code
  • 51. Automates code deployments to any instance Handles the complexity of updating your applications Avoid downtime during application deployment Deploy to Amazon EC2 or on-premise servers, in any language and on any operating system Integrates with 3rd party tools and AWS services AWS CodeDeploy
  • 52. AWS CodeDeploy concepts Application Revision #1 Revision #2 Revision #3 What to deploy? Revision #1 How to deploy? Instance Instance Instance Deployment Group Auto-Scaling Group Where to deploy?
  • 53. version: 0.0 os: linux files: - source: / destination: /var/www/html • Send application files to one directory and configuration files to another • Set specific permissions on specific directories & files • Remove/Add instance to ELB • Install dependency packages • Start Apache • Confirm successful deploy • More! permissions: - object: /var/www/html pattern: “*.html” owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh How It Works: Package app with Appspec.yml
  • 54. How It Works: Specify targets Group instances by: • Auto Scaling Group • Amazon EC2 Tag • On-Premises Tag Dev Deployment Group AgentAgent Agent Prod Deployment Group AgentAgent Agent AgentAgent Agent
  • 55. How It Works: Deploy • AWS CLI & SDKs • AWS Console • AWS CodePipeline & CI/CD Partners • S3, GitHub aws deploy create-deployment --application-name MyApp --deployment-group-name TargetGroup --s3-location bucket=MyBucket,key=MyApp.zip
  • 56. v2 v1 v1 v1 v1 v1 v1 v1 v2 v2 v1 v1 v1 v1 v1 v1 v2 v2 v2 v2 v1 v1 v1 v1 v2 v2 v2 v2 v2 v2 v2 v2 One-at-a-time Min. healthy hosts = 99% [Custom] Min. healthy hosts = 75% Half-at-a-time Min. healthy hosts = 50% All-at-once Min. healthy hosts = 0 Choose your deployment configuration
  • 58. Summary • Create/update/manage AWS resources and their configuration and properties with CloudFormation • You can configure OpsWorks and CodeDeploy via CloudFormation • Use OpsWorks for ongoing tweaks to software/configuration of host based applications and the operating system • You can configure and deploy CodeDeploy’s agent with OpsWorks • Use CodeDeploy to deploy your applications and their configurations
  • 59. Best practices • Your CloudFormation templates and Chef cookbooks should go in separate repositories • Include appspec.yml file and related scripts in your application’s code repositories • Every commit should cause an execution of your continuous delivery pipeline to lint, validate and/or test • Use each related service’s CLI/console/APIs to update or deploy as necessary
  • 60. AWS Resources Operating System and Host Configuration Application Configuration AWS CloudFormation AWS OpsWorks AWS CodeDeploy Amazon Virtual Private Cloud (VPC) Amazon Elastic Compute Cloud (EC2) AWS Identity and Access Management (IAM) Amazon Relational Database Service (RDS) Amazon Simple Storage Service (S3) AWS CodePipeline … Windows Registry Linux Networking OpenSSH LDAP AD Domain Registration Centralized logging System Metrics Deployment agents Host monitoring … Application dependencies Application configuration Service registration Management scripts Database credentials …
  • 62. Learn More • AWS CloudFormation • https://aws.amazon.com/cloudformation/ • https://aws.amazon.com/documentation/cloudformation/ • https://aws.amazon.com/cloudformation/aws-cloudformation-templates/ • AWS OpsWorks • https://aws.amazon.com/opsworks/ • https://aws.amazon.com/documentation/opsworks/ • https://github.com/aws/opsworks-cookbooks • AWS CodeDeploy • https://aws.amazon.com/codedeploy/ • https://aws.amazon.com/documentation/codedeploy/ • https://github.com/awslabs/aws-codedeploy-samples