Amazon Management and Monitoring Services - Resp
Amazon Management and Monitoring Services - Resp
Amazon Management and Monitoring Services - Resp
Prelude
This course is a combination of some of AWS services used for Management, Monitoring and
Deployment of resources.
To have hassle-free learning, you are expected to have a good understanding of AWS
Essentials and core services like S3.
If you are new here, please complete the prerequisites mentioned.
Context
We have a plenty of services in AWS fulfilling the use cases of projects that vary from simple
computing system to advanced Robotics.
There are more than 150 services available in AWS. There are services for maintenance and
management of the other resources. These servcies are useful to the user in both setting up
and carry-out maintenance as a hassle-free task.
Here, we are going to discuss a few services like :
1. CloudFormation
2. CloudWatch
3. CloudTrail
4. CloudFront
5. SMS
6. SQS
Infrastructure as Code
Infrastructure as Code (IaC) is a technique used for infrastructure provisioning through a
description or code, instead of using regular manual configuration of Instances, Networks,
Databases, and so on.
There are many tools that use the IaC technique for infrastructure
provisioning. CloudFormation is one such tool.
More on IaC
The following video gives you a clear idea of IaC and its importance.
Advantages of IaC
What is CloudFormation?
The following is a sample template used to create an EC2 instance and attach an EBS volume
to it:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "A sample template",
"Resources" : {
"MyEC2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"ImageId" : "ami-0ff8a91507f77f867",
"InstanceType" : "t2.micro",
"KeyName" : "testkey",
"BlockDeviceMappings" : [
{
"DeviceName" : "/dev/sdm",
"Ebs" : {
"VolumeType" : "io1",
"Iops" : "200",
"DeleteOnTermination" : "false",
"VolumeSize" : "20"
}
}
]
}
}
}
}
2. Stacks: AWS CloudFormation stack is a group of resources that can be controlled and
created by using the template. You can create, update, and delete a collection of
resources by creating, updating, and deleting stacks. Stacks can be created by using
console, CLI, and API.
3. Change Sets: To make changes to the running resources in the stack, you have to
update the stack. However, before making changes to the resources through updated
stack, you can create a Change Set which consists of the required changes. These
help to examine the impact of the changes made on the resources.
The following video explains template schema and how to use them to design a template, and
a sample stack creation.
CloudFormation Designer
ntrinsic Functions
Intrinsic Functions are useful when you want to assign values to properties in a template that
are not available until runtime.
CloudFormation provides a set of intrinsic functions that help in managing stacks:
Fn : : Base64 : Returns the Base64 representation of the input string.
Fn : : Cidr : Returns an array of CIDR address blocks, and the number of blocks is
dependent on the parameter used.
Fn : : FindInMap : Returns values corresponding to keys in a two-level map that are
declared in the Mappings section.
Fn : : GetAtt : Returns the value of an attribute from a resource.
Fn : : GetAZs : Returns the array list of AZs in a specified region.
Intrinsic Functions
Helper Scripts are python scripts that help in installing softwares and services in EC2 instances
as a part of a stack.
They are of four types :
1. Cfn-init : Reads the template metadata from AWS : : CloudFormation : : Init, parses the
metadata, installs packages, writes files, and starts/stops services.
2. Cfn-Signal : Makes CloudFormation indicate the successful creation of EC2 instances
with all softwares installed.
3. Cfn-get-metadata : Used to print metadata block from CloudFormation and print it.
4. Cfn-hup : Used to detect changes in instance metadata, and performs specified
actions.
For more details about Helper Scripts, refer to helper scripts.
Nested Stacks
Nested Stacks are stacks created within another stack by using AWS : : CloudFormation : :
Stack.*
Are very useful when you use resources with the same configuration multiple times.
These can be consolidated into a stack, and further used.
Can have more nested stacks within themselves.
Each nested stack has a parent stack and only one root stack.
Using nested stacks is considered a good practice.
Refer to the following video to understand how Nested Stacks work in CloudFormation:
CloudWatch - Introduction
CloudWatch is a real-time management and monitoring service provided by AWS, which:
Provides you with complete insight of an application at one place.
Is extensively used for application resources such as EC2 instances, ELBs, Database
Servers, EBS, and so on.
Helps you collect metrics, log files, set alarms, and also automatically react to changes
in resources.
Automatically provides metrics for CPU utilization, latency, and request counts.
Is very useful when taking automated actions, troubleshooting issues, requiring a total
insight of the application, and optimizing the application.
Is used to create custom dashboards to display metrics.
Can be accessed through the Console, AWS CLI, CloudWatch API, and various SDKs.
CloudWatch Components
CloudWatch Monitoring
CloudWatch Monitoring...
Monitoring Types
CloudWatch Dashboard
There will be at least five metrics for a single EC2 instance. Therefore, how do you manage
hundreds of them? Moreover, many of them might not be useful to you. To overcome this, we
have Dashboards which are customizable home pages.
Dashboards help in monitoring all required metrics and alarms from different regions in
a single page.
It creates a common view of resource data which can be shared by the team, and
enables faster resolutions to issues.
With Dashboards, you can display the latest value of a metric, a simple line graph of
one or more metrics, or a stacked area graph of multiple metrics.
The following video shows how custom metrics are published to CloudWatch:
CloudWatch Logs
CloudWatch Logs is a place to monitor, access, and store logs from various AWS services
such as EC2 instances, Lambda, S3, VPC, and also the application running on EC2.
There are few terms that are exclusive to CloudWatch, and helps us understand Logging better:
Log Events : A log event is a record of activity of the application or resource being
monitored by CloudWatch. It contains two properties: the timestamp of the event, and
the raw event message.
Log Streams : A log stream is the stream of events from an instance or application that
is being monitored.
Log Groups : A log group is the collection of log streams that have the same retention,
monitoring and access control. Each log stream belongs to a log group, and there is no
limit on how many log streams can be there in a log group.
Metric Filters : Metric filters extract metric observations from events, and transform
them into data points in CloudWatch metrics.
The following video shows you how logs work, by using the example of streaming logs from an
EC2 instance to CloudWatch:
CloudWatch Logs Insight is a highly available, and interactive log analytics service for
CloudWatch logs.
For more information about log insights, refer to AWS official documentation.
CloudWatch Events
Events is also an importance feature of CloudWatch. It provides a near real-time event stream,
as changes take place in the AWS resources present. The change of events can trigger
predefined actions on targets, such as send notifications and emails, invoke lambda functions,
change instance state,and so on.
By using simple rules, you can match events, and route them to one or more targets,
functions, or streams.
Helps to identify operational changes when they take place.
Can be used to schedule automated actions that self-trigger at certain times, by using
cron or rate expressions.
It supports many crucial services as its targets in AWS.
AWS CloudTrail
CloudTrail is a security and management service which enables governance, compliance, and
risk auditing of the AWS account. It continuously logs and monitors activities related to actions
in your account.
Every action taken through console, AWS CLI, AWS SDK's, and other AWS services is
logged. - The information logged contains the identity of the API caller, time of the call,
the source IP, request parameters, and response of the call.
It helps in security analysis, resource change tracking, and compliance auditing.
It is enabled by default when the AWS account is created.
All the events are saved in event history in CloudTrail and saved for 90 days.
You can create a trail to archive, analyze, and respond to changes, and log all of them
to the S3 bucket you specify.
Trails : Trails are used to retain the record of API event logs in S3. It can be understood as a
container for retaining event logs. This trail helps to use CloudWatch in conjunction with
CloudTrail, to enable metrics and alarms.
Organization Trails : This is a configuration in CloudTrail, in which the event logs of the users
in an account are delivered to the master account's S3 bucket, CloudWatch logs, and events.
This provides a uniform event logging strategy.
A trail is created with the same name in all user accounts, when it is created by the master.
The following video shows how CloudTrail is enabled, and how it is retained in S3:
The following video demonstrates the working of CloudTrail in conjunction with CloudWatch:
Validating CloudTrail Logs
In an organization, many people may have access to CloudTrail logs, and also privileges to alter
the log files. Therefore, to find out if a log has been edited, deleted, or left unchanged after
CloudTrail delivery, you can use CloudTrail log file integrity validation.
This is built by using industry standard algorithms such as SHA-256 for hashing, and
SHA-256 with RSA for digital signing, making it impossible to alter log files without
detection.
When CloudTrail log validation is enabled every hour, it also creates and delivers a file
named digest file that references the log files for the last hour, and contains a hash of
each file.
AWS CLI is used to validate CloudTrail log integrity.
The following video shows how compliance is achieved by using log file validation:
What is CloudFront
CloudFront is a content delivery network (CDN) service that offers low-latent, fast, secure, and
high-speed static and dynamic content delivery to end-users, globally.
Is integrated with AWS global infrastructure, and works with many services.
Uses edge locations of AWS which are a distributed network of data centers.
Works seamlessly with services like S3, Elastic Load Balancer, EC2 instances, and
AWS Shield for DDoS migration and AWS Lambda.
Advantages of CloudFront
CloudFront is one of the most used services of AWS. It has a huge customer base, and
organizations like King, Prime Video, Rovio, Discovery, Supercell, Bandai Namco use it for its
following iconic features:
The CloudFront network has 166 POP locations across the globe, aiding in superior
performance and high-speed delivery to users.
CloudFront is a highly secure CDN, providing both application and network security at
no extra cost. Also, users have the ability to configure AWS Certificate manager to
manage SSL certificates.
It works with many AWS services, and with Lambda@Edge functions triggered by
CloudFront events, you can take your code closer to the user to improve
responsiveness and serve private content.
It is very useful in video streaming, both live and pre-recorded video.
It is a comparatively less-expensive service, and you are only charged for content
delivered and requested.
Edge locations also known as the POP locations, are situated at almost all strategic locations,
globally.
Therefore, when a user requests content from the origin server which might be an S3 Bucket or
HTTP server, EC2 instance,
1. The request goes through the Edge location, and if the content is already cached there,
it is provided to the requested user immediately.
2. If not cached in the Edge location (first request by first user), the Edge location's (POP
location's) request is routed to the origin. The content is then sent to the Edge location,
and from there it is delivered to the user.
3. The content is cached in the Edge location, and stays there as per the defined TTL.
4. Another user who requests the same object is serviced from the Edge location, with low
latency.
Types of Origins
The previous example showed an S3 Bucket serving content to the CloudFront. However, it also
supports various other services to act as an origin:
Using MediaStore Container/Package Channel : This is used to stream video content
to CloudFront, both on-demand and live streaming. On-demand video is streamed by
using services like AWS Elemental Media Convert to convert media files into
streaming package and is stored in S3.
For live streaming, either a live-event or 24/7 live channel, encoders such as Media live
or AWS Elemental Media Package are used. Also, there are many third-party tools
and servers too that does the encoding, converts the video to deliverable format, and
uses CloudFront for delivery.
For the detailed steps and tasks involved in video streaming for both on-demand and live video
streaming, refer to AWS Documentation.
Types of Origins...
EC2 Server or other HTTP server : In this, the origin is a custom HTTP server, or an
EC2 instance that you manage. An S3 bucket that is configured for website hosting is
also a custom origin. When you configure this HTTP custom server as origin, you
specify the DNS, HTTP, or HTTPS ports and protocol that CloudFront should use to
fetch content.
In this custom HTTP origin server, CloudFront does not support RTMPS and private
content delivery.
CloudFront origin groups : These origin groups are used for failover scenarios when
you need high availability. Use origin failover to designate a primary origin for
CloudFront, and a secondary origin, so that CloudFront automatically switches to the
secondary origin when the primary origin returns specific HTTP status code.
When data is transferred over the internet, there could be many unwanted threats. We must
ensure that the data is not corrupted, and also restrict the content delivered to certain users, or
users in a particular area.
CloudFront provides a set of solutions for this:
Using HTTPS : CloudFront users can utilize HTTPS protocols to request objects. It can
also be used to make HTTPS requests to the origin. This makes the connections
encrypted, and highly secure.
This can be enabled in the viewer protocol policy and origin protocol policy sections of
the dashboards respectively. You can also add your own certificates for other domains.
Using WAF : By using Web Application Firewall (WAF), you can restrict access to
content, based on the request location, IP address, values of query strings, protocol
used, and so on. After you create a web ACL with WAF, update the web distribution of
CloudFront.
WAF is a very useful security service provided by AWS, and is associated with
CloudFront for high-level security.
Using Origin Access Identity (OAI) : CloudFront does not expose the S3 URLs.
However, users may know these URLs from previous S3 URLs. If these S3 links are
used, the CloudFront-signed URLs become obsolete. Therefore, to ensure usage of
CloudFront URLs only, the OAI feature is used.
You change the bucket policy such that only the OAI has permission to access the
objects in the bucket.
Using Field-Level Encryption : Although HTTPS allows you to enable end-to-end
encryption, field-level encryption enables an additional level of security, which protects
certain data throughout system processing. This is used for securing user-sensitive, and
highly confidential information.
AWS SNS is a highly available, secure, fully managed messaging service that helps you
decouple microservices, distributed systems, and serverless architecture.*
SNS can be used to send notifications to end-users through mobile push, SMS, and
email.
SNS helps in filtering notifications, by dropping a filtering logic in the subscriber system,
and message-routing policies in the publisher side. Thus, you receive notifications of
your interest only.
With encryption, you can secure message content, and restrict the publisher and
subscriber.
Refer to the following video to know how to create a task, publish tasks, create subscriptions,
and so on:
SNS Pricing
With SNS free tier, your first million push notifications are free every month.
You pay based on the number of notifications you publish.
SNS currently allows a maximum limit of 256 KB for published messages.
Each 64 KB chunk of published data is billed as 1 request.
Delivery prices vary depending on the end point, and SNS offers SMS services to more
than 200 countries.
Amazon SQS
Amazon Simple Queue Service (SQS) is a web service that enables web service applications
quickly and reliably queue messages, that one component or application generates, to be used
by another component or application.
Standard Queue
Standard Queue is the general type of queue, and is used widely for many applications. It is
categorized by the following:
They provide a loose First In First Out (FIFO) capability that does not guarantee
preserving the order of messages.
These queues are designed to be highly scalable, and have unlimited throughput. This
is a reason for it not being linear.
Messages are delivered at least once, and sometimes more than once.
Is used when throughput between applications is important.
FIFO Queue
First In First Out (FIFO) Queue is a new type of queue service introduced recently. In FIFO,
The queue preserves the exact order in which messages are received.
The queue provide exactly-once processing, which means that each message is
delivered once, and is available until a consumer processes it and deletes it.
The throughput in this queue is limited to 3000 messages per second with batching, and
300 messages per second without batching.
Duplicates introduced into the queue are deleted within a 5-minute de-duplication
interval.
FIFO queue is used for applications where the events order is important.
Course Summary
You have reached the end of the course. Let us summarize what you have learnt thus far:
CloudFormation - Cloud infrastructure is a key process in DevOps provided by AWS.
The advantages and ease of using a Template designer.
CloudWatch - A monitoring service of AWS that keeps track of metrics of services, logs
and events. This is crucial for the maintenance of applications and infrastructure.
CloudTrail- A security and management service of AWS, that tracks all activities in an
account that happens through CLI, Console, API, and so on.
CloudFront - A CDN service that plays a key role in streaming data and delivery of
static website components, ensuring less latency.
SNS - A messaging service of AWS that coordinates and manages the delivery of
messages to subscribing endpoints.
SQS - A pull message queue service that acts as a buffer between components
producing and consuming data.
Advantages and use cases of Standard and FIFO queues.
Conclusion
Amazon is growing larger everyday by adding new services to its portfolio. Existing services are
being upgraded, and new features are being added regularly.
SQS FIFO service is a recent addition, and the expiration time for metrics has been extended
from 2 weeks to 15 months.
This course covers all the important concepts. You can keep a track of changes and updates
at AWS Documentation.
2)Which of the following is used to pass information such as username and password to
a template?
>>paramenters(wrong)
3)Nested stacks in AWS will have many parent stacks, and _______ root stack.
>one
4)Which of the following intrinsic functions returns the output of another stack?
>>Fn :: ImportValue
5)Deletion of stack results in deletion of all resources created by the stack.
>>ture
>>git
---------------------------
>>10years
>>5 mins
>> ture
4)CloudTrail events are saved in event history for a duration of ___ days.
>>90days
5)High resolution metrics can be read and retrieved at ________ intervals.
>>all
>>bucket policy
-----------------------------------
1)The CloudTrail log file integrity validation process validates if logs are altered or not.
>>true
2)Which of the following methods help CloudFront make users request content through
signed URLs?
>>all(wrong)
>>HTTPS request(wrong)
3)CloudFront offers security at both the 4th and 7th layer of the OSI application model.
>>yes
>>ture
5)Which of the following service is used to stream on-demand video for users using
CloudFront?
>>4 days
7)CloudFormation can be integrated with both Chef and Puppet DevOps tools.
>>ture
>>256kb
>>detailed
>>ture
Memory Usage
Push, pull
The number of trails you can have per region are ____.
Orchid
Alarms can be used to send notifications to users, and to perform actions on behalf of
the user.
True
In template schema, _______ is the only mandatory argument that must be defined.
Resources
Amazon Management
AWS CloudFormation is a free service
provision.
- True
Which of the following intrinsic functions
returns the
output of another stack?
- Fn :: ImportValue
Nested stacks in AWS will have many parent
stacks,
and _______ root stack.
- One
In template schema, _______ is the only
mandatory
argument that must be defined.
- Resources
Which of the following is not an IaC tool?
- Git
CloudFormation can be integrated with both
Chef
and Puppet DevOps tools.
- True
Which of the following is used to pass
information
such as username and password to a template?
- Parameters
IaC is a key DevOps practice, and is used in
conjunction with _____________.
- Continuous Delivery
Deletion of stack results in deletion of all
resources
created by the stack.
- True
Deletion of stack results in deletion of all
resources
created by the stack.
- Orchid
Alarms can be used to send notifications to
users, and
to perform actions on behalf of the user.
- True
Which of the following services support detailed
monitoring only?
- ELB
Basic monitoring is done in __________intervals
and is free.
- 5-minute
For high resolution metrics, the data points of
metrics
lesser than 60 seconds are stored for
__________.
- 3 hours
High resolution metrics can be read and retrieved
at ________ intervals.
- All the options
Custom metrics monitoring is always
__________
monitoring.
- Detailed
The expiration time for CloudWatch metrics
is _________.
- 15 months
CloudTrail events are saved in event history for a
duration of ___ days.
- 90
The maximum retention period for CloudWatch
logs is?
- 10 years
Only VPC flow logs can be directly published to
S3.
- True
CloudWatch does not automatically provide
metrics
for _____________.
- Memory usage
Metrics from on-premise servers and
applications can
be collected by using CloudWatch.
- True
The bucket created by CloudTrail has a unique
_________ attached to it.
- Bucket policy
CloudTrail logs are ________ files.
- JSON
Which of the following methods help
CloudFront make
users request content through signed URLs?
- OIA
If an S3-origin is configured as a website
endpoint,
does it support RTMP distribution?
- No
CloudFront becomes futile in which of the
following cases?
- Requests coming from a single place
The number of trails you can have per region are
_____.
-5
Which of the following service is used to stream
on-demand video for users using CloudFront?
- Elemental media convert
The CloudTrail log file integrity validation
process
validates if logs are altered or not.
- True
CloudFront offers security at both the 4th and
7th layer
of the OSI application model.
- True
FINAL
1. Which of the following service is used to
stream
on-demand video for users using CloudFront?
€ Elemental media convert
2. The default retention period of an SQS
message
in a queue is______.
€ 4 days
3. Deletion of stack results in deletion of all
resources
created by the stack.
€ True
4. In template schema, _______ is the only
mandatory
argument that must be defined.
€ Resources
5. The number of trails you can have per
region are ____.
€5
6. CloudTrail events are saved in the event
history for
a duration of _______ days.
€ 90
7. In an AWS design template, the “depends
on” property of a resource is represented by an
_________ color dot.
€ Orchid
8. Alarms can be used to send notifications to
users,
and to perform actions on behalf of the user.
€ True
9. CloudWatch does not automatically provide
metrics for _____________.
€ Memory Usage
10. CloudFormation can be integrated with both
Chef
and Puppet DevOps tools.
€ True
11. For high resolution metrics, the data points
of metrics lesser than 60 seconds are stored
for __________.
€ 3 hours
12. The bucket created by CloudTrail has a
unique
_________ attached to it.
€ Bucket policy
13. The expiration time for CloudWatch metrics
is _________.
€ 15 months
14. CloudFront offers security at both the 4th
and 7th
layer of the OSI application model.
€ True
15. The maximum size of a single published
message is _______.
€ 256 Kb
Last modified: 2 Feb 2020