SECTION 1

Python for the AWS Cloud

© Digital Cloud Training | https://digitalcloud.training

 Python for the AWS
Cloud

© Digital Cloud Training | https://digitalcloud.training

 AWS Account Overview

© Digital Cloud Training | https://digitalcloud.training

 AWS Account Overview

IAM can be used to

AWS Account
create users, groups,
roles and policies
Unique email
address required

AWS IAM

User Group Role Policy

It’s an IAM best practice to

Account Root User create individual users and
to not use the Root account
The Root user
has full control
over the account
© Digital Cloud Training | https://digitalcloud.training
 AWS Account Overview Authentication: IAM
principals authenticate to
IAM using the console,
API, or CLI
AWS Account
AWS Management
Authorization: IAM Console
principals can then
create resources AWS IAM
across AWS Regions
us-west-1 us-east-1 ap-southeast-2

EC2 RDS EC2 RDS EC2 RDS

S3 ALB S3 ALB S3 ALB

All AWS identities and

resources are created
© Digital Cloud Training | https://digitalcloud.training within the AWS account
 Create your AWS Free Tier
Account

© Digital Cloud Training | https://digitalcloud.training

 What you need…

Credit card for setting up the account and

paying any bills

Unique email address for this account Check if you can use a
dynamic alias with an existing
email address
john@gmail.com

john+ACCOUNT-ALIAS-1@gmail.com
john+ACCOUNT-ALIAS-2@gmail.com
AWS account name / alias

Phone to receive an SMS verification code

© Digital Cloud Training | https://digitalcloud.training
 IDEs

© Digital Cloud Training | https://digitalcloud.training

 IDE - Integrated Development Environment
What is an IDE? Local IDEs
• A software app that provides a set of tools for • Installed on a local machine
software development • Requires manual setup and
• A code editor configuration
• Debugger • Limited to local system
• Compiler resources
• And other features that facilitate the • Hardware and OS dependent
development process PyCharm

Cloud-based IDEs
• Accessible from any device with internet
• Scales with project needs (CPU, memory, storage)
• Collaborative features for real-time editing and sharing
• Pre-configured with essential tools
• Integrated with AWS services for seamless deployment

AWS Cloud9

© Digital Cloud Training | https://digitalcloud.training

 SECTION 2

AWS Fundamentals [OPTIONAL]

© Digital Cloud Training | https://digitalcloud.training

 AWS IAM Overview

© Digital Cloud Training | https://digitalcloud.training

 AWS Identity and Access Management (IAM)
IAM Principals must be authenticated to
send requests (with a few exceptions)
AWS Account

RunInstances

User Role Federated Application

AWS IAM User Actions are
EC2
authorized
A principal is a person or application that on AWS
GetBucket
can make a request for an action or resources
Console
operation on an AWS resource
S3

CLI
CreateUser
Identity- Resource-
based policy based policy
API AWS determines whether IAM
to authorize the request
(allow/deny)
© Digital Cloud Training | https://digitalcloud.training
 Setup Individual User
Account

© Digital Cloud Training | https://digitalcloud.training

 Root User vs IAM User

User Login Details Permissions

Root User Email address Full - Unrestricted

Friendly name: John

+
AWS account ID or Alias
IAM User IAM Permissions
Policy

© Digital Cloud Training | https://digitalcloud.training

 Amazon Elastic Compute
Cloud (EC2)

© Digital Cloud Training | https://digitalcloud.training

 Amazon EC2
EC2 instances run
Windows, Linux, or
MacOS
An EC2 instance is a
virtual server

EC2 Instance EC2 Instance EC2 Instance Website

EC2 hosts are managed Windows OS

by AWS
EC2 Instance EC2 Instance EC2 Instance

EC2 Instance EC2 Instance EC2 Instance A selection of instance types

come with varying combinations
EC2 Host Server of CPU, memory, storage and
networking

© Digital Cloud Training | https://digitalcloud.training

 Launching an EC2 Instance
Select an instance type

EC2 Instance A snapshot is a point-in-time

backup of an instance Family Type vCPUs Memory
An AMI defines the (GiB)
configuration of the
General purpose t2.micro 1 1
instance
Compute optimized c5n.large 2 5.25
Amazon Machine Memory optimized r5ad.large 2 16
EBS Snapshot
Image (AMI)
Storage optimized d2.xlarge 4 30.5

GPU instances g2.2xlarge 8 15

You can customize your

Linux Microsoft instance and create a
custom AMI The instance type defines
Windows the hardware profile (and
cost)

Customized AMI
© Digital Cloud Training | https://digitalcloud.training
 Benefits of Amazon EC2
• Elastic computing – easily launch hundreds to thousands of
EC2 instances within minutes
• Complete control – you control the EC2 instances with full
root/administrative access
• Flexible – Choice of instance types, operating systems, and
software packages
• Reliable – EC2 offers very high levels of availability and
instances can be rapidly commissioned and replaced
• Secure – Fully integrated with Amazon VPC and security
features
• Inexpensive – Low cost, pay for what you use

© Digital Cloud Training | https://digitalcloud.training

 Launch EC2 Instances
(Windows + Linux)

© Digital Cloud Training | https://digitalcloud.training

 Amazon EC2 Instance in a Public Subnet
Region
The Internet Gateway enables access
to/from the Internet
EC2 instance is VPC
launched
Availability Zone

Public subnet
Security group

AWS Management
Console EC2 Instance Internet Gateway
EBS Volume
Admin
Data is stored on an EBS volume
(virtual hard drive) A Security Group controls inbound
and outbound traffic Admin connects to EC2 Instance
over the Internet

© Digital Cloud Training | https://digitalcloud.training

 EC2 Instance Connect and
SSH

© Digital Cloud Training | https://digitalcloud.training

 EC2 Instance Connectivity Options

SSH/Putty RDP EC2 Instance Connect Session Manager

All operating systems Windows only All operating systems All operating systems

Terminal / CLI Desktop / visual Terminal / CLI / Terminal / CLI

browser
SSH daemon Remote Desktop SSH daemon SSM agent
Service
Uses instance key pair Requires RDP client Temporary key pair IAM access control

Port 22 must be open Port 3389 must be Port 22 must be open No ports must be open
open
Anyone with key pair can Need user name and IAM access control via IAM access control via
access instance password to login policy policy

© Digital Cloud Training | https://digitalcloud.training

 RDP to Windows Instance

© Digital Cloud Training | https://digitalcloud.training

 Accessing Services – Access
Keys and IAM Roles

© Digital Cloud Training | https://digitalcloud.training

 Access Keys
AWS Cloud
The access key is VPC
associated with an
IAM account Availability Zone

Public subnet
AWS CLI configured
with access keys

S3 Bucket IAM User EC2 Instance

Private subnet

The access key will

use permissions
assigned to the IAM Policy
user

© Digital Cloud Training | https://digitalcloud.training

 Amazon EC2 Instance Profiles (IAM Roles for EC2)
AWS Cloud
VPC

The role is assumed Availability Zone

by the EC2 instance Public subnet
No credentials are
stored on the instance

IAM Role
S3 Bucket EC2 Instance

Private subnet

Policy
The policy determines
the access permissions

© Digital Cloud Training | https://digitalcloud.training

 Access Keys and IAM Roles

© Digital Cloud Training | https://digitalcloud.training

 Amazon VPC Overview

© Digital Cloud Training | https://digitalcloud.training

 Amazon Virtual Private Cloud (VPC)
A VPC is a logically isolated
portion of the AWS cloud within a
region
Region

VPC
The VPC router takes
care of routing within the
Availability Zone
VPC and outside of the
Public subnet VPC
Subnets are Main Route Table
created within
AZs Destination Target
EC2 Instance
10.0.0.0/16 Local
0.0.0.0/0 igw-id
Availability Zone
Router Internet
Private subnet gateway
The route table is used
to configure the VPC
You can launch EC2 instances into router
EC2 Instance
your VPC subnets
An Internet Gateway is
attached to a VPC and used
to connect to the Internet

© Digital Cloud Training | https://digitalcloud.training

 Amazon VPC
Each VPC has a different block of CIDR stands for Classless
Region IP addresses Interdomain Routing

VPC VPC
CIDR 10.0.0.0/16 CIDR 10.1.0.0/16

Availability Zone Availability Zone Availability Zone Availability Zone

Public subnet Public subnet Public subnet Public subnet

10.0.0.0/24 10.0.1.0/24 10.1.0.0/24 10.1.2.0/24

Private subnet Private subnet Private subnet Private subnet

10.0.2.0/24 10.0.3.0/24 10.1.3.0/24 10.1.4.0/24

Each subnet has a block of IP

addresses from the CIDR
block
You can create multiple VPCs
within each region

© Digital Cloud Training | https://digitalcloud.training

 Amazon VPC Components
VPC Component What it is
Virtual Private Cloud A logically isolated virtual network in the AWS cloud
(VPC)
Subnet A segment of a VPC’s IP address range where you can place groups

of isolated resources
Internet Gateway/Egress- The Amazon VPC side of a connection to the public Internet for IPv4/IPv6
only Internet Gateway
Router Routers interconnect subnets and direct traffic between Internet gateways, virtual private
gateways, NAT gateways, and subnets
Peering Connection Direct connection between two VPCs
VPC Endpoints Private connection to public AWS services
NAT Instance Enables Internet access for EC2 instances in private subnets managed by you)
NAT Gateway Enables Internet access for EC2 instances in private subnets (managed by AWS)
Virtual Private Gateway The Amazon VPC side of a Virtual Private Network (VPN) connection
Customer Gateway Customer side of a VPN connection
AWS Direct Connect High speed, high bandwidth, private network connection from customer to aws
Security Group Instance-level firewall
Network ACL Subnet-level firewall

© Digital Cloud Training | https://digitalcloud.training

 Amazon VPC Core Knowledge

• A virtual private cloud (VPC) is a virtual network dedicated to your AWS

account

• Analogous to having your own data center inside AWS

• It is logically isolated from other virtual networks in the AWS Cloud

• Provides complete control over the virtual networking environment

including selection of IP ranges, creation of subnets, and configuration
of route tables and gateways

• You can launch your AWS resources, such as Amazon EC2 instances,
into your VPC

© Digital Cloud Training | https://digitalcloud.training

 Amazon VPC Core Knowledge

• When you create a VPC, you must specify a range of IPv4

addresses for the VPC in the form of a Classless Inter-Domain
Routing (CIDR) block; for example, 10.0.0.0/16
• A VPC spans all the Availability Zones in the region
• You have full control over who has access to the AWS resources
inside your VPC
• By default you can create up to 5 VPCs per region
• A default VPC is created in each region with a subnet in each AZ

© Digital Cloud Training | https://digitalcloud.training

 Create a Custom VPC

© Digital Cloud Training | https://digitalcloud.training

 Create a Custom VPC

Region

VPC CIDR 10.0.0.0/16

Main Route Table
Availability Zone
Destination Target
Private subnet Public subnet
10.0.0.0/16 Local
10.0.3.0/24 10.0.1.0/24
0.0.0.0/0 igw-id

Availability Zone
Internet
Private Route table Main Route table Private Route Table
Private subnet Public subnet gateway
10.0.4.0/24 10.0.2.0/24 Destination Target
10.0.0.0/16 Local

© Digital Cloud Training | https://digitalcloud.training

 Amazon S3 Overview

© Digital Cloud Training | https://digitalcloud.training

 Amazon Simple Storage Service (S3)
A bucket is a
container for
objects
S3 Bucket

An object is a file
you upload
Object

Accessing objects in a bucket:

Object
https://bucket.s3.aws-region.amazonaws.com/key
You can store millions
of objects in a bucket https://s3.aws-region.amazonaws.com/bucket/key
Object

The HTTP protocol is used with a REST API

(e.g. GET, PUT, POST, SELECT, DELETE)

© Digital Cloud Training | https://digitalcloud.training

 Amazon Simple Storage Service (S3)
• You can store any type of file in S3
• Files can be anywhere from 0 bytes to 5 TB
• There is unlimited storage available
• S3 is a universal namespace so bucket names must be
unique globally
• However, you create your buckets within a REGION
• It is a best practice to create buckets in regions that
are physically closest to your users to reduce latency
• There is no hierarchy for objects within a bucket
• Delivers strong read-after-write consistency

© Digital Cloud Training | https://digitalcloud.training

 Buckets, Folders, and Objects

https://mybucket.s3-website-us-west-1.amazonaws.com
The object name is the Key the
mybucket data is the Value

https://mybucket.s3.us-west-1.amazonaws.com/Orders.xlsx
Orders.xlsx

https://mybucket.s3.us-west-1.amazonaws.com/Project+Alpha/Sales+Proposal.docx

Sales Proposal.docx

https://mybucket.s3.us-west-1.amazonaws.com/Project+Alpha/Project+Brief.docx

Project Brief.docx

A Folder is a shared prefix for

grouping objects

© Digital Cloud Training | https://digitalcloud.training

 Buckets, Folders, and Objects
• Folders can be created within folders
• Buckets cannot be created within other buckets
• An objects consists of:
• Key (the name of the object)
• Version ID
• Value (actual data)
• Metadata
• Subresources
• Access control information

© Digital Cloud Training | https://digitalcloud.training

 Accessing Amazon S3
Amazon S3 is a public EC2 instances connect using private
service addresses

Private Connection
Amazon S3
User browses with a VPC
standard web browser S3 Gateway Endpoint
Public subnet

Internet EC2 Instance

Internet Client gateway
Public Internet
EC2 instances Private subnet
connect using public
addresses

EC2 Instance

Application
Application uses REST API
programmatically

© Digital Cloud Training | https://digitalcloud.training

 Amazon S3 Buckets
and Objects

© Digital Cloud Training | https://digitalcloud.training

 SECTION 3

Python Fundamentals

© Digital Cloud Training | https://digitalcloud.training

 Python Overview

© Digital Cloud Training | https://digitalcloud.training

 Python Overview

5X-10X

3x-5X

Example use cases

© Digital Cloud Training | https://digitalcloud.training

 Commenting Code

© Digital Cloud Training | https://digitalcloud.training

 Commenting Code

Comment for
your future self

Comment for others

© Digital Cloud Training | https://digitalcloud.training

 Print Function

© Digital Cloud Training | https://digitalcloud.training

 Print Function
• Print outputs data to the console
• Useful for:

© Digital Cloud Training | https://digitalcloud.training

 Variables

© Digital Cloud Training | https://digitalcloud.training

 Variables

Containers Transfer of Code Maintenance

information

© Digital Cloud Training | https://digitalcloud.training

 Data Types
int str

float bool

© Digital Cloud Training | https://digitalcloud.training

 Data Types

Integer String
Represent whole numbers. Strings are sequences of
character data.

Floating-Point Number Boolean

Float numbers are specified One of two values, True or False.
with a decimal point.

© Digital Cloud Training | https://digitalcloud.training

 Operators

© Digital Cloud Training | https://digitalcloud.training

 Operators

Assignment Arithmetic
Can be used to
Operators used to assign values to variables. concatenate strings.

Comparison
Compare values and return if the comparison is true or Logical
false. Can be used to combine conditional
statements.

Membership
Can be used to test
whether a value is a
member of a
sequence, like strings,
lists, or tuples.

© Digital Cloud Training | https://digitalcloud.training

 Conditional Statements

© Digital Cloud Training | https://digitalcloud.training

 Conditional Statements

• A piece of code that executes • In a Python program, the if statement is how

instructions based on a particular state you perform this sort of decision-making.
• Useful for:
• Controlling the flow of a program
• Executing different code based on
different conditions

© Digital Cloud Training | https://digitalcloud.training

 Functions

© Digital Cloud Training | https://digitalcloud.training

 Functions

What is a Function?
• A block of organized and reusable code that
performs a specific task
• Functions help break a program into smaller,
more manageable pieces
• Code reuse and modularity
• Essential for code that is:
• Efficient
• Maintainable
• Scalable
• Built-in & user-defined functions
• Think of functions as factories

Function
Inputs Output
© Digital Cloud Training | https://digitalcloud.training
 Lists

© Digital Cloud Training | https://digitalcloud.training

 Lists

What are lists and why use them?

• An ordered collection of items
• Especially useful when you need to work with a collection of related items
• Containers that help you store and manage multiple pieces of data that can be of different
data types
• Think of a list as a grocery shopping list

How to work with lists:

• Creating a list
• Accessing list elements
• Modifying list elements

© Digital Cloud Training | https://digitalcloud.training

 Dictionaries

© Digital Cloud Training | https://digitalcloud.training

 Dictionaries

What are dictionaries and why use them?

• An unordered collection of key-value pairs
• Think of a python dictionary as a real-world dictionary
• Each key is unique

How to work with dictionaries: • Creating a dictionary

• Accessing dictionary values
• Modifying dictionary values
• Adding and removing key-value pairs

• Dictionary methods
• keys()
• values()
© Digital Cloud Training | https://digitalcloud.training
 Loops

© Digital Cloud Training | https://digitalcloud.training

 Loops
Definite iteration - The number of repetitions is specified explicitly in advance

Indefinite iteration - The number of repetitions isn’t specified explicitly in

advance.

© Digital Cloud Training | https://digitalcloud.training

 Debugging

© Digital Cloud Training | https://digitalcloud.training

 Debugging

Debugging is the practice of identifying and fixing errors

• Inspecting code
• Print statements
• Creating and viewing logs
• Debugging in IDE’s
• Setting breakpoints
• Stepping through code

© Digital Cloud Training | https://digitalcloud.training

 SECTION 4

AWS Services Using Python

© Digital Cloud Training | https://digitalcloud.training

 AWS Boto3 SDK

© Digital Cloud Training | https://digitalcloud.training

 AWS Boto3 SDK
• Boto3 contains easy-to-use APIs to many AWS
services, including:

An SDK (Software S3 EC2 DynamoDB CloudWatch

Development Kit) is a set
of preconfigured tools and
libraries automation

Boto3 allows us to write

Python code that interacts
with AWS services

create_fleet()

filter_log_events()

upload_file() put_item()

© Digital Cloud Training | https://digitalcloud.training

 AWS Boto3 SDK Cont.

Resource Client
• Higher-level, object-oriented API • Use when you need fine-grained control
• Represents AWS service objects (e.g., S3 and access to all available operations
bucket, EC2 instance) • Lower-level, service-oriented API
• Allows actions directly on objects • Provides access to all AWS service actions
• Use when you prefer a more Pythonic and
concise interface

© Digital Cloud Training | https://digitalcloud.training

 S3: CRUD Operations and Boto3

© Digital Cloud Training | https://digitalcloud.training

 S3: CRUD Operations and Boto3
What is CRUD?
• Four basic operations that can be
performed on data in a database
or a data service like S3

© Digital Cloud Training | https://digitalcloud.training

 EC2: Basics, Launching, and Management

© Digital Cloud Training | https://digitalcloud.training

 EC2: Basics and Boto3

EC2 (Elastic Compute Cloud)

• A core component of the AWS platform
• Resizable, scalable compute capacity in the cloud
• Simplifies the set up and scaling of virtual servers
• Works in conjunction with Amazon VPC to provide security
and robust networking functionality

Using Boto3 with EC2

• Automates and streamline tasks and operations
• You can launch and manage EC2 instances using Python code
• Resource and Client
• Creates one or more new instances
• Flexible because of many parameters
• run_instances() Client Method
• create_instances() Resource Method

© Digital Cloud Training | https://digitalcloud.training

 EC2: Launching Instances

• If you don’t specify a subnet ID, a

default subnet from default VPC
will be chosen
• If you don’t specify a private IPV4
address, AWS will choose one from
the IPv4 range of your subnet
• If you don’t specify a security group
ID, AWS uses the default security
group
• You can create a launch template,
which is a resource that contains
the parameters to launch an
instance
• An instance is ready for you to use
when it’s in the running state

© Digital Cloud Training | https://digitalcloud.training

 EC2: Managing Instances

• Using a resource, reference the

instance by passing the instance ID
• Call operations on an instance
using an action
• attach_volume
• create_tags
• delete_tags
• monitor
• reboot
• reload
• start
• stop
• terminate

© Digital Cloud Training | https://digitalcloud.training

 VPC: Basics, Creation, and Management

© Digital Cloud Training | https://digitalcloud.training

 VPC: Introduction and Basics

VPC (Virtual Private Cloud) AWS Cloud

• Scalable and highly configurable AWS service

• Provides networking functionality MY VPC
• A customizable logically isolated section of the
AWS Cloud
• Including selection of your own IP address range, EC2

creation of subnets, and configuration of route

tables and network gateways
• Into which you can launch AWS resources
Boto3
• VPC and EC2 Relation - logical grouping from an
API design perspective
• EC2 Resource or client

© Digital Cloud Training | https://digitalcloud.training

 VPC: Creation and Management

• Previous examples had their own client or resource • You can create multiple VPCs
• VPC is created with EC2 client or resource • Helpful for managing separate environments
• Methods used with VPCs include: • Creating and tagging a VPC
create_vpc(CidrBlock='10.0.0.0/16')
create_tags(**kwargs)

AWS Cloud

Create Tags DEV TEST

Create
Create Internet
Subnet Gateway MY VPC PROD
Create Route 10.0.0.0/16
Create VPC Create Route
Table

EC2 Resource/Client

© Digital Cloud Training | https://digitalcloud.training

 RDS: Basics, Aurora Serverless, DB Management

© Digital Cloud Training | https://digitalcloud.training

 Amazon RDS

RDS - Allows you to run isolated database Security and Access

environments in the cloud RDS provides multiple layers of security, including:
• network isolation
• encryption at rest and in transit
Supported Databases • Fine-grained access control using (IAM)

MySQL PostgreS Microsoft MariaD Amazon Aurora

QL SQL Server B

Managed Service
Amazon RDS takes care of the tedious database Use Cases
administration tasks such as:
• Backups
• Software Patching
• Monitoring E-commerce platforms Customer relationship
• Scaling management apps

© Digital Cloud Training | https://digitalcloud.training

 RDS: Amazon Aurora Serverless
An on-demand version of Amazon Aurora with auto-scaling • Aurora Serverless only works with PostgreSQL and
capabilities. MySQL
Managed via RDS which handles provisioning,
patching, backup, restore, and other tasks
MySQL PostgreS
Removes complexity of QL
• The required computing capacity is configured via
provisioning and managing
Aurora Capacity Units (ACUs), which are a
database capacity
combination of virtual CPUs and memory
Aurora • 1 ACU has approximately 2 GiB of memory
Seamlessly scale compute and
Serverless
memory capacity
ACU ACU ACU
Highly available
Built on distributed, fault-tolerant, self-healing
Aurora storage with 6-way replication to Use Case
protect against data loss.
Cost-effective
Pay only for the resources iRobot
you consume

© Digital Cloud Training | https://digitalcloud.training

 RDS: Aurora Serverless V1 and V2
Aurora Serverless V1 Boto3
• Suitable for apps without frequent traffic • RDS Client
• Can cut capacity to zero ACUs during idle periods • create_db_cluster()
• Scale up latency (cold start) • describe_db_clusters()
• Scales by doubling ACUs (1, 2, 4, 8, 16 … 256)
• Best for test and dev environments
• Console Query Editor
• About $0.06 an hour per ACU Aurora Serverless • modify_db_cluster()
V1 • delete_db_cluster()

Aurora Serverless V2
• Supports all features of Aurora
• Suitable for production environments
• Instant scaling to hundreds-of-thousands of transactions per
second
• Scales in increments of 0.5 ACUs
• Cannot cut capacity to zero
• Does not support the query editor
• About $0.12 an hour per ACU Aurora Serverless
V2
© Digital Cloud Training | https://digitalcloud.training
 SECTION 5

Automating AWS Tasks With Lambda

Functions

© Digital Cloud Training | https://digitalcloud.training

 AWS Lambda

© Digital Cloud Training | https://digitalcloud.training

 Amazon Lambda Functions
A serverless compute service that automatically runs your code Supported Languages
in response to events and manages the underlying compute
resources for you.

Common Use Cases

Real-time file
Resize
processing
images

Scales in
Highly available
response to
infrastructure Only pay for the Parse/filter
each trigger
compute time
and number of
requests that you
© Digital Cloud Training | https://digitalcloud.training consume
 Lambda & Python
Food Delivery App Example
Why Lambda is important
• More efficient code execution Pricing Lambda
• Lambda's serverless nature lets developers focus • Compute and request charges • 3 million requests per month
• Free Tier Usage includes • 120 milliseconds per request
more on their code and less on infrastructure
• 400,000 GB-seconds • Configured with 1.5 GB of
management • 1 million requests per month memory, on an x86 based
• Quicker deployment times • x86 based processor price: processor
$0.0000166667 USD per GB-second
Pandas Pillow
• Requests: $0.20 USD per 1M
Why Python?
• Readability Compute Charges:
• Extensive libraries 3 million * 120 milliseconds = 360,0000 seconds
• Python simplicity + power of Lambda requestsseconds * 1.5 GB = 540,000 GB-s
360,000
• Robust, scalable serverless applications 540,000 GB-s - 400,000 GB-s (Free Tier) = 140,000 GB-s
140,000 GB-s * $0.0000166667 = $2.33
Example of services used with
Lambda in this course Request Charges: Total cost to run Lambda:
3M requests – 1M (free tier) = 2M $2.73 USD per month
2M x $0.2/M = $0.40

© Digital Cloud Training | https://digitalcloud.training

 Automating EC2 with Lambda

© Digital Cloud Training | https://digitalcloud.training

 Automating EC2 with Lambda
Why Automate EC2 with Lambda? How Automate EC2 with Lambda?

EC2 instances can trigger events

Efficiency • Instance launching
• Instance scheduling • Terminating
• Auto-scaling • State changes
• System events responses

Lambda can respond to these events

Scalability Error Reduction
• Stops an EC2 instance
when it's been idle for
too long

• Create backups at regular

intervals to prevent any
data loss

Cost-
Effectiveness
© Digital Cloud Training | https://digitalcloud.training
 Automating EC2 Backups with Lambda
Scenario
Your EC2 instance contains crucial data that would be
very costly to lose.

Solution
Best practice: Create backups at regular intervals to
prevent any data loss in case of instance failures or
errors.

Setting the schedule

Use EventBridge Scheduler to trigger the lambda
function at regular intervals. EventBridge

© Digital Cloud Training | https://digitalcloud.training

 Automating EC2 with Lambda HOL

© Digital Cloud Training | https://digitalcloud.training

 Automating EC2 Backups with Lambda HOL
AWS Cloud

Task:
Create a lambda function which creates a snapshot of our
EC2 instance at regular intervals.
EventBridge
Scheduler
Prerequisites:
• Create ‘LambdaEC2DailySnapshot’ lambda function
• Download lambda function into your Cloud9
environment
• Create or have an EC2 instance ready to use

LambdaEC2DailySnapshot

IAM
Permissions

© Digital Cloud Training | https://digitalcloud.training

 Automating S3 with Lambda

© Digital Cloud Training | https://digitalcloud.training

 Automating S3 with Lambda: Use Cases
Why Automate S3? Real-World Applications
Log Analysis
File Transformation Archiving Specific Data

Efficiency

Common Use Cases

Processing Alerts Clean-up

© Digital Cloud Training | https://digitalcloud.training

 Automating S3: Key Concepts

Event-Driven Architecture and Programming Lambda and S3 Interaction

• In Python, we often use event-driven programming to
respond to real-time events
• AWS expands this concept with an event-driven
architecture Example of real-time
• Python plays a key role here, as it's one of the primary data validation
languages used for writing Lambda functions.
• Events in S3 can trigger responses in Lambda

S3 Events and Event Notifications

• S3 supports a variety of events
• We can set S3 to send an event notification whenever
these events occur, which can trigger a Lambda
function.

• Delete • Put
• DeleteMarkerCreated • Post
• Copy
© Digital Cloud Training | https://digitalcloud.training
 Automating S3 with Lambda: Real-time Data
Validation HOL

© Digital Cloud Training | https://digitalcloud.training

 Real-time Data Validation HOL: Automating S3 with Lambda
Overview:
You will set up a Lambda function which will be triggered whenever
AWS Cloud
a new CSV file is uploaded to an S3 bucket. It will then validate the
file and if any discrepancies are found, it will move the file to an S3 Permissions
'error' bucket.

Prerequisites: BillingBucketParser
1. Boilerplate (default) Lambda function ‘BillingBucketParser’
downloaded in Cloud9 environment.
2. Two S3 buckets with default settings.
a) ‘dct-billing-x’ for uploading csv files.
b) ‘dct-billing-errors-x’ for error files.
3. Download the provided csv test files.
dct-billing-x dct-billing-errors-x
Tasks:
1. Add necessary permissions for the Lambda role.
2. Write the Lambda Python code.
3. Set up the S3 trigger.

© Digital Cloud Training | https://digitalcloud.training

 Automating RDS with Lambda

© Digital Cloud Training | https://digitalcloud.training

 Automating RDS with Lambda
RDS & Lambda - A Powerful Combination
Automate • Scheduled SQL queries • Lambda can help maintain data
Data Alteration
DB Tasks • Backups and snapshots cleanliness and integrity
• Data archiving and purging activities based on • Pre-processing or transforming data
business rules before it's inserted into RDS
• Post-processing data after it's retrieved
from RDS

Scalability • Lambda and RDS can scale independently

• Lambda can handle more requests without
• Lambda can handle business logic of
overloading the RDS instance Microservices &
various microservices, working with RDS
• More granular and cost-effective scaling Serverless Apps to:
• Implement efficient decoupling of
compute and storage services
Event-Driven DB • Sending alerts based on specific DB • Respond to HTTP requests through
conditions Amazon API Gateway and perform CRUD
Operations • Custom failover logic for high availability operations on RDS
scenarios
• Auto-scaling instances based on usage
patterns

© Digital Cloud Training | https://digitalcloud.training

 Automating RDS with Lambda

Data Transformation Automating ETL Processes:

• Example: Normalizing currencies in financial data before storing
• Lambda can automate this ETL process
in RDS
• Upon the arrival of new data Lambda can read, transform
• AWS Lambda can perform these transformations as part of the
and load the data into an RDS instance
ETL (Extract, Transform, Load) process

© Digital Cloud Training | https://digitalcloud.training

 Automating RDS with Lambda HOL

© Digital Cloud Training | https://digitalcloud.training

 Automating RDS with Lambda HOL

Overview:
Configure a Lambda function that’s triggered whenever a new billing CSV file is uploaded Tasks:
to an S3 bucket. The function will read the file, convert any billing amount not in USD 1.Write and test the Lambda Python code in Cloud9 IDE.
into USD, and insert the records into an Aurora Serverless V1 database. 2.Add necessary permissions to the Lambda execution role.
1. Access to S3 to read the input CSV file.
Prerequisites: 2. Access to RDS to write data to Aurora Serverless.
1. Create an Aurora Serverless V1 database cluster 3. Access to Secrets Manager to retrieve database
a) Create the DB cluster we generated in the “RDS Hands-on Lab” from the credentials.
“AWS Services using Python” section of this course 3.Set up the S3 bucket trigger to invoke the Lambda function
2. Boilerplate (default) Python Lambda function 'BillingDataConversionAndIngestion’ whenever a new CSV file is uploaded.
downloaded into Cloud9 environment. 4.Test the Lambda function by uploading a CSV file to the 'dct-
3. An S3 bucket with default settings for uploading CSV files: 'dct-billing-processed-x’. billing-processed-x' S3 bucket and verify the data insertion in
4. Download the provided CSV test files. Aurora Serverless database.

AWS Cloud

Permissions

BillingDataConversionAndIngestion

CSV Billing File dct-billing-processed-x Aurora Serverless V1

© Digital Cloud Training | https://digitalcloud.training

 Automating VPC Operations with Lambda

© Digital Cloud Training | https://digitalcloud.training

 Automating VPC Operations with Lambda
Why combine Lambda with VPC
Operations?
Event-Driven Improving • Automate the enforcement of security
• Update network configurations in
Network response to changes in other AWS Security rules and standards
resources • Respond to potential security threats or
Operations
• Automatically react to changes in unusual network activities
network traffic or utilization

Automate Network • Automatically adjust network Access

Tasks Control Lists (ACLs)
• Create and delete subnets based on
demand or schedules
• Configuring security group rules to
adapt to changes in your environment
• Ensure optimal usage of our resources

© Digital Cloud Training | https://digitalcloud.training

 Automating Elastic IP Management with AWS Lambda

Elastic IPs (EIPs)

• Static, public IPv4 addresses that you can allocate to your account AWS Cloud
• To use an Elastic IP address, you first allocate one to your
account, and then associate it with a network interface or an
instance
• You can use an EIP to mask the failure of an instance or software
by rapidly remapping the address to another instance in your
account
• While EIPs are free when they are associated with an instance,
there is a small hourly charge when an EIP is not being used

Example Scenario:
• You have several Elastic IPs that are not currently associated with
an instance and result in unnecessary costs Lambda
• A Lambda function
• checking for any unassociated Elastic Ips EventBridge
• Releasing them to save costs
• Triggered by EventBridge every day
• This demonstrates how we can automate and streamline VPC
operations and tasks using Lambda

© Digital Cloud Training | https://digitalcloud.training

 Applying Lambda & VPC: Real-World Scenarios
Real Use-Cases

Scalability & Efficiency DevOps

Lambda can be set to automatically Teams can use Lambda to automatically
adjust the number of instances within a assign or unassign Elastic IPs to instances
VPC based on the current load. This when they are launched or terminated,
ensures you're using resources optimally. ensuring optimal resource utilization and
• Peak usage times: launch additional cost-efficiency.
instances
• Off-peak time: terminate unnecessary Summary
instances
• How Lambda can automate VPC operations
• Keeping costs in check
• Automate tasks
• Increase efficiency
Security Lambda can be used to automate security • Improve security
checks, such as verifying that all Security • We dove into a practical example of managing Elastic IPs
Groups in a VPC follow organizational automatically
guidelines. • The combination of Lambda and VPC opens a host of
If a security group is found not in compliance, possibilities for automating tasks within your environment
Lambda can automatically send notifications and allowing your teams to focus on what matters most.
or even adjust the rules to match the
standards.

© Digital Cloud Training | https://digitalcloud.training

 Automating VPC with Lambda HOL

© Digital Cloud Training | https://digitalcloud.training

 Automating VPC Operations with Lambda HOL

Scenario:
Your company regularly uses Elastic IPs for EC2 instances but at times some of
these IPs are left unassociated and result in unnecessary costs. Your task is to
automate the process of checking for these unassociated Elastic IPs and releasing AWS Cloud
them.
Permissions
Overview:
We’ll create a Lambda function that is triggered daily by EventBridge. The Lambda
function will check for any unassociated Elastic IP addresses in our VPC and release
them. This way, we ensure we don't incur unnecessary costs from unused Elastic EventBridge ManageEIPs
IPs.

Prerequisites:
1. Boilerplate (default) Python Lambda function 'ManageEIPs’ downloaded into
Cloud9 environment.
EIP 1 EIP 3
Tasks:
1. Create three EIPs and one EC2 instance. Associate one of the EIPs with the EIP 2
EC2 instance
2. Write and test the Lambda Python code in Cloud9 IDE.
3. Upload the local Cloud9 ‘ManageEIP’ version of our function to Lambda
4. Add the necessary EC2 permissions to the Lambda execution role.
5. Set up the EventBridge trigger and test it

© Digital Cloud Training | https://digitalcloud.training

 SECTION 6

Advanced AWS and Python Topics

© Digital Cloud Training | https://digitalcloud.training

 Understanding and Using SNS, SQS, and SES

© Digital Cloud Training | https://digitalcloud.training

 Amazon Simple Notification Service (SNS)

What is SNS? AWS Cloud

• A fully managed messaging service for
communication
• Application-to-application (A2A)
• Application-to-person (A2P)
• Supports the publishing of messages to User Places Order
DispatchItem
multiple subscribing endpoints or clients Topic
• SNS Topic
• A logical access point and
communication channel

Order Tracking
System
Warehouse Mgmt.
System

© Digital Cloud Training | https://digitalcloud.training

 Amazon Simple Queue Service (SQS)

What is SQS? • A fully managed message queuing service that enables you AWS Cloud
to decouple and scale DispatchItem
• Microservices Topic
• Distributed systems
• Serverless applications
• Eliminates the complexity and overhead associated with
managing and operating message-oriented middleware
• Stores messages on multiple servers for redundancy and to
OrderProcessingQueue
ensure message durability

Warehouse Mgmt.
System

© Digital Cloud Training | https://digitalcloud.training

 Amazon Simple Email Service (SES)

What is SES? AWS Cloud

• An emailing service designed to send
• Marketing
• Notifications
OrderProcessingQueue
• Transactional emails
• Allows us to easily automate email notifications

OrderConfirmationProcessor

SES Item Sent Email

© Digital Cloud Training | https://digitalcloud.training

 SNS and SQS HOL

© Digital Cloud Training | https://digitalcloud.training

 SNS and SQS Hands-On Lesson

Scenario:
Your company has a system for validating CSV billing files stored in an S3 bucket for North American customers. In order to get real-time international tax
data, you need to make a 3rd party API call. Occasionally, errors might occur during calls to a 3rd party service, which will prevent the billing file from being
processed correctly. Your task is to automate the process of handling these errors, notifying the relevant employees, and ensuring error-related
information is stored for further investigation.

Overview:
We’ll modify the finalized version of the “Automating S3 with Lambda: Real-time Data Validation HOL” by creating an additional lambda
‘RetryBillingParser’ function, an SNS topic, and an SQS queue. If there is an error in the ‘BillingBucketParser’ function’s mock API call to a 3rd party service
‘BillingBucketParser’ publishes to the SNS topic. An email is sent will be sent out and a subscribed SQS queue triggers the ‘RetryBillingParser’ function. This
function re-attempts the data validation.

Prerequisites:
• Completed Automating S3 with Lambda: Real-time Data Validation HOL
• Boilerplate Python Lambda functions 'RetryBillingParser’
• S3 bucket ‘dct-billing-processed’

© Digital Cloud Training | https://digitalcloud.training

 SNS and SQS Hands-On Lesson

Existing S3 Real-time Data Validation HOL

AWS Cloud

Move file to
error bucket
dct-billing-errors
Yes

Parse Billing
Data Error
Files

dct-billing BillingBucketParser No

Do Nothing

© Digital Cloud Training | https://digitalcloud.training

 SNS and SQS Hands-On Lesson

AWS Cloud

No Move file to
SQS Queue error bucket
SNS
RetryBillingParser dct-billing-
Topic
errors
Yes

Successful API Call? Parse Billing

Data Error
Yes Files

dct-billing BillingBucketParser
No

Move file to
processed
bucket dct-billing-
processed

Mock 3rd
Party API Call API

© Digital Cloud Training | https://digitalcloud.training

 Understanding & Using Glue and EMR with Python

© Digital Cloud Training | https://digitalcloud.training

 Using AWS Glue
AWS Cloud
What is AWS Glue?
• Glue is a fully managed ETL (Extract, Transform, Load) service
that simplifies the process of discovering and cataloging your
data across various data stores
• At the heart of Glue is the Data Catalog, a centralized
metadata repository

• Glue Crawlers connect to your source data, extract metadata and create table Table x
definitions in the Data Catalog Table y
• Securing access to your data is crucial. Lake Formation simplifies the process of setting Table z
up, securing, and managing data Crawler Glue Data
• Lake Formation ensures that only those users and services (such as AWS EMR) with Catalog
granted permissions can access the data in your Data Catalog

Scenario:
• Consider a company that holds vast amounts of data across several S3 buckets. By
using Glue Crawlers, they can automate the process of extracting, transforming, and
loading this data for analytics
Lake Formation
• With Lake Formation, they can secure their Data Catalog, making sure only authorized User
users have access to specific data

© Digital Cloud Training | https://digitalcloud.training

 AWS Elastic MapReduce (EMR)
What is EMR? A big data platform that enables businesses to
process large datasets quickly and cost-effectively
using popular frameworks such as Apache Hadoop
and Apache Spark.

Some essential features of EMR include:

• Scalability
• Flexibility
• Security

Apache Spark is an open-source, distributed

computing system used for big data processing and
analytics.
• PySpark is the Python library for Spark which allows for high-
throughput data processing
• PySpark and EMR form a powerful toolset for big data analytics
• Use cases include log analysis, data transformations, financial
analysis, machine learning, etc.…
• We can use Boto3 to create an EMR cluster, EMR Steps
add steps (tasks) to the cluster, check the
status of the steps, and terminate clusters

© Digital Cloud Training | https://digitalcloud.training

 Glue and EMR Hands-On Lesson

© Digital Cloud Training | https://digitalcloud.training

 Glue and EMR Hands-On Lesson

Scenario: Prerequisites:
Your company processes a multitude of transaction files daily which detail Download the provided files: S3 buckets:
billing, products sold, and their quantities. This data is stored in different • emr_software_settings.json • dct-billing-processed
CSV files within S3. There's a need to consolidate this data, analyze it, and • billing_data_dairy_07_2023.csv • dct-billing-data-lake-x
derive insights like the gross profit. You are tasked with automating the data • units_sold_07_2023.csv • units-sold/
processing and analytics pipeline using AWS Glue and EMR. • production_costs_07_2023.csv • production-costs/
• reports/
• pyspark_script/
Overview:
1. Store the raw CSV data in S3 buckets
2. Use Glue Crawlers to catalog this data
3. Spin up an EMR cluster, which we’ll use to run a PySpark script
a) This script will read the data from the Glue catalog and
determine the gross profit for each product sold
b) Store gross profit results back into S3 for reporting units_sold_07_2023.csv

production_cost_07_2023.csv

billing_data_dairy_07_2023.csv

© Digital Cloud Training | https://digitalcloud.training

 Glue and EMR Hands-On Lesson

AWS Cloud

Crawler Billing
Crawlers
Populates Units Sold
parse csv data
Data Catalog Production Costs
dct-billing-processed
Glue Crawlers Glue Data Catalog

dct-billing-data-lake Generate
Gross Profit Process Data
units-sold/ reports/
Report
production-costs/ pyspark-script/ EMR Cluster

IAM Lake
Formation

© Digital Cloud Training | https://digitalcloud.training

 SNS and SQS Hands-On Lesson

AWS Cloud

No Move file to
SQS Queue error bucket
SNS
RetryBillingParser dct-billing-
Topic
errors
Yes

Successful API Call? Parse Billing

Data Error
Yes Files

dct-billing BillingBucketParser
No

Move file to
processed
bucket dct-billing-
processed

Mock 3rd
Party API Call API

© Digital Cloud Training | https://digitalcloud.training

 Understanding & Using Glue and EMR with Python

© Digital Cloud Training | https://digitalcloud.training

 Using AWS Glue
AWS Cloud
What is AWS Glue?
• Glue is a fully managed ETL (Extract, Transform, Load) service
that simplifies the process of discovering and cataloging your
data across various data stores
• At the heart of Glue is the Data Catalog, a centralized
metadata repository

• Glue Crawlers connect to your source data, extract metadata and create table Table x
definitions in the Data Catalog Table y
• Securing access to your data is crucial. Lake Formation simplifies the process of setting Table z
up, securing, and managing data Crawler Glue Data
• Lake Formation ensures that only those users and services (such as AWS EMR) with Catalog
granted permissions can access the data in your Data Catalog

Scenario:
• Consider a company that holds vast amounts of data across several S3 buckets. By
using Glue Crawlers, they can automate the process of extracting, transforming, and
loading this data for analytics
Lake Formation
• With Lake Formation, they can secure their Data Catalog, making sure only authorized User
users have access to specific data

© Digital Cloud Training | https://digitalcloud.training

 AWS Elastic MapReduce (EMR)
What is EMR? A big data platform that enables businesses to
process large datasets quickly and cost-effectively
using popular frameworks such as Apache Hadoop
and Apache Spark.

Some essential features of EMR include:

• Scalability
• Flexibility
• Security

Apache Spark is an open-source, distributed

computing system used for big data processing and
analytics.
• PySpark is the Python library for Spark which allows for high-
throughput data processing
• PySpark and EMR form a powerful toolset for big data analytics
• Use cases include log analysis, data transformations, financial
analysis, machine learning, etc.…
• We can use Boto3 to create an EMR cluster, EMR Steps
add steps (tasks) to the cluster, check the
status of the steps, and terminate clusters

© Digital Cloud Training | https://digitalcloud.training

 Glue and EMR Hands-On Lesson

© Digital Cloud Training | https://digitalcloud.training

 Glue and EMR Hands-On Lesson

Scenario: Prerequisites:
Your company processes a multitude of transaction files daily which detail Download the provided files: S3 buckets:
billing, products sold, and their quantities. This data is stored in different • emr_software_settings.json • dct-billing-processed
CSV files within S3. There's a need to consolidate this data, analyze it, and • billing_data_dairy_07_2023.csv • dct-billing-data-lake-x
derive insights like the gross profit. You are tasked with automating the data • units_sold_07_2023.csv • units-sold/
processing and analytics pipeline using AWS Glue and EMR. • production_costs_07_2023.csv • production-costs/
• reports/
• pyspark_script/
Overview:
1. Store the raw CSV data in S3 buckets
2. Use Glue Crawlers to catalog this data
3. Spin up an EMR cluster, which we’ll use to run a PySpark script
a) This script will read the data from the Glue catalog and
determine the gross profit for each product sold
b) Store gross profit results back into S3 for reporting units_sold_07_2023.csv

production_cost_07_2023.csv

billing_data_dairy_07_2023.csv

© Digital Cloud Training | https://digitalcloud.training

 Glue and EMR Hands-On Lesson

AWS Cloud

Crawler Billing
Crawlers
Populates Units Sold
parse csv data
Data Catalog Production Costs
dct-billing-processed
Glue Crawlers Glue Data Catalog

dct-billing-data-lake Generate
Gross Profit Process Data
units-sold/ reports/
Report
production-costs/ pyspark-script/ EMR Cluster

IAM Lake
Formation

© Digital Cloud Training | https://digitalcloud.training

 Managing and Automating AWS Security with Python

© Digital Cloud Training | https://digitalcloud.training

 Managing Security in AWS with Python

Importance of AWS Security Automating Security Tasks

Security in the cloud is a shared responsibility Automation ensures that security checks and responses
between the user and AWS. This distinction happen in real-time
necessitates effective security management
strategies.
Security Hub - Aggregates and prioritizes
your security alerts or findings, from
Python and AWS Security services such as:
With the Boto3 SDK, Python can interact with AWS services • GuardDuty
programmatically, providing endless possibilities for enhancing • Inspector
security management tasks • Macie
• You can automate the processing of security alerts and
findings and respond to the potential security threats
raised by these services

IAM Config Security Hub

© Digital Cloud Training | https://digitalcloud.training

 Managing Security with Lambda and Python

Python, coupled with AWS Lambda, provides a powerful Automated Security Audit System
platform for automating security tasks. Performs automatic audits of your AWS resources for
potential security issues. Such as Security Group rules
For example, if Security Hub detects an unauthorized that allow traffic from any IP address (0.0.0.0/0).
access attempt, a Lambda function could automatically
strengthen security group rules, revoke IAM credentials, AWS Cloud
or isolate affected resources.

EventBridge

Do an SGs allow
Yes traffic from any IP
address?

© Digital Cloud Training | https://digitalcloud.training

 Automated Security Auditing HOL

© Digital Cloud Training | https://digitalcloud.training

 Automated Security Auditing Hands-On Lesson

Scenario: AWS Cloud

Your company needs to create a system to audit AWS
security groups for potential security issues, such as rules
that allow traffic from any IP address (0.0.0.0/0). Yes
SNS
Your task is to create a system that checks security Topic
groups automatically, compiles a report of its findings,
and sends this report via email. Do any SGs
allow traffic
from any IP
address?
Prerequisites: EventBridge DCTSecurityGroupAudit
• Create Boilerplate (Default) Python Lambda function Scheduler
No
'DCTSecurityGroupAudit’
• Download 'DCTSecurityGroupAudit’ into Cloud9
Permissions Do Nothing

© Digital Cloud Training | https://digitalcloud.training

 SECTION 7
Leveraging Tools for Python and AWS
Development

© Digital Cloud Training | https://digitalcloud.training

 Pair Programming with GitHub Copilot

© Digital Cloud Training | https://digitalcloud.training

 Pair Programming with GitHub Copilot

Task:

Create a local python script with the help of Copilot that processes ‘Units Sold’ files and pushes them to S3

Prerequisites:

1. Download the provided files:

1. units_sold_gh_copilot1.csv
2. units_sold_gh_copilot2.csv

1. GitHub account

1. GitHub Copilot subscription (free trial)

1. IDE which supports Copilot

1. Configure the AWS CLI:

a) Check the “AWS CLI Setup and Configuration” lecture from Section 1
b) https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

6. ‘dct-billing-data-lake-x’ bucket

6. ‘units-sold’ subfolder

© Digital Cloud Training | https://digitalcloud.training

 Pair Programming with GitHub Copilot

Previous Glue and EMR Hands-On Lesson

Crawlers parse Crawler

csv data Populates Data
Catalog
Units Sold CSV Files
Glue Crawlers Glue Data Catalog

Script
Processes Files

Generate Gross Process

dct-billing-data-lake-x Data
Profit Report
production-costs/
Push to s3 EMR Cluster
units-sold/
‘units-sold’

© Digital Cloud Training | https://digitalcloud.training

 Python for the AWS
Cloud

© Digital Cloud Training | https://digitalcloud.training