AWS Lambda

AWS
Lambda
Solita.
AWS Lambda
Contents
1. AWS Lambda
2. Other AWS tools used in this exercise
3. Accounts & Roles
4. AWS Web Console
5. Exercise part 1: Analysing file metadata from S3
6. Exercise part 2: Writing analysis result to DynamoDB
7. Demo: Analyze large log file using EMR
”
Lambda comes from the Lambda
Calculus and refers to
anonymous functions in
programming.
”
AWS Lambda is a compute service
that runs your code in response to
events and automatically manages
the compute resources for you
AWS Lambda
› Currently supports Javascript syntax using AWS NodeJS SDK

› About all of the AWS services usable from Lambda
› Suitable for building micro services, triggered by various AWS services
› Lambda function can be triggered from S3 event, DynamoDB stream or
Kinesis at the moment
› It’s like bash, where you can call various unix-commands and pipe them
together
› Price is based on max amount of RAM and duration of the function call
”
Other AWS tools used in
this exercise
S3 – Simple Storage System
› Redundant, scalable object storage

› Buckets are object (file) containers
› Can serve static web content (usually using CloudFront in front)
› Can set object lifecycle, e.g. archiving to Glacier or deletion after
defined time
› Can set triggers for object events
› Price based on used storage space
DynamoDB – NoSQL database
› Scalable (read and write performance defined per

table)
› Document and key-value data models
› Price is based on the user defined read and write
performance values
EMR – Elastic MapReduce
› Hadoop as a service
› Price based on amount and type of cluster nodes
”
Accounts & Roles
Accounts
› Accounts created per user

› Rights limited to services used in this exercise
› S3 bucket name must match:
fi.solita.lambda.<account_username>.*
Roles
› invocationrole: assumed by S3 when triggering the Lambda

function
› right to call the Lambda
› executionrole: assumed by Lambda when it is executing

› right to call used AWS services
› emr-instance-role: assumed by the Hadoop instances

› emr-service-role: assumed by the Elastic MapReduce
”
AWS Web Console
AWS Web Console
› https://739080850666.signin.aws.amazon.com/
console
”
Exercise
”
Exercise part 1:
Analyzing file metadata
from S3
1. Creating a Lambda function
› AWS Web Console -> Services -> Compute -> Lambda

› Create a Lambda function
› Name: fi-solita-lambda-<user>-test
› Description: Lambda testing
› Code entry type: Edit code inline
› Code template: S3 Get Object
› Handler name: handler
› Role name: executionrole
› Use defaults for advanced settings
› Create Lambda function

› See the Function ARN, such as: arn:aws:lambda:eu-west-1:739080850666:function:fi-solita-lambda-samion-test, needed
on next phase
2. Creating an S3 bucket and trigger for Lambda function
› AWS Web Console -> Services -> Storage & Content Delivery ->
S3
› Create Bucket
› Bucket Name: ‘fi.solita.lambda.<user>.test
› Region: Ireland
› No logging
› Create
› Click on the bucket name and select Properties
› Create trigger for the Lambda function under Events
› Name: BucketEventToLambda
› Events: ObjectCreated
› Send To: Lambda function
› Lambda function ARN: <the one from step 1>, e.g. arn:aws:lambda:eu-
west-1:739080850666:function:fi-solita-lambda-samion-test
› Invocation role ARN: arn:aws:iam::739080850666:role/invocationrole
Testing
› Upload test-input.txt to your bucket

› Follow the CloudWatch log files, link available from
the Lambda function list page
”
Exercise part 2: Writing
the metadata to
DynamoDB
DynamoDB usage
› File lambda-exercise-part2.js contains DynamoDB

usage example
› Change username samion to your own
”
Demo – analyze log file
using EMR
Source and output
› Source is a plain text log file which contains device

state change events
› Aim is to collect the state change events and calculate
sums of state changes per device
› Output is json with device name and state change
count
Lambda - input
› This lambda function is triggered by incoming log

file in input bucket
› It copies the log file to EMR input bucket
› It creates and runs a job flow using Elastic
MapReduce, i.e. a Hadoop cluster
MapReduce
› mapper.js
› Reads through the input log file, skips unrelated lines and creates a
map from device names to device state change events
› reducer.js
› Reads through state change events of one device and counts them,
writes json with device name and count
Lambda - output
› This lambda function is triggered by the Elastic

MapReduce output, being written to another bucket
› It reads the output file and writes the device names
and counts to DynamoDB either item by item or
using batches
Documentation
› Lambda: http://aws.amazon.com/lambda
› Lambda Developer Guide: http://docs.aws.amazon.com/lambda/latest/dg/lambda-dg.pdf
› Lambda blogs: https://aws.amazon.com/blogs/compute/category/aws-lambda/
› S3: http://aws.amazon.com/s3
› DynamoDB: http://aws.amazon.com/dynamodb
› EMR:
› Node.js SDK: http://aws.amazon.com/sdk-for-node-js
› Node.js Streaming EMR:
http://blogs.aws.amazon.com/bigdata/post/TxVX5RCSD785H6/Node-js-Streaming-MapRed
uce-with-Amazon-
EMR

AWS Lambda

Uploaded by

Copyright:

Available Formats

AWS Lambda

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AWS Lambda

Uploaded by

Copyright:

Available Formats

AWS

› Currently supports Javascript syntax using AWS NodeJS SDK

› Redundant, scalable object storage

› Scalable (read and write performance defined per

› Accounts created per user

› invocationrole: assumed by S3 when triggering the Lambda

› executionrole: assumed by Lambda when it is executing

› emr-instance-role: assumed by the Hadoop instances

› AWS Web Console -> Services -> Compute -> Lambda

› Create Lambda function

› Upload test-input.txt to your bucket

› File lambda-exercise-part2.js contains DynamoDB

› Source is a plain text log file which contains device

› This lambda function is triggered by incoming log

› This lambda function is triggered by the Elastic

You might also like