BEDCon 2016 - Kay Lerch on "Will trade an ESB for an agile integration solution in the cloud"

Will trade an ESB for an agile
Integration solution in the
Cloud
Berlin Expert Days,
16. September 2016, Urania Berlin
@KayLerch | Engineering Manager | Immobilien Scout

Agenda
BEDCon 2016 | @KayLerch
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy
 Drawing the big picture of a hybrid integration solution

Competitive forces and what they do with Enterprise IT
Competitive
forcesMarket
IT of an E-commerce business
Fast-paced feature
development (“Lean IT”)
Core business systems and
services (“Classical IT”)
Innovation / Agility / Speed
Agile teams
Microservices / modularized
DevOps
Efficiency / compliance /
reliability
Project organization
Legacy / monolithic
Operations management
Release-driven Deployments
API-led Connectivity Proprietary Interfaces
Boosting
time to
market
CLD
Continuous DeploymentProduct

“Two speed IT” parts heavily depend on each other not only for the good
Competitive
forcesMarket
A digital business
Fast-paced feature
Agile teams
DevOps
Continuous Deployment
reliability
Legacy / monolithic
Boosting
time to
market
CLD
Organizational
clash
Technical
clash
„Two Speed IT“
Product

Overcome technical burden with Integration strategy (SOA?)
Competitive
forcesMarket
A digital business
Fast-paced feature
Agile teams
DevOps
reliability
Legacy / monolithic
Boosting
time to
market
CLD
„Two Speed IT“
Product
Organizational
clash
Technical
clash
ICC
ESB

Well, not … Integration projects tend to be traditional (often) for good reason
Competitive
forcesMarket
A digital business
Fast-paced feature
Agile teams
DevOps
reliability
Legacy / monolithic
Boosting
time to
market
CLD
„Two Speed IT“
Product
Organizational
clash
Technical
clash
ICC
ESB

An integration strategy should follow a “bimodal” approach where so-
called hybrid integration platforms (HIP) strive for Self Service Integration
Competitive
forcesMarket
A digital business
Fast-paced feature
Agile teams
DevOps
reliability
Legacy / monolithic
Boosting
time to
market
CLD
„Two Speed IT“
Product
ISG
HIP

The digital ecosystem brings a lot of new potential but also threat
Mobile
IoT
User Data
Cloud Social Networks Digital marketplace
Digital ecosystem
discruptive

Businesses need to adopt and integrate these potential
Mobile
IoT
User Data
Digital ecosystem
discruptive

An HIP got its name from being the gateway for those new stuff
Mobile
IoT
User Data
Digital ecosystem
discruptive

HIP is a concept whereas iPaaS, iSaaS are (commercial) solutions in the cloud
Mobile
IoT
User Data
Digital ecosystem
discruptive
iPaaS
iSaaS

Ok, you got your BINGO! Let’s move on in the real world …

Given an ESB in charge of syncing business data across the enterprise
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article

This centralized hub is key for processes arching over multiple domains
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article

It encapsulates the burden of integrating system interfaces with brokers
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
{ } { }
{ }
{ }

Teams depend on those brokers in order to change their interfaces
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
{ } { }
{ }
{ }
API
API
API
API

It would be great to push responsibility (integration logic) to the edges
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }

It would also be great to delegate process ownership (orchestration logic)
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }

ESB left over for “dirty work” (messaging, tracking, governance, …)
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }

It should also be in the cloud to not be isolated from the digital ecosystem
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }

Why can’t this thing just be like an HIP?
HIP
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
{ }
{ }
{ }
{ }

As our company moves to AWS anyway, let’s give SWF a try
• There is PAYG
• AWS ecosystem is huge, it serves all the hip stuff like serverless architecture (Lambda, API Gateway),
IoT, Mobile integration, Messaging, Elastic computing, Container deployments and more
• There‘s a big community around AWS
• SWF (Simple Workflows) is used by NASA for processing data from the Mars-Rover on earth
You got me at „Mars“…

Agenda
 Disruptive forces and what they do with enterprise IT
 An ideal integration platform
 AWS Simple Workflows (SWF) in a nutshell
 Demo time
 Leveraging SWF to get rid of a classical ESB solution
 Reclaim process ownership and end-2-end-autonomy

SWF is a workflow engine scheduling tasks for all the workflow participants

A workflow starter simply kicks off a workflow with some input via API
1
{ "childPolicy": "string", "domain": "string", "executionStartToCloseTimeout": "string", "input": "string", "lambdaRole": "string", "tagList": [
"string" ], "taskList": { "name": "string" }, "taskPriority": "string", "taskStartToCloseTimeout": "string", "workflowId": "string", "workflowType":
{ "name": "string", "version": "string" } }
{ "runId": "string" }

SWF has no clue what comes next so it schedules a “decision task”
2

A decider owns the actual workflow logic. It is an application polling for
tasks via API and returns decisions to the SWF engine
3
{ "domain": "string", "identity": "string", "maximumPageSize": number, "nextPageToken": "string", "reverseOrder": boolean, "taskList": { "name":
"string" } }
{ "decisions": [ { "cancelTimerDecisionAttributes": { … }, "cancelWorkflowExecutionDecisionAttributes": { … },
"completeWorkflowExecutionDecisionAttributes": { … }, "continueAsNewWorkflowExecutionDecisionAttributes": { … }, "recordMarkerDecisionAttributes": {
… }, "requestCancelActivityTaskDecisionAttributes": { … }, "requestCancelExternalWorkflowExecutionDecisionAttributes": { … },
"scheduleActivityTaskDecisionAttributes": { … }, "scheduleLambdaFunctionDecisionAttributes": { … },
"signalExternalWorkflowExecutionDecisionAttributes": { … }, "startChildWorkflowExecutionDecisionAttributes": { … }, "startTimerDecisionAttributes":
{ … } } ], "executionContext": "string", "taskToken": "string" }

SWF fulfills the remotely given decisions –> e.g. it schedules an activity task
4

An activity worker polls for tasks from a task list, works on it and returns a result
5
{ "domain": "string", "identity": "string", "taskList": { "name": "string" } }
{ "result": "string", "taskToken": "string" }
{ "details": "string", "reason": "string", "taskToken": "string" }

SWF receives the result – again has no clue how to go on – so it schedules
another decision task
6

The decider receives the task. It now contains a detailed protocol of what
happened in this workflow-execution. Based on that it gives the next decision(s)
7

SWF schedules the next task – according to the last decision by the decider.
Activity workers can be serverless Lambda-functions as well
8

This goes on and on until the decider aka workflow worker decides for
completing the workflow. What again is executed by SWF itself.

You keep track of your workflow executions in AWS console

AWS CloudWatch provides performance metrics for your workflows

SWF-powered IoT-Photobooth
2) Shoot a photo
View progress on breadboard
3) View on Slack
1) Press a button

AWS Lambda
AWS SWF
Lambda
Task
Lambda
Upload to S3
& Start
Workflow
Lambda
Update Shadow
Lambda
Task
Lambda
Get S3 Photo and
Post Message to
Slack
Lambda
Task
Lambda
Task
Lambda
Task
Lambda
Shorten Url
with Bitly
Lambda
Update Shadow
Lambda
Update Shadow
AWS EC2
WorkflowWorker
DeciderImpl
Raspberry Pi
AWS IoT
Thing
Shadow
Python Script
GPIO
Rule
AWS S3
Bucket
PutFile
StartWFInvoke
MQTT
Publish
MQTT
Receive
Update
Shadow
Poll Tasks
Shoot
Impulse
Impulse

So let’s use SWF for replacing the ESB in our company.
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
{ } { }
{ }
{ }
API
API
API
API

Client-side worker integrate their services by pulling tasks from SWF
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
wor
ker
wor
ker
wor
ker
wor
ker
Acti
vity
Acti
vity
Acti
vityActi
vity
AWS
SWF

Workers poll for tasks in task lists associated with declarative activity type
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
API
API
Acti
vity
Acti
vity
Acti
vityActi
vity
Task
list
Task
list
Task
list
Task
list
HTTP
REST
HTTP
REST
HTTP
REST
HTTP
REST
AWS
SWF
wor
ker
wor
ker
wor
ker
wor
ker

Workers can be Lambda functions as well in charge of the edge team
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
Acti
vity
Acti
vity
Acti
vityActi
vity
Task
list
Task
list HTTP
REST
HTTP
REST
AWS
SWF
wor
ker
wor
ker
lam
bda
AWS lam
bda
AWS
Event
Trigger
Event
Trigger

That’s nice, but there’s still a central orchestration controlled by one decider
ESB
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
Acti
vity
Acti
vity
Acti
vityActi
vity
Task
list
Task
list HTTP
REST
HTTP
REST
AWS
SWF
wor
ker
wor
ker
lam
bda
AWS lam
bda
AWS
Event
Trigger
Event
Trigger

Delegate parts of the orchestration in child workflows reclaims process ownership
SINA
Salesforce
CRM Sales
CC
ERP
AMGFinance
Checkout
Service
ARTE
Fraud
Service
SHIELD
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Customer
Contract
User Trust
Open Payment
Payment Profile
Article
Performance Rec
User Trust
Customer
Contract
Article
API
API
HTTP
REST
AWS
SWF
wor
ker
lam
bda
AWS
Event
TriggerEvent
Trigger
HTTP
REST
AWS
Acti
vity
Task
list
wor
ker
Sub
WF
Acti
vity
Task
list

Reclaim process ownership leads to real end-2-end-autonomy of teams
Sales

End-2-End autonomy along the whole lifecycle of a worker
Sales

We call them Microworker as this is what they really are
Sales

BEDCon 2016 - Kay Lerch on "Will trade an ESB for an agile integration solution in the cloud"

Q & A
It‘s time to ask questions and give feedback.
Thanks for joining my session ...

Backup (SWF at JPL Datacenter of NASA)
Sales

Backup (SWF-powered processing of images from Mars rover)
Sales

Backup (SWF-powered contract data distribution at Scout24)
Sales

BEDCon 2016 - Kay Lerch on "Will trade an ESB for an agile integration solution in the cloud"

More Related Content

BEDCon 2016 - Kay Lerch on "Will trade an ESB for an agile integration solution in the cloud"

Editor's Notes