Effective Fault Handling in Oracle SOA Suite 11g

Ronald van Luttikhuizen [Vennster] Guido Schmutz [Trivadis]

1-Oct-2012 | Oracle OpenWorld & JavaOne

1|x

Guido Schmutz
Working for Trivadis for more than 15 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer, Software Architect for Java, Oracle, SOA and EDA Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 20 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Twitter: gschmutz
2|x

Ronald van Luttikhuizen

Managing Partner at Vennster
Oracle ACE Director for Fusion Middleware and SOA Author of different articles, co-author Oracle SOA Book 11g book Upcoming book SOA Made Simple Architect, consultant, trainer for Oracle, SOA, EDA, Java More than 10 years of software development and architecture experience Contact: ronald.van.luttikhuizen@vennster.nl Blog: blog.vennster.nl Twitter: rluttikhuizen
3|x

Agenda
1. 2. 3. 4. 5. What is Fault Handling ? Fault Handling in SOA vs. traditional systems Scenario and Patterns Implementation of Scenario Summary and Best Practices

4|x

Fault Handling
The goal of every programmer should be to write unbreakable software Extent of achievement depends on how good expected and unexpected exception conditions are handled and managed Object-oriented languages such as C++ and Java provide an efficient way for handling exceptions using constructs such as try, catch, and finally With a SOA, most of what is available at language level is still valid and usable SOA raises different challenges once starting orchestrating services and creating composite applications Prevention vs. handling
5|x

What is a Fault ?
Something happened outside normal operational activity or happy flow
Technical error Programming error Faulty operation by user Exceptional business behavior

Prevention and handling

6|x

Two Types of Faults

Business faults
Faults that service clients can expect and recover from Failure to meet a particular business requirement Often: expected, business value, contractual and recoverable

Technical faults
Faults that service clients do not expect and cannot (easily) recover from Results of unexpected errors during runtime, e.g. null pointer errors, resources not available, and so on Often: unexpected, technical, implementation and non-recoverable
7|x

Business Fault
<wsdl:operation name="orderProduct"> <wsdl:input message="order:OrderProductRequestMessage"/> <wsdl:output message="order:OrderProductResponseMessage"/> <wsdl:fault message="order:ProductNotInStockFaultMessage" name="ProductNotInStockFault"/> <wsdl:fault message="order:CustomerNotFoundFaultMessage" name="CustomerNotFoundFault"/> </wsdl:operation> <xsd:element name="CustomerNotFoundFaultMessage"> <xsd:complexType> <xsd:sequence> <xsd:element name="CustName" type="xsd:string"/> <xsd:element name="City" type="xsd:string"/> </xsd:sequence> </xsd:complexType> </xsd:element>

1. Service contract including fault

2. Fault message payload

8|x

Business Fault (II)

<soap:Envelope> <soap:Header/> <soap:Body> <soap:Fault> <faultcode>CST-1234</faultcode> <faultstring>Customer not found</faultstring> <detail> <CustomerNotFoundFault> <CustName>John Doe</CustName> <City>Long Beach</City> </CustomerNotFoundFault> </detail> </soap:Fault> </soap:Body> </soap:Envelope>

3. Actual service response

9|x

Technical Fault
<wsdl:operation name="orderProduct> <wsdl:input message="order:OrderProductMessage"/> <wsdl:output message="order:OrderProductResponseMessage"/> <wsdl:fault message="order:ProductNotInStockFaultMessage" name="ProductNotInStockFault"/> <wsdl:fault message="order:CustomerNotFoundFaultMessage" name="CustomerNotFoundFault"/> </wsdl:operation>

1. Service contract including fault

2. Actual service response

<soap:Body> <soap:Fault> <faultcode>S:Server</faultcode> <faultstring>Could not connect to URL 127.0.0.1 on port 8001</faultstring> </soap:Fault> </soap:Body>

10 | x

Agenda
1. 2. 3. 4. 5. What is Fault Handling ? Fault Handling in SOA vs. traditional systems Scenario and Patterns Implementation of Scenario Summary and Best Practices

11 | x

Fault Handling SOA vs. traditional systems

External
BPM User Interface

ESB

Implementation

Implementation

Implementation

Multiple service consumers Services part of larger unit Heterogeneous & external components Long running processes Asynchronous Timed events Often enterprise-wide Transactions

12 | x

Agenda
1. 2. 3. 4. 5. What is Fault Handling ? Fault Handling in SOA vs. traditional systems Scenario and Patterns Implementation of Scenario Summary and Best Practices

13 | x

Old System with limited scalability

Short Network interruptions

No 7*24 avail. for single instance of credit card service Response sometimes get lost

Fault if product is no longer available

Not always available

14 | x

Patterns for Fault Tolerant Software

Compensation Exception shielding (Limit) retry Share the load Alternative Exception handler Heartbeat Throttling
15 | x

Fault Recovery Strategies

Inaction Ignore the request Balk Admit failure Guarded suspension Suspend execution until conditions for correct execution are established Provisional action Pretend to perform the request, but do not commit until success is granted Recovery Perform an acceptable alternative

16 | x

Fault Recovery Strategies

Rollback Try to proceed, but on failure, undo the effects of a failed action Retry Repeatedly attempt a failed action after recovering from failed attempts Appeal to higher authority Ask someone to apply judgment and steer the software to an acceptable resolution Resign Minimize damage, write log information, then signal definite and safe failure

17 | x

Agenda
1. 2. 3. 4. 5. What is Fault Handling ? Fault Handling in SOA vs. traditional systems Scenario and Patterns Implementation of Scenario Summary and Best Practices

18 | x

19 | x

Product Management Result Caching

Result Cache

Problem
Not to overload the old, non-scalable product system with the new demand

Solution
Use Result Caching to cache the product information (read-only operation) Use Service Throttling to limit the number of concurrent requests
20 | x

Product Management Result Caching

Results are returned from cache rather than always invoking the external service
Product data is rather static, so ideal candidate for caching
Proxy Service
2

Business Service
3

Product DB

OSB

Result Cache

21 | x

Product Management Service Throttling

Restrict the number of messages on the message flow to a Business Service
Message Buffer

Proxy Service

Business Service

Product DB

OSB

Set from Operational Settings on the OSB console

22 | x

Credit Card Booking Retry Configuration

Retry

Problem
Unstable network between us and the external services

Solution
Use Retry mechanism of OSB to try multiple times No Fault Management necessary for service consumer if network interruption is only for a short time
23 | x

Credit Card Booking Retry Configuration

Configured on the business service in OSB
Proxy Service Business Service
1 after 2s 2

Credit Card Service

OSB

5x

24 | x

Credit Card Booking Service Pooling

Service Pooling

Problem
Credit Card Service does not guarantee 7*24 availability for one single instance

Solution
Use the multiple instances (endpoints) that the company provides and use service pooling feature of OSB No Fault Management for the service consumer if at least one endpoint is available
25 | x

Credit Card Booking Service Pooling

Credit Card Service instance 1 Proxy Service Business Service Credit Card Service instance 2 Credit Card Service instance 3

OSB

26 | x

Order Management Transaction configuration

Transaction of OSB Service Consumer

Problem

Transaction of OSB Service Consumer

Guarantee that the message will be delivered to the order management system

Solution
Make sure that queues are available, even if the Handle Order system is not Make sure that queuing runs in the same transaction as the service consumer
27 | x

Transactions in OSB
Explanation and Demo of Transactions in OSB

28 | x

Order Management (II) Fault Message on Callback Contract

Fault Message on Callback Contract

Problem
Need to return a Product No Longer Available Business Fault over an Asynchronous MEP

Solution
Design a separate Fault Message and Operation on the Callback contract (WSDL) and use that
29 | x

Order Management (II) Fault Message on Callback

Business Fault modeled as another operation on the Callback WSDL

30 | x

Order History Fault Management Framework

Use Fault Policy Management In Mediator to configure retry

Problem
Order History System not available should have no impact on Business Process

Solution
Use Mediator with Fault Management Framework to configure retry independent of availability of Order History Web Service
31 | x

Order History Fault Management Framework

<faultPolicyBindings version="2.0.1"> <composite faultPolicy="OrderProcessFaultPolicy"/> </faultPolicyBindings> <faultPolicies> <faultPolicy version="2.0.1" id="OrderProcessFaultPolicy"> <Conditions> <action ref="RetryAction"/> </Conditions> <Actions> <Action id="RetryAction"> <Retry> <retryCount>3</retryCount> <retryInterval>2</retryInterval> <exponentialBackoff/> <retryFailureAction ref="HumanInterventionAction"/> <retrySuccessAction/> </Retry> </Action> </Actions> </faultPolicy> </faultPolicies>

32 | x

Order Handling Process Return errors as synchronous response

Problem
Fault Handling Fault Handling Reply with Fault

Both Product Management and Credit Card Booking can return Business Faults

Solution
Handle errors and map them to errors returned to the service consumer (i.e. the caller of the process)

33 | x

Order Handling Process Return errors as synchronous response

Handle Business Faults in BPEL error handler and reply with an error

34 | x

Order Handling Process (II) Handle missing callback with timeout

Problem
Order Processing response message can get lost in the Order Processing system, i.e. the callback message will never arrive in the process

Solution
Pick with timeout Compensate

Timeout on the Wait For Answer with a BPEL pick activity with a timeout Undo the process by doing compensation Use the BPEL compensate activity together with compensation handler to undo the Booking of the Credit Card
35 | x

Order Handling Process (II) Handle missing callback with timeout

Pick Activity for handling callback message with timeout branch

36 | x

Order Handling Process (III) Compensation Handling

Problem
Compensation Handler

Order Processing callback message can be a Product No Longer Available Business Fault

Solution
Handle Business Fault and Compensate

Undo the process by doing compensation Use the BPEL compensate activity together with compensation handler to undo the Booking of the Credit Card

37 | x

Order Handling Process (III) Compensation Handling

Compensate activity invokes compensation handling on the inner scope
Can only be invoked from within a fault handler or another compensation handler

38 | x

Order Handling Process (IV) Non-idempotent operations

Problem
Credit Card Booking is a non-idempotent operation

Solution
Non-Idempotent

To avoid BPEL calling the Book Card operation again (not really possible here), we have set the idempotent Property on the partner link to FALSE

Idempotent

39 | x

Order Handling Process (V) Generic Error Handler w. Fault Policy Framework
Problem
Unexpected (technical) error

Unexpected (technical) fault Multiple processes that deal with unexpected faults in their own way

Solution
Use fault handler mechanism to enqueue on error queue without adding process logic Create one process to listen to error queue and handle faults Retrieve process information by using (composite) sensors
40 | x

Order Handling Process (V) Generic Error Handler w. Fault Policy Framework
<faultPolicyBindings version="2.0.1"> <composite faultPolicy="GenericFaultPolicy"/> </faultPolicyBindings> <faultPolicies> <faultPolicy version="2.0.1" id="GenericFaultPolicy"> <Conditions> <action ref="GenericAction"/> </Conditions> <Actions> <Action id="GenericAction"> <javaAction className="nl.vennster.GenericHandler defaultAction=HumanIntervention"> <returnValue value="HumanIntervention" ref=" HumanIntervention"/> </javaAction> </Action> </Actions> </faultPolicy> </faultPolicies> <property name="oracle.composite.faultPolicyFile">oramds:/apps/fault-policies.xml</property> <property name="oracle.composite.faultBindingFile">oramds:/apps/fault-bindings.xml</property>

41 | x

Order Handling Process (V) Generic Error Handler w. Fault Policy Framework
Explanation of generic fault handler

42 | x

Agenda
1. 2. 3. 4. 5. What is Fault Handling ? Fault Handling in SOA vs. traditional systems Scenario and Patterns Implementation of Scenario Summary and Best Practices

43 | x

Summary
Issue Overloading product management system Credit Card Service does not guarantee 7*24 uptime due to e.g. network problems Guarantee message delivery to order management system Solution ThrottlingResult cache Muliple endpoints Service pooling Availability of queues Enqueue and dequeue in service consumer transaction Separate operation and fault message Product OSB OSB OSB (and SOA Suite for XA propagation to OSB) OSB and SOA Suite (callback contract between the two)

Returning business fault over async MEP from order management system Order history service not available Business fault handling from service to process to consumer Detect missing response message Handle product no longer available Avoid calling credit card booking twice Processes needing to deal with unexpected technical faults. All processes solving it in their own way using process logic.

Retry in Mediator using fault policy framework SOA Suite Catch faults in process and reply fault to consumer Timeout in pick activity Compensation Set non-idempotent property Fault policy frameworks, error queue, generic error handler, SOA Suite APIs & composite sensors. OSB and SOA Suite (correct contracts) SOA Suite SOA Suite SOA Suite SOA Suite

44 | x

Best Practices
Differentiate between business and technical faults Design service contracts with faults in mind: formally describe business faults in service contracts Dont use exceptions as gotos Design with criticality, likeliness to fail, and cost in mind Differentiate fault patterns in OSB and BPM/BPEL
OSB: Retry, throttling, transaction boundaries BPM/BPEL: Compensation, business fault handling, generic fault handler, timeout

Handle unexpected errors generically Make services autonomous Fault-handling on scope of services and in wider perspective
45 | x