Workflow Performance Tuning in Release 12.1
Workflow Performance Tuning in Release 12.1
Workflow Performance Tuning in Release 12.1
in Release 12
Karen Brownfield
Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
About the Speaker
• Oracle Ace
• Oracle Certified Specialist (EBS and Fusion)
• Over 20 years E-Business Suite support
• 14 years Oracle Workflow design and support
• OAUG Board 1994-2009, 2014-2015, former President
• Member ATG Customer Advisory Board
• Co-Chair Oracle EBS User Management SIG
• Over 100 presentations worldwide
• Co-Author multiple books on E-Business Suite
– The ABCs of Oracle Workflow for E-Business Suite
Release 11i and Release 12
– The Release 12 Primer – Shining a Light on the
2 Release 12 World
Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
• Established in 2001
• SBA 8(a) Small Business
disadvantaged company
• GSA Schedule contract
GS-35F-0680V
• Texas State HUB vendor
• For more information, go
to our web site at
www.Infosemantics.com
– R12.1.3, R12.2, OBIEE
public vision instances
– Posted presentations on
functional and technical
topics
3 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Agenda
• Introduction
• Patch Current
• Clean Up Old and Errored
• Purge
• Background Processing
• Queue Performance
• Mailer Performance
• Miscellaneous Performance Aides
4 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Which Are You?
The opposite of the ostrich is the rooster who is alert and awake
early to see what is on the horizon.
Rather than fear, he crows loudly a warning to be heeded by all.
Source: http://users.cybertime.net/~ajgood/ostrich.html
5 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Workflow Analyzer
6 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Workflow Analyzer
7 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Workflow Community
• Go to the Oracle
Support
Community tab
• Enter “Workflow”
to go to the
workflow sub-
space
8 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Workflow Community
9 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Patch Current
Patch Current
12 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
From Workflow Analyzer
Mailer Patches
13 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
From Workflow Analyzer
Non-Mailer Patches
14 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Clean Up Errors and
Old Workflows
From Workflow Analyzer
16 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Clean up Errors
18 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Clean up Associated Error
Item Types
• Perform following query
SELECT item_type
,parent_item_type
,DECODE (end_date, NULL, 'OPEN', 'CLOSED')
error_type_status
,COUNT (*)
FROM wf_items
WHERE parent_item_type is not null
AND item_type in ('CUNNLWF','DOSFLOW','DOSFLOWE',
'ECXERROR','HRSSA','HRSTAND','HXCEMP','IBUHPSUB','OKLAMERR',
'OMERROR','PARMAAP','PARMATRX','POERROR','WFSTD','XDPWFSTD',
'ZPBWFERR', 'WFERROR')
GROUP BY item_type
,parent_item_type
,DECODE (end_date, NULL, 'OPEN', 'CLOSED')
ORDER BY 4 DESC,item_type,parent_item_type;
19 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Clean up Associated Error
Item Types
22 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Clean Up Event Errors
23 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Clean up Event Errors
(showing portion of script)
Adjust each
update
similarly
25 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
From Workflow Analyzer
Product Specific Workflows
26 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Configure (Setup)
Seeded Workflows
• Read the documentation
– Setup
– How the workflow behaves
– My Oracle Support white papers, notes
• Setup not just Builder
– Profile Options
– Approvals Management Engine (AME)
– Hierarchies
– Other Screens
• Account Generator Top Process
• PO Documents – identify workflow to run by document
• GL ledgers page – are you using approvals
27 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge
Purge!!!
31 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge – What Happens
33 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
From Workflow Analyzer
From
Finished
Concurrent
Requests
34 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Catch Up Purging
Be Careful!!
• Purge by Item Type or Age to reduce time of each run
– Commit keeps rollback from being an issue
• Run with “Core Workflow Only” = Y
• After catching up, run one more time with “Core
Workflow Only” = N to purge old design data and
orphaned notifications
– Then only need “N” once/week (orphan notifications)
• While 10g, 11g automatically reset high water marks,
removing empty space may still be recommended
35 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Rebuild or not to Rebuild
Starting
Point
After “Catch-
After “Catch-
up” Purge
up” Purge
Export/Import
37 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
From Workflow Analyzer
38 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge Issues
Synchronous Persistence Type
• Synchronous is not a choice for Persistence Type
• Lookup code, Persistence Type, is Access Level
System (i.e. can’t add codes)
• Solution
– Help | Diagnostics | Examine
• Field: CUSTOMIZATION_LEVEL
• Value: E
– Add Lookup Code for Sync
– Restore CUSTOMIZATION_LEVEL to S
39 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge Issues
Completed With Errors
• ‘Completed with Errors’ – end date is set in
WF_ITEMS but at least one activity exists in
WF_ACTIVITY_STATUSES with status of ERROR or
ACTIVE
– Can’t abort, can’t purge, and if linked to parent or
child, parent or child cannot be purged
Same workflow
Each Error has at
least 1 Active
40 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge Issues
Completed With Errors
• Modified version of $FND_TOP/sql/wfrmtype.sql
41 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge Issues
Completed With Errors
42 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge – MOS Notes
43 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge – MOS Notes
Parent / Child Problems
• 752383.1 “Purge Obsolete Workflow Runtime Data
Concurrent Request (FNDWFPR) Is Not Purging Data”
SELECT c.item_type child,
DECODE (c.end_date, NULL,'OPEN','CLOSED') child_status,
c.parent_item_type parent,
DECODE (c.parent_item_type, NULL, 'NOPARENT',
DECODE (p.end_date,NULL,'OPEN',
'CLOSED'))parent_status,
COUNT (*)
FROM wf_items p, wf_items c
WHERE p.item_type(+) = c.parent_item_type Business
AND p.item_key(+) = c.parent_item_key
GROUP BY c.item_type, DECODE(c.end_date,NULL,'OPEN',CLOSED'), Event
c.parent_item_type, Errors
DECODE (c.parent_item_type, NULL, 'NOPARENT',
DECODE (p.end_date, NULL, 'OPEN', 'CLOSED'))
ORDER BY c.item_type, c.parent_item_type;
44 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge – MOS Notes
Parent / Child Problems
• 1378954.1 “bde_wf_process_tree.sql – For analyzing
the Root, Children, Grandchildren Associations of a
Single Workflow”
45 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge – MOS Notes
Product Specific
• Order Management
– 751026.1 “FNDWFPR – Purge Obsolete Workflow Runtime
Data – OEOH / OEOL Performance Issues”
• Scripts to help close attached children
– 398822.1 “Order Management Suite – Data Fix Script
Patch”
– 405275.1 “How to Detect Data Corruption and Purge More
Eligible OEOH/OEOL Workflow Items for Order
Management Workflow”
– 878032.1 “How To Use Concurrent Program “Purge Order
Management Workflow””
• 11i.10 – Patch 9845873
• 12.1.2+ – included (not available for 12.0.x)
• Purges closed lines even when header still open
• “Attempt to Close” – purges OMERROR / WFERROR that are
orphan or attached to no-longer-error nodes
46 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Purge – MOS Notes
Product Specific
• Purchasing
– 458886.1 “How To Diagnose Issues Related To Purge
Of Purchasing Workflow Data That Remain Even After
Running The ‘Purge Obsolete Workflow Runtime Data’
Concurrent Program?”
• Scripts to help close attached children
47 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Background Processing
Background Engines
49 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Background Engines
50 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
From Workflow Analyzer
‘APPS’
not part of
workflow
name
• Note counts of Ready, run SQL and see if queue is
steady or growing
– Growing – decrease wait time for next execution of
Background Engine
– Low or Empty – increase wait time
51 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Background Engines
53 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Loops in Workflow
(from Workflow Analyzer)
54 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
AQ_TM_PROCESSES
(from Workflow Analyzer)
55 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Background Engine Runs a
Long Time
• 560144.1 “11.5.10.4: Workflow Background Process
Seems to Take Longer After Rup4” (cont)
– JOB_QUEUE_PROCESSES at least 5
• OAM recommends 10
• Oracle seeds this to 2 – change it (ignore note
396009.1, updated Feb 2012)
• MOS note 1530928.1, dated Feb 2013, value of 2
causes Row Lock Contention on Mailer
• MOS note 578831.1 – explains how to monitor
• MOS note 271855.1 – if database prior to 11.2,
perform regular rebuilds/coalesces on all dequeue
indexes/IOTs associated with an AQ table
56 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
JOB_QUEUE_PROCESSES
(from Workflow Analyzer)
61 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Advanced Queue Performance
WF_CONTROL
• Controls all other queues
• Run ‘Control Queue Cleanup’ every 12 hours
– In every instance
• 469045.1 “Troubleshooting WF_CONTROL Agent
Issues”
– Discussion of this queue
– Scripts to run to ensure subscribers are valid and dead
subscribers are removed properly
62 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Advanced Queue Performance
WF_DEFERRED
• Subscriptions to Events Phase > 100
• Workflows started by events
• 334348.1 “Low Performance Processing Messages in
WF_DEFERRED Queue”; 468650.1 “Troubleshooting
WF_DEFERRED Agent Listeners Performance”
– Use SQL to determine Events in queue
– Identify if events not being dequeued in timely fashion
• Time in queue>2X sleep time for queue
– Identify events with long processing time
• Trace code and identify issues (bugs, tuning, etc)
63 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Advanced Queue Performance
WF_DEFERRED
• 334348.1 “Low Performance Processing Message in
WF_DEFERRED Queue”; 467650.1 “Troubleshooting
WF_DEFERRED Agent Listeners Performance” (cont)
– Identify Events with high volume
• Create additional generic agent listeners
• Create specific agent listeners
• Increase ‘Inbound Thread Count’
(PROCESSOR_IN_THREAD_COUNT) by 1 until
performance acceptable
– Temporarily set retention time to 0
64 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Advanced Queue Performance
WF_DEFERRED
• Queue may be corrupt
– Receiving Errors “ORA-24033:No Recipients for
Message”
– Rebuild using instructions in note 286394.1 “How to
rebuild the WF_DEFERRED queue”
65 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Mailer Performance
Notification Mailer
68 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Notification Mailer
Click it,
issues
outweigh
benefits
• Processor Close on Read Timeout
– 315748.1 – unclick for performance
• We don’t agree with this note
– 422870.1 – unless clicked, not removed from Process
folder
– 332152.1 – must be clicked if running multiple mailers
using same SMTP Server (Outbound Name) or will
get contention and locking
– 437986.1 – must be clicked or messages get stuck in
Inbox
71 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Notification Mailer
72 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Notification Mailer
0 is SYSADMIN
73 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Notification Mailer
• Tag Files
– Out of Office, Undeliverable – set to Ignore
• 388709.1 “Email Notification Failures Are Causing
The Email Servers To Crash”
• Uncheck Mailer parameter “Send warning for
unsolicited e-mail”
– 431359.1 “Setting up a Tag in the Mailer configuration
files to handle unsolicited mail”
• Uncheck Mailer parameter “Send e-mails for canceled
notifications”
75 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Miscellaneous
Performance Aides
Item Attributes “As Needed”
77 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Item Attributes “As Needed”
• #ONDEMANDATTR
– Process Activity Attribute
– For top-level runnable process
– Type must be Text or Lookup
• Use Yes/No Lookup Type
• For Text, Value must be Y
– Do not assign an item attribute as the value
– Runtime copy only created when SetItemAttr<> used
• If referenced prior to this call, default value used
– Experiment with a particular workflow
• HRSSA, REQAPPRV, POAPPRV, etc
78 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Profile Options
79 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Profile Options – R12.2
82 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Pinning
83 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Run 64-bit Database
84 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Wffngen.sql
85 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Order Management
86 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Partition Tables
87 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Partition Tables
Available from
Amazon.com,
Barnes & Noble
(bn.com)
Lulu.com
(pdf version)
89 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.
Thank you!!
Karen.Brownfield@Infosemantics.com
90 Copyright © 2014 Karen Brownfield All Rights Reserved . Any other commercial product names herein are trademark, registered trademarks or service marks of their respective owners.