Oracle WebLogic Diagnostics & Perfomance tuning

Oracle WebLogic
Troubleshoot, Diagnostics &
Performance tuning
Michel Schildmeijer, 13 juni 2013

Agenda
17:10-18:00
• What does Performance include?
• Performance hints, tips & tricks
• Questions

Agenda
18:00
• Refreshments

My personalia
• Michel Schildmeijer, 46 years, living in Amsterdam
• Oracle Fusion Middleware specialist
WebLogic 6 to 12, OSB, Tuxedo, SOA
• Since April 2012 part of Qualogy
• Part of Qualogy’s Exalogic Squad team

WebLogic Troubleshoot overview
• System errors, messages lost, bad performance
• Where do I begin??
• What method should I follow?

• Using these troubleshooting methods helps you to:
• A structured way of detecting issues
• Get behind the cause of the problems that occur
• Methods or patterns to follow:
• Use of Operating system tools
• Use of JDK tools
• Use of WebLogic Server tools
• Most likely you will often use a combi-nation

• O/S tools:
• sar: system activity
• mpstat: per-processor statistics
• vmstat: virtual memory statistics
• netstat: network statistics
• iostat: Input/Output statistics
• top

• Start with a simple check – it seems so obvious
but:
• Is there enough diskspace left
• Is there enough RAM available
• Does the system do a lot of paging

• On Operating System:
• See if WebLogic's operating system Java process is running
• Unix: ps -ef | grep -v ' weblogic.Name' | grep weblogic.Name –
color
to see which server instances are running on that host
• Simple O/S tools like top, nmon, sar can help you on the way
• Check the contents of the process's system-out/err logs
• Usually in the WebLogic Server log location or redirected to the
Server Logs
• Check the contents of the WebLogic
server and domain logs
• Located at: domain/servers/
server/logs

Tools to troubleshoot
• WebLogic Server generic:
• WebLogic Diagnostics Framework
• Logs
• WLST
• JVM specific :
• JRockit Mission Control,
• Jstat,
• JConsole

Tools to investigate
Oracle Fusion Middleware basis
• Operating system basic HotSpot JDK tools:
• FMW basis: de Applicatie Server
• jps -l
• jinfo pid
• jmap pid
• jstat -optionpid
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• pid is gathered using ps –ef …. Or
using jps
• Welke Applicatie Server gebruikt Oracle?

• jps –l

• Operating system basic JRockit JDK tools:
• jps -l
• jrcmd pid --MissonControl Command line
• pid is gathered using ps –ef …. Or
using jps

• jrcmd
• Wat jrcmd doet pid -l een | grep Applicatie jrockit.gc.Server?
latest.heapSize

• Force a dump by:
• kill -3 pid … stack in STDOUT
• Usually in
domain/servers/logs/serverinstance/serverinstance.out

WebLogic troubleshoot methods
• Keep it as simple as it is :
• In the AdminConsole
• WLST
• connect('weblogic', 'weblogic', 't3://localhost:7001')
serverRuntime()
print cmo.getHealthState()
• Or in clustered mode
domainConfig()
serverList=cmo.getServers();
domainRuntime()
cd('/ServerLifeCycleRuntimes/server')
cmo.getState()

WebLogic troubleshoot methods
• Just a simple example, but parameterize your scripts

WebLogic taking threaddumps
• A threaddump is a snapshot of the state of a
WebLogic Server Instance
• Different ways to take a threaddump:
• kill -3 pid the O/S way
• Using the JDK tools
• WLST(WebLogic Scripting Tool)
cd ('Servers')
ls()
cd ('AdminServer')
ls()
threadDump()

WebLogic taking threaddumps
• In the AdminConsole

The use of taking threaddumps
• Thread Dumps can give you:
• View of the state of application server threads at that
instant in time
• Information about potential issues such as
• Hot spots within code which seem to be called often
• Portions of code where the application seems to be hung
• Locking and thread synchronization issues in an application

• Take several dumps to investigate various states
• Threads high in stack can be treated as idle, mostly
waiting for some work to do
• Blocked or Wait on socketAccept() listener
thread waiting for socket connections

Understanding Java threads
• JVM process consists Java and JVM internal
threads
• JVM threads: Garbage Collect, Optimizer and
Finalizer
• Each thread has a stack to store runtime data
• Overflow will happen if this stack will store
more than its size
• Each O/S has its default stacksize
• Can be extended with -Xss

Understanding Locks
• Are meant to synchronize between threads
• Lock contention is a thread waiting for another to
be released
• Fat locks: Have a history of contention (several
threads try to take the lock simultaneously), or
waiting for notification
• Thin locks: Without any contention.
• Recursive locks: A lock on a
thread several times without having been released.

• WebLogic specific:
• Ignore threads in ExecuteThread.waitForRequest()
• Muxer and processSocket threads are mainly for
houskeeping
• A Muxer which is blocked should not be waiting for a
lock by a non Muxer

• Hanging threads will be diagnosed as STUCK
• Default time to get STUCK is 600 secs
• Thread has not completed it’s work

WebLogic Log and Debug options
• Logging services are important for diagnostics
• WebLogic produces output by using a Message
Catalog
• Using the standard java.util.logging
• Can be extended using log4j modules
• All kinds of events, like startup/shutdown,
appl.deployments, subsystem failures
are written to logs

WebLogic Log and debug methods

• Several severity levels like:
• TRACE, DEBUG,INFO ..
• Important to set the proper level
• For FMW specific components, ODL-HANDLER can
be used

• Subsystem logs can be from:
• HTTP
• Transaction Manager
• JDBC
• JMS server
• WebLogic Auditing provider

WebLogic debug options
• Several Subsystem can be debugged :
• HTTP
• JTA
• JDBC
• JMS server
• Security/SSL
Seems obvious, but only use it
when it’s neccessary

• JDBC debug
• Setting debug options in the
AdminConsole

WebLogic Log4j
• OpenSource Log engine from Apache
• Enable the LogMBean.isLog4jLoggingEnabled
attribute
• In WLST: cmo.setLog4jLoggingEnabled(true)
• Add WebLogic-specific Log4j classes,
WL_HOME/server/lib/wllog4j.jar and the log4j.jar
file to the server CLASSPATH

WebLogic Log4j
• Enable log4j in AdminConsole

• Or use –D options in startup script
-Dweblogic.debug.DebugJDBCSQL=true
Prints information about all JDBC methods
invoked, arguments and return values, and thrown exceptions
Depending what item you want to debug: connection leaks,
queries

• Or use WLST
serverConfig() or domainConfig()
debug =
getMBean('/Servers/mserver/ServerDebug/mserver')
debug.setDebugJDBCInternal(true)

Oracle diagnostic Logging (ODL)
• As an extension on the default logging handlers
• Comes with FMW installation, not WebLogic default
• Logging view and config in Fusion Middleware Control
• Can be set to several trace levels
• Output in servername-diagnostic.log

WebLogic JDBC common errors
• Wrong or missing drivers in Classpath
• Connection errors by:
• Usernames/passwords
• Connection leaks by bad apps
• Set Inactive timeout to a value (def is 0)
• Insufficient connection parameters
• Max Capacity, Reserve Time Out(-1)
• Invalid JDBC URLS
• weblogic.common.ResourceException:
Could not create pool connection.
The DBMS driver exception wa…

WebLogic and JVM diagnostics
based on JRockit JDK

The JVM
• JVM is a byte code interpreter
• JVM heap is the area of memory used by
the JVM

JVM from code to machine
• A Java application runs a piece of code
• Code enters the JVM ( JRockit, HotSpot)
• It enters several stages:
• Operations
• Data structure
• Transformation ( to bytecode)
Java code
JJVVMM
Operatons
Data
Structure
Transformation
110110100

• Crash can occur during
• Compile process
• Cause: invalid native code
• Internal Code – Garbage Collect ( Segmentation Fault)
• Cause: GC Fault leading into bad memory access
• Long running GC can result in paging an CPU overload
• Application code
• Like 3rd Party drivers
• (JDBC, MQ libraries)
• Software modules

• Runtime behaviour:
o JVM runs as an OS process
o Execute work through threading model
(concurrency)
o Garbage Collector
• Each Weblogic Server (instance) runs inside a
separate JVM (instance).

JVM troubleshoot
• What to look for in java thread dumps:
• Low Memory Detector
• (Daemon thread to detect and report low memory conditions)
• CompilerThreadn
• (Daemon thread/s to compile optimize byte code)
• Signal Dispatcher
• (Daemon thread to respond OS signals)

JVM troubleshoot
• Finalizer
• Daemon thread to handle objects in finalizer queue
• Reference Handler
• Daemon thread to handle objects in reference queue
• VM Thread
• Main thread from JVM
• GC task thread#n
• Thread/s to perform garbage collection

• JRockit
• JRockit Mission Control
• Grinder
• HotSpot
• JConsole
• JVisualVM
JVM tools

JRockit Mission Control
• JRockit Mission Control provides:
• Console
• Live JVM statistics
• Graphs and metrics about local or remote systems
• Basic JMX interaction with MBeans
• Runtime data
• An in-depth look at live metrics, like memory, CPU and stack trace on
specific threads
• Profile methods or exceptions
and execute commands

JRockit Mission Control
• Memory Leak Detector
• Shows overall trend
• The growth rate of each type (class) is shown, rather than just the
percentage of the heap they take up.
• No common classes from showing at the top (like char[] arrays)
• Visual representation of hot objects in the heap
• By type
• See how each class is referenced by other classes
• By instance
• Can also see how each instance of each class is referenced
• Shows where objects are allocated
• See stack trace of each method holding
an instance and what percentage of instances are held by each method.

WebLogic Diagnostics Framework

WLDF
• Framework for monitoring and diagnostics
• Consolidated access for MBean data,
instrumentation, logging, debugging,

WLDF
• Source for monitoring data
• Server application instrumentation notifications
based on the output : log, email, SNMP, JMX, JMS
• Request dyeing and tracing
• Data archiving for later access
• WLDF Dashboard

• Key Features
• Gathering input from JMX MBeans
• WebLogic subsystems and application code can be
instrumented to provide events and watches
• Notifications can be defined for events and watches
• Console extension for monitoring of current and
historical data
• Data archiving
WLDF

WLDF
• Can be configured on filesystem or Database
• Configure with Data retirement

WLDF
• Create Diagnostic Module
• Collect Metrics. A collection includes:
• The MBean type to query
• The specific MBean instance names to query (all instances,
• by default)
• The MBean attributes to collect (all attributes, by default)
• How often to gather data
• Create Watches Notifications

Oracle WebLogic Diagnostics & Perfomance tuning

Performance Terms and Definitions
• Performance : How system’s response time and throughput
are affected by adding load.
• Capacity : Maximum threshold a system is under a given set
of conditions.
• Scalability : how does a system responds to increasing load
by adding additional resources.
• Key indicators are :
• Response time
• Throughput

Performance objectives
• Locate and minimize bottlenecks in Your System
• Gather info about Application(s)
• Concurrent number of users and interfaces.
• Number and size of app requests.
• Amount of data and its consistency.
• Determining your target CPU utilization
• Tune your Database(s) ( client or repos)
• Tune WebLogic Server Performance Parameters
• Tune Your JVM
• Tune the Operating System
• Tuning WebLogic Persistent Store

Tune WebLogic Server
• Investigate and analyze!
• Look at the following

Native I/O and muxers
• Enable by default
• Will use a O/S threadpool instead from “borrowing”
from WebLogic def exec.Q
• Native muxers use platform-specific native binaries to
read data from sockets
• Non native or Java sockets :
• Uses pure Java to read data from sockets.
• Blocks until there is data to be read
from a socket(can cause issues
on server level)

WebLogic Thread Count
• Represents the number of simultaneous
operations performed by apps that use the execute
queue.
Application
WebLogic Socket Muxer
WebLogic Kernel assigns
Execute Req to
ExecuteThread 1
Execute Request
ExecuteThread 1 handles
the request (for example
executes a JDBC request)
WebLogic Kernel handles
self tuning ThreadPool,
Workmanagers, allocations
The app now handles the
thread itself and gives back
result

WebLogic Thread Count
• Do not set this too high; can impact you
performance
• Default contains 15 threads
• Can be changed in config.xml
• Leave this default

Applications and Execute Queues
• Default All apps use all execute queues
• Configuring multiple execute queues can provide
additional control for applications.
• By using multiple execute queues, selected
applications have access to a fixed number of
execute threads, regardless of the load on
WebLogic Server.

How to enable
• To use user-defined execute queues include the
use81-style-execute-queues sub-element in the
config.xml file
• Validation of config.xml will fail, set
• setDomainEnv.sh
-Dweblogic.configuration.schemaValidationEnabled=false
• config.xml
server
nameWLS12cExaSvr1/name
……..
use81-style-execute-queuestrue/use81-style-execute-queues

Threads as SocketReaders
• ThreadPoolPercentSocketReaders
• Default set to 33 ( range is 1-99)
• Allocating threads as readers can increase
accepting requests

WebLogic ChunkSize
• Piece of memory for read and write data to sockets
• A pool of chunks is maintained
• For applications with large amounts of data
• Parameters are:
• weblogic.Chunksize—Size of a chunk (in bytes). Set to the network's maximum transfer
unit (MTU), after subtracting from the value any Ethernet or TCP header sizes
• weblogic.utils.io.chunkpoolsize—Sets the maximum size of the chunk pool. Monitor the
CPU profile or use a memory/ heap profiler for call stacks invoking the constructor
weblogic.utils.io.Chunk.
• weblogic.PartitionSize—Sets the number of pool partitions used (default is 4).
Partitioning the thread pool spreads the potential
for contention over more than one partition
To be set in the setDomainEnv.sh

WebLogic Backlog
• Specifies how many TCP connections can be
buffered. These are on the TCP stack but not
yet accepted by an app.
• Default is 50
• Maximum depends on O/S

WebLogic Backlog
• When to tune?
• When getting “connection refused” in high throughput
apps ( SOA composites)
• If tuning, increase by 25% of it’s default until errors
dissapear

JDBC Connectionpool tuning
• Prepared Statement Cache
• Keeps compiled SQL statements in memory
• Set to LRU
• A data source with 10 connections deployed on 2 servers,
with the Statement Cache to 10 (default), 200 cursors can be
used on the database for these cached statements
• MaxCapacity
• Max of physical connections
• Db drivers can limit connections
• Number should equal number of client sessions

Tuning Startup
• In some cases it takes ages after startup to
continue
• Generate a stackup

Tuning Startup
• This happens when using default seed
generator:
• Use of dev/urandom during startup in stead of
random
• Specifiy in startup properties -
Djava.security.egd=file:///dev/urandom
• Or in $JAVA_HOME/jre/lib/security/java.security
securerandom.source=file:/dev/./urandom

Tuning the JVM
• Most aspects of JVM tuning relate to:
• Sufficient memory heap
• An efficient garbage collection scheme
• Choice between two Unix threading models: green
and native threads

Tuning the JVM
• Most important to focus on:
• Garbage collection scheme
• Memory heap
• The Java heap is a runtime repository for live objects, dead
objects and free memory.
• The JVM heap size determines how often and how long
the VM spends collecting garbage
• Large heap size :
• full garbage collection slower and less frequent
• Small heap size
• full garbage collection is faster
and frequency is more.

• Goal is:
Tuning the JVM
• Minimize Garbage Collect
• Set heapsize that GC does no happen all the time

Tuning the JVM
• Investigate with –verbosegc option
• How often does GC run
• How long does it take?
• Full garbage collection should not take longer than 3 to 5
seconds. Lower heap if major GC time is greater.

Tuning the JVM
• Set Xms and Xmx parameters to specify initial
and Max HeapSize
• Xmx is the Java Object Heap, not the System
Heap – which is larger
• Setting these values equal will cause no
overhead of heapmanagement
• PermSize and MaxPermSize
For stored classes, methods

Let’s keep in touch!!
@MNEMONIC01
@Qualogy_news
@QAFE
Enterprise Management@@@

nl.linkedin.com/in/mschldmr
Enterprise Management@@@

http://www.qualogy.com
Enterprise Management@@@h

Oracle WebLogic Diagnostics & Perfomance tuning

More Related Content

Oracle WebLogic Diagnostics & Perfomance tuning