!"#$%&'() +%&,-&.

"(/% 0'$1 2&"/3% 45$%(6%6 789 :&"/%

Caiy Nillsap
Nethou R Coipoiation, Southlake, Texas, 0SA
http://twitter.com/CaryMillsap
cary.millsap@method-r.com
!"#$%"& ()**+)(+(,
-." /0%1 2"3"41 56&71" 08 1.$% 676"2 $% 7#7$97:9" 82"" 08 0:9$;71$04 71 .116<++/"1.0&=2>30/+&0?4907&%>
@2739"A% "B1"4&"& CDE 1273" &717 %12"7/ 30417$4% 7 9$4"72 %"F5"41$79 2"302& 08 "#"2G
&717:7%" 3799 74& "#"2G 06"271$4; %G%1"/ 3799 1.71 1." @2739" H"24"9 "B"351"% $4
2"%604%" 10 1." 30&" 1.71 G05 ?2$1"> -." 1273" I$9" 30417$4% 1$/$4;% 1.71 "47:9" G05 10
62"3$%"9G /"7%52" .0? 904; G052 30&" 17H"% 10 254 74& ?.G> -." &"17$9"& $4802/71$04
7:051 $4&$#$&579 &717:7%" 3799% 74& 06"271$4; %G%1"/ 3799% $% #$179 10 :5$9&$4; %3797:9"
7669$371$04% J7% ?"99 7% 1205:9"%.001$4; 1."/ $4 620&531$04K> -." $4802/71$04 $4 1."2"
$% 7 ;09& /$4"L 74& %0/" 08 $1 $% 7#7$97:9" 40?."2" "9%">
:;<94 2= >2?:4?:7
1. ................................................. Peifoimance anu Tiacing 1
2. ....................................................... Why You Shoulu Tiace 2
S. .................................... What Bappens When You Tiace 2
4. .......................................................... Contiolling the Tiace 2
S. ..................................................................... Session Tiacing S
6. ............................................................. Enu-to-Enu Tiacing 4
7. ................................................. Enu-to-Enu Pieiequisites 4
8. ..................................................... Finuing Youi Tiace File S
9. ...................................................... Peifoimance Intiusion 6
1u. .......................... Tiacing in Nultiplexeu Aichitectuies 6
11. ............................................. Tiacing While You Bevelop 7
12. ...................................................... Tiace File uuiueu Toui 7
1S. ............................................................. Tiace File Alphabet 9
14. .......................................... Tiace File Alphabet: BB Calls 9
1S. ........................................ Tiace File Alphabet: 0S Calls 1u
16. ................................................................... Tiace File Flow 1u
17. ........................................ Tiace File Aiithmetic: Pait 1 1u
18. ........................................ Tiace File Aiithmetic: Pait 2 11
19. ........................................ Tiace File Aiithmetic: Pait S 11
2u. ...................................................................... Tiacing Tiaps 11
21. ........................................ Commeicial Tiace File Tools 12
22. ............................................................ Acknowleugments 12
2S. ............................................................... About the Authoi 12
24. ................................................................ Revision Bistoiy 1S
2S. ................................... Appenuix: "Bello Woilu" Tiace 14
@ +4A=2A!;?>4 ;?B :A;>C?D
You click a button on a web foim. It's a button that you
click all the time. Noimally, the system you'ie using
iesponus to that click almost instantaneously, but
now. Now, it's taking moie than S seconus eveiy time
you uo it, anu sometimes it takes moie than 1u. The
obvious anu veiy simple question is: ?.G.
The vast majoiity of people who expeiience a pioblem
like this nevei finu out the answei.
Think foi a minute about what the answei ought to
look like. If you'ie a IT piofessional, then you've
piobably seen so many iepoits that the simple answei
uoesn't even occui to you. Beie's what M want: I want a
iepoit telling me wheie /G time went.
I uon't want to see how busy the whole system was
when my click took S seconus; I uon't want to see all
the "bau" SQL that executeu at oi neai the time when I
hau my pioblem; anu I uon't want to see all the
uiffeient kinus of waiting that the system was uoing
when I clickeu. I just want to see wheie /G time went,
untainteu by infoimation about anything not uiiectly
ielevant to /G iesponse time.
With 0iacle, you can have it. The featuie that gives it
to you is incluueu with eveiy ielease anu eveiy euition
of the 0iacle Batabase, fiom Expiess Euition (XE) to
Enteipiise Euition (EE), since 1992. You uon't have to
2u11 Nethou R Coipoiation. All iights ieseiveu. 1
buy an 0ption oi a Pack to use it. It's calleu "B1"4&"&
CDE 1273".
E FGH H2I 7G2I9B :A;>4
0iacle extenueu SQL tiace helps you uiagnose, it helps
you leain, anu it helps you uebug:
1

System auministiatois, uiagnosticians, anu tiouble-

shooteis will finu it invaluable foi solving pioblems
with piogiams foi which the souice is not ieauily
available (oi not easily navigable) since they uo not
neeu to be iecompileu in oiuei to tiace them.

Stuuents, hackeis, anu the oveily-cuiious will finu

that a gieat ueal can be leaineu about a system anu
its system calls by tiacing even oiuinaiy piogiams.

Anu piogiammeis will finu tiace uata veiy useful

foi bug isolation, sanity checking, anu peifoimance
optimization. Some pioblems cannot be solveu
without it.
What makes tiacing so goou compaieu to the othei
peifoimance infoimation you can get fiom 0iacle is
that with tiacing, you get a lineai sequential iecoiu of
eveiything youi 0iacle keinel piocess uoes while it's
tiying to compute the answei you've askeu foi. You
uon't have to woiiy about gaps in youi uata (like with
0iacle's Active Session Bistoiy) oi figuiing out how to
isolate youi piocess's woik in the huge pictuie of
eveiything the system is uoing iight now (like with
some of 0iacle's #N fixeu views anu all of the u0I tools
that use them).
With tiace uata, you see eveiy uatabase call youi
application makes to the 0iacle keinel, anu you see
eveiy system call youi keinel makes to its host
opeiating system. Anu it's the peifect tool foi
uevelopeis, because you uon't have to have loaus of
complicateu peimissions to use it. I believe you can
leain moie about 0iacle peifoimance with tiace uata
in a uay than you can leain stuuying #N uata foi a
month, because tiace uata so uiiectly maps to the
ieality of how the usei, the application coue, anu the
uatabase inteiact.
Tiace files aie the founuation of almost eveiything you
neeu to know about 0iacle peifoimance.
J FG;: G;++4?7 FG4? H2I :A;>4
Tiacing is an attiibute of an @2739" %"%%$04.
2
Foi each
session with tiace enableu, any @2739" H"24"9 6203"%%
S

that executes instiuctions foi that session wiites tiace
uata foi that session to the piocess's cuiiently open
tiace file. Some facts:

Tiacing is a session attiibute.

But 0iacle keinel piocesses aie what wiite youi

tiace uata.

Since moie than one piocess can execute

instiuctions foi a given 0iacle session (foi example,
by using 0iacle Paiallel Execution featuies, oi by
enabling 0iacle Shaieu Seivei), youi tiace uata can
appeai in moie than one file.

The moie exotic youi aichitectuie (foi example, the

moie tieis you use, the moie pooling featuies you
use), the moie uifficult it may be to map youi tiace
uata to a specific usei's iequest.
Exhibit 1 shows a simple case in which a usei tiaces an
0iacle session connecteu thiough %F9695%, causing an
0iacle keinel piocess to wiite tiace uata to a >123 file.
OB.$:$1 *> -." @2739" H"24"9 6203"%%L 10 ?.$3. 7
5%"2 3044"31% 1.205;. %0/" 1G6" 08 7669$371$04L
$% 1." 6203"%% 1.71 ?2$1"% 1." 1273" I$9">
K >2?:A299C?D :G4 :A;>4
You can tiace any 0iacle session on youi system if you
have peimission to execute the stanuaiu 0iacle PL
SQL package calleu &:/%P/04$102. This package gives
2u11 Nethou R Coipoiation. All iights ieseiveu. 2
1
0iacle extenueu SQL tiace is useful foi the same kinus of ieasons that the 0nix %1273" tool is useful. Ny uesciiption heie is
paiaphiaseu fiom the %1273" manual page at .116<++9$45B30//74&>02;+/74P67;"%+%1273"*>.1/9.
2
An 0iacle session is an entity with a unique %$&-%"2$79Q combination within an 0iacle instance, which you can see in #N%"%%$04.
S
An 0iacle keinel piocess is an opeiating system piocess (which you can inspect with, foi example, a Linux commanu like 6%, 9%08,
oi %1273") that executes the instiuctions iequiieu by an 0iacle session.
you a lot of contiol ovei which sessions you tiace.
Beie aie some of its pioceuuies:
%"%%$04P1273"P"47:9"
Enable tiacing foi eithei the uatabase session
fiom which the pioceuuie is calleu, oi a specific
session iuentifieu by its session iuentifiei anu
seiial numbei.
&717:7%"P1273"P"47:9"
Enable tiacing foi a specific instance oi a whole
uatabase.
0iacle pioviues a &$%7:9" pioceuuie to match each
"47:9" pioceuuie.
When you use &:/%P/04$102, I iecommenu that you
use the uefault value of waits=>true. If you use the
value waits=>false, you'll be using SQL tiace, but not
extenueu SQL tiace. You'll get a tiace of the 0iacle
uatabase calls youi application makes, but not the 0S
calls that 0iacle makes. Without a iecoiu of those 0S
calls in youi tiace file, you'll not be able to account foi
potentially significant amounts of iesponse time.
I also iecommenu that you use the non-uefault value
binds=>true. Setting binds=>true causes the 0iacle
Batabase to emit the values that the application bounu
to its placeholuei vaiiables into the tiace uata stieam.
Foi example, if someone has executeu this SQL
statement:
update salary set amount = :v1 where id = :v2
.then youi tiace uata will show what values weie
assigneu to #* anu #(. This is invaluable infoimation
foi cieating iepiouucible test cases foi solving
pioblems. Bowevei, be awaie of the uata secuiity
implication of emitting binu uata into a tiace file;
when you uo it, you aie copying potentially sensitive
uata fiom youi uatabase to youi file system.
L 7477C2? :A;>C?D
0ne of the nice things about 0iacle tiacing is that
you'ie not iequiieu to change an application in oiuei
to use it. Even if the application you'ie using is
completely un-mouifiable anu was in no way uesigneu
to coopeiate with tiacing, you still have the following
veiy poweiful tools at youi uisposal:
&:/%P/04$102>%"%%$04P1273"P"47:9"
With %"%%$04P1273"P"47:9", you can tiace any
session on youi system if you know its session iu
anu seiial numbei. You just have to be meticulous
to enable the tiace immeuiately befoie the usei
issues the slow iequest, anu then uisable the tiace
immeuiately aftei the slow iequest is fulfilleu.
Tiiggeis
Foi fast-moving sessions that uon't give you time
to iuentify theii session ius befoie they begin
uoing woik (like that Pio*C piogiam that iuns foi
2u seconus, but that shoulu iun in 2), tiiggeis aie
a convenient tool foi inseiting coue to enable
tiacing. You can, foi example, cieate an 781"2 90;04
tiiggei whose bouy uses:
session_trace_enable(
session_id => null,
serial_num => null,

)
This coue will enable tiacing foi the session that's
logging on. Theie's no neeu foi a coiiesponuing
:"802" 90;088 tiace uisablement, by the way,
because logging off automatically uisables the
tiace.
%G%P3041"B1
The %G%P3041"B1 function gives you uozens of hooks
you can use insiue of a tiiggei bouy to iuentify the
session that's logging in. Foi example,
sys_context('USERENV','IP_ADDRESS')
.ietuins the IP auuiess of the machine fiom
which the client is connecteu. With %G%P3041"B1,
you can wiite PLSQL coue in youi 781"2 90;04
tiiggei bouy that can ueciue to tiace only logins
oiiginating fiom a specific IP auuiess, only logins
iequesteu by ceitain client 0S useis, oi logins
matching any of the othei ciiteiia that %G%P3041"B1
makes available to you.
Running suspicious SQL in youi own session
Theie aie lots of ways you can iuentify suspicious
SQL statements: you'll finu them in 0EN, in youi
AWR oi ABBN iepoits, oi by queiying #N%F9%171.
Copy them anu iun them in SQL*Plus oi 0iacle
SQL Bevelopei. You can plug in whatevei binu
vaiiable values you neeu anu tiace to youi heait's
content.
4
&:/%P/04$102>&717:7%"P1273"P"47:9"
If all else fails, theie's always the nucleai bomb
option of tiacing eveiything that happens ovei
some time inteival you choose. Bewaie, howevei:
enabling tiace foi a whole instance oi uatabase
2u11 Nethou R Coipoiation. All iights ieseiveu. S
4
...Subject, natuially, to constiaints on whethei you aie peimitteu access to the objects manipulateu by the statement, anu so on.
Also, please make suie that, befoie you execute a SQL statement that might change iows in the uatabase, you uo not have an 7510=
30//$1 featuie enableu.
will only activate tiacing foi 4"? sessions that log
in aftei you enable the tiace. It won't enable
tiacing foi sessions that aie alieauy loggeu in.
M 4?BN:2N4?B :A;>C?D
Theie aie thiee majoi milestones in the histoiy of
tiacing in the 0iacle Batabase:

In 1988, 0iacle veision 6 intiouuceu CDE 1273",

which let us uiscovei the stait anu enu time of eveiy
uatabase call an application executeu.

In 199S, 0iacle veision 7 intiouuceu "B1"4&"& SQL

tiace, which let us uiscovei the uuiation of eveiy
system call the uatabase executeu.

In 2uuS, 0iacle veision 1u intiouuceu "4&=10="4&

extenueu SQL tiace, which let us activate tiacing,
not just foi sessions whose session ius we knew, but
foi sessions with specific attiibute values.
The thiiu milestoneenu-to-enu tiacingmakes it
possible foi you to specify the $41"41 to tiace a
business task in auvance of that task evei iunning.
Then, the 0iacle Batabase will automatically enable
tiacing foi any session that executes the task you have
specifieu.
The way you specify the stanuing oiuei to tiace
ceitain tasks is with the following &:/%P/04$102
pioceuuies:
39$"41P$&P1273"P"47:9"
Enable tiacing foi any uatabase session whose
client iuentifiei matches the one you specify.
%"2#P/0&P731P1273"P"47:9"
Enable tiacing foi any uatabase session whose
seivice, mouule, anu action names match the ones
you specify.
Foi example, if you wanteu to tiace any usei whose
client iu is 192.168.1.104, you'u use
dbms_monitor.client_id_trace_enable(
client_id => '192.168.1.104',
waits => true,
binds => true,
plan_stat => 'ALL_EXECUTIONS'
)
.Anu that's that. No mattei when the next peison
loggeu in fiom 192.168.1.1u4, his woik woulu be
tiaceu. If you wanteu to tiace any business task whose
seivice is calleu FINAPPS, whose mouule is calleu
GLPPOS, anu whose action is calleu Concurrent
Request, you'u execute:
dbms_monitor.serv_mod_act_trace_enable(
service_name => 'FINAPPS',
module_name => 'GLPPOS',
action_name => 'Concurrent Request'
waits => true,
binds => true,
plan_stat => 'ALL_EXECUTIONS'
)
.Then you'u be set to tiace youi next seveial uL
Posting iuns.
Befoie enu-to-enu tiacing, the haiuest thing about
tiacing was getting a tiace file that iepiesents exactly
the business task that the usei is expeiiencing iight
theie at hei keyboaiu. When you tiace a session baseu
on its session iu, you almost always captuie eithei
moie tiace uata than you ieally want, oi not enough.
That's because it takes you a little time to iuentify the
session's iu befoie you can execute the
%"%%$04P1273"P"47:9" call.
Bowevei, when you activate tiacing foi a specific
client iu oi seivice-mouule-action name combination,
you cieate a stanuing oiuei to 0iacle, instiucting the
keinel to begin tiacing any session that changes its
client iu to youi specific value, oi whenevei it changes
its seivei-mouule-action name combination to youi
specific values. Anu then 0iacle will stop tiacing
immeuiately when that session changes its iuentifying
infoimation to something uiffeient.
Puie magic.
O 4?BN:2N4?B +A4A48IC7C:47
But theie's a catch with enu-to-enu tiacing. Bow can
&:/%P/04$102 know a session's client iu oi seivice-
mouule-action names. The answei: youi application
has to pioviue those attiibute values. If youi
application uoesn't name its tasks oi set the client iu
attiibute foi its sessions in a useful way, then you can't
use the &:/%P/04$102 enu-to-enu tiacing featuies.
If you'ie using an off-the-shelf application, youi only
iecouise may be to live without the enu-to-enu tiacing
featuies (though you can still use the techniques
uesciibeu in section S) anu lobby youi venuoi to
instiument a futuie veision of youi application in the
mannei I'm about to uesciibe. If you can change youi
application souice coue youiself, instiumenting foi
enu-to-enu tiacing is not haiu to uo. The only thing
you ieally have to uo is have each session set what
0iacle calls its 5%"2 %"%%$04 .74&9" 7112$:51"%. Theie aie
seveial ways to uo it.
0ne way you can set those session hanule attiibutes
foi youi coue is to use these stanuaiu 0iacle PLSQL
packages:
2u11 Nethou R Coipoiation. All iights ieseiveu. 4
&:/%P%"%%$04
The %"1P$&"41$I$"2 pioceuuie lets you set the calling
session's client iuentifiei.
&:/%P7669$371$04P$480
The %"1P731$04 pioceuuie lets you set the calling
session's action name, anu the %"1P/0&59"
pioceuuie lets you set the calling session's mouule
anu action names togethei.
If you piefei a simplei appioach to setting youi
session hanule attiibutes, you can use the
Instiumentation Libiaiy foi 0iacle (IL0), a fiee open-
souice pioject that Nethou R Coipoiation maintains at
SouiceFoige. IL0 gives you PLSQL packages that
make it ueau simple foi an application uevelopei to
uefine wheie in an application's coue path each
business task begins anu enus. It sets all the "hooks"
that youi systems BBAs will neeu latei to tiace
specific business tasks by name. IL0 gives you seveial
extia featuies as well, which aie uetaileu in the
pioject's uocumentation.
If it botheis you to inseit auuitional uatabase calls into
youi application (anu if you'ie builuing an application
that will seive thousanus of uatabase iequests pei
seconu, it shoulu), then you shoulu choose anothei
option. Some 0iacle APIs allow you to pass seivice-
mouule-action names anu a client-iu fiom the
application to the uatabase as an 7112$:51" of each of
the uatabase calls that you weie alieauy ueteimineu
to make.
The 0iacle usei session hanule attiibutes you'll neeu
to set aie these:
OCI_ATTR_CLIENT_IDENTIFIER
OCI_ATTR_SERVICE
OCI_ATTR_MODULE
OCI_ATTR_ACTION
You can set them, foi example, with the 0iacle Call
Inteiface (0CI) @RMS112C"1 function, oi with the }BBC
%"1O4&-0O4&T"12$3% methou. These aie client-siue
function calls that uon't iequiie a iounu-tiip to the
uatabase. 0nce you have maue a call to set youi usei
session hanule attiibutes, each uatabase call that youi
application coue makes aftei that is maikeu with the
iuentifying infoimation that &:/%P/04$102 neeus.
Setting youi 0iacle usei session hanule attiibutes
youiself enables you to instiument youi application
with no extia uatabase calls at all, which means no
extia netwoik iounu-tiips between the application
anu the uatabase. It's the most scalable way to
instiument youi coue.
To summaiize, you can instiument youi application
coue foi enu-to-enu tiacing by calling stanuaiu 0iacle
PLSQL pioceuuies (oi calling them thiough someone
else's package, like IL0), oi you can uo it with no extia
uatabase calls by setting session hanule attiibutes
uiiectly with an 0iacle API like 0CI oi }BBC.
If youi application is not uesigneu to set these
attiibutes foi youi sessions, then you won't be able to
use the 39$"41P$&P1273"P"47:9" oi
%"2#P/0&P731P1273"P"47:9" pioceuuies in
&:/%P/04$102. You'll have to use only the
%"%%$04P1273"P"47:9" anu &717:7%"P1273"P"47:9"
pioceuuies insteau, until you can convince the people
who built youi application to instiument the
application so that you can measuie anu manage its
peifoimance while you opeiate it. 0ntil then, not only
will you not be able to tiace as flexibly, you also won't
be able to gioup application peifoimance statistics
using othei applications like 0iacle Enteipiise
Nanagei.
P =C?BC?D H2IA :A;>4 =C94
0nce you've tiaceu youi coue, you'll neeu to finu youi
tiace file. The piocess iunning youi 0iacle keinel coue
wiites youi tiace file to the opeiating system uiiectoiy
nameu by an 0iacle instance paiametei. In 0iacle
veision 11, the 0iacle keinel will wiite youi files into
a uiiectoiy that you can iuentify by using the following
SQL statement:
select * from v$diag_info where name='Diag Trace'
In oluei veisions of 0iacle, you can iuentify youi tiace
file uiiectoiies by using this statement:
select * from v$parameter where name in
('user_dump_dest','background_dumnp_dest')
In 0iacle veision 1u anu piioi, the 5%"2P&5/6P&"%1
uiiectoiy is piobably wheie most of youi tiace files
will be. If you use 0iacle paiallel execution featuies,
then you'll finu some of youi tiace files in the
:73H;2054&P&5/6P&"%1 uiiectoiy.
Biffeient poits of 0iacle use uiffeient naming
conventions foi tiace files. Youi tiace file names will
piobably look something like one of the following:
xe_ora_10840.trc
prod7_23389_ora.trc
ora_1492_delta1.trc
ORA01215.trc
fin1_ora_11297_POSTING.trc
MERKUR_S7_FG_ORACLE_013.trc
Tiace files may look uiffeient on uiffeient platfoims
anu on uiffeient veisions of 0iacle, but you can count
on youi tiace file names containing at least some of
the following elements:

The stiing ora.

2u11 Nethou R Coipoiation. All iights ieseiveu. S

Youi 0iacle instance name.

Youi 0iacle keinel piocess iu (on Niciosoft

Winuows, it will be youi piocess's thieau iu).

If you set a 1273"I$9"P$&"41$I$"2 within youi 0iacle

session (with an 791"2 %"%%$04 commanu), then the
stiing value of that paiametei.

The suffix .trc.

If you'ie wiiting coue that will connect to an 0iacle
instance that someone else manages, you'll neeu to
cooiuinate with that peison to get peimissions to ieau
youi tiace files. Without access to youi tiace files,
optimizing the coue you wiite is going to be a 901 moie
expensive foi youi company.
Q +4A=2A!;?>4 C?:AI7C2?
0ne thing that eveiyone always wants to know is,
"What is the peifoimance penalty of tiacing." People
get a little neivous about all those extia timei calls
that theii 0iacle keinel piocesses must execute when
you tuin tiace on.
What they uon't iealize is that theii 0iacle keinel
piocess is making all those timei calls anyway,
whethei tiacing is activateu oi not. Bow else coulu it
publish the timing infoimation you see in fixeu views
like #N%"%%171, #N%"%%$04P"#"41, anu #N%"%%$04P?7$1. The
actual peifoimance intiusion of tiacing is the cost of
the auuitional I0 calls iequiieu to peisist to youi
tiace file the infoimation 0iacle is alieauy collecting.
Nost people shoulu tiace only a few sessions at a time.
If you uon't have specializeu tools, you'll nevei in youi
life be able to sift thiough all the tiace files that an
0iacle Batabase system can geneiate in a goou, busy
houi of woik. If you want to tiace a lot of sessions at
once (anu theie aie legitimate ieasons foi wanting to),
you can tiace as many sessions as youi haiuwaie
capacity will allow. If you tiace infiequently, then you
uon't have to be too caieful about youi tiace file
uestination. If you tiace a lot of sessions
simultaneously, then just make suie you wiite youi
tiace files to a uiive aiiay that has all the stoiage
capacity 74& thioughput capacity you neeu, anu then
tiace all you want.
0nless you'ie using a veision of 0iacle that has a
tiace-ielateu peifoimance bug (0iacle bug Suu9SS9,
intiouuceu in veision 9.2.u.2, was a famous one), youi
piogiam will piobably iun with S% of the uuiation
that it takes to iun it without tiacing.
S
If a piogiam's
iesponse time gets moie than 1u% woise when you
tiace it, then you shoulu check whethei youi I0
subsystem to which you'ie wiiting tiace files is ok,
check whethei youi session's setting of %171$%1$3%P9"#"9
is what you want it to be, anu check youi 0iacle
Suppoit uatabase foi bugs.
People who ask about the peifoimance intiusion of
tiacing geneially ask because they'ie feaiful that
they'll get into tiouble if they uo anything that might
make peifoimance woise than it alieauy is. Bowevei,
youi goal in tiacing is to leain how much fastei a
piogiam ought to be iunning. Isn't that infoimation
woith at least as much as a one-time iesponse time
penalty inuuceu by a tiace. I've seen people woik on
pioblems foi months on enu without tiace uata, when
they coulu have cleaieu eveiything up with just a
single one-time tiace. Even if tiacing imposeu five
times the peifoimance penalty that it uoes, I woulun't
use it any less.
@R :A;>C?D C? !I9:C+94S4B ;A>GC:4>:IA47
I mentioneu in section S that tiacing is an attiibute of
the 0iacle session. Bowevei, many systems these uays
have many layeis of softwaie between the usei anu
the 0iacle session, anu many of those layeis employ
pooling technologies that conceal the usei's iuentity
fiom hei 0iacle session.
The solution, in geneial, is this: wheie theie's a will,
theie's a way. If you aie committeu to tiacing a
specific usei's specific business task, it can be uone.
Beie aie some of the tools you can use to uo it:
0sei session hanule attiibutes
The seivice, mouule, action, anu client iu
attiibutes aie the iight anu piopei cuie foi tiacing
in pooleu aichitectuies. When the application
iuentifies its useis to the 0iacle uatabase, the
&:/%P/04$102 package lets you enable tiacing foi
exactly the tasks you want tiaceu, anu then tools
like 0iacle's 123%"%% help you uo the woik of
concatenating the vaiious fiagments of tiace uata
that will uesciibe how youi task spent its time.
Session isolation
If youi application uoesn't set its session hanule
attiibutes that allow you to uistinguish the
business task you'ie inteiesteu in fiom the
thousanus of othei business tasks iunning at the
same time, then isolate youi usei into a
configuiation wheie you can make the uistinction.
2u11 Nethou R Coipoiation. All iights ieseiveu. 6
S
That's iight, plus oi minus. Sometimes a piogiam will iun fastei when you tiace it. It sounus nice, but it's usually not what you
want.
Foi example, insteau of youi usei connecting to
hei usual .116<++???>766>30/ site, wheie hei task
gets stiippeu of hei iuentity as it enteis some
session pool oi connection pool with a thousanu
othei tasks, have hei connect to .116<++
???>766>30/<UU, wheie hei task is the only task
in a session pool oi connection pool with a pool
size of exactly one. You can tiace eveiy session
connecting to the uatabase thiough poit 88,
which, at the time you'ie uoing youi analysis, you
know will be nobouy else but the one usei you'ie
analyzing.
@@ :A;>C?D FGC94 H2I B4T492+
The most impoitant time foi a uevelopei to tiace is
eaily in the softwaie uevelopment lifecycle, wheie
uefects aie cheapei anu easiei to fix than any othei
time in youi pioject.
6
Foitunately, tiacing is easiest in
the uevelopment enviionment.
I can iemembei since my eailiest uays of woiking with
0iacle having some kinu of %F9695% sciipt that will tuin
tiace on oi off when you call it. It coulu be as simple as
two sciipts 1273"04>%F9 anu 1273"088>%F9 that look like
this:
$ cat traceon.sql
exec dbms_monitor.session_trace_enable(binds=>true);
$
$ cat traceoff.sql
exec dbms_monitor.session_trace_disable();
When you wiite youi coue, it woiks like this:
@traceon
-- Your code goes here
@traceoff
Nowauays, I use the NR Tiace extension foi 0iacle
SQL Bevelopei. When NR Tiace is enableu, eveiy !54
C32$61 button click cieates a tiace file, which NR Tiace
automatically fetches foi you fiom youi uatabase
seivei to youi woikstation. It also lets you tag youi
local tiace files to make them easy to finu latei.
The best thing a uevelopei can uo to piomote high
peifoimance ovei an application's long lifespan is to
make it easy to tiace the application thioughout its
piouuction lifecycle. 0f couise, you uo this by couing
the application to set its own usei session hanule
attiibutes, as I've uesciibeu in section 7.
Applications woik bettei when the woilu woiks like
this:
1. Application uevelopeis tiace theii SQL anu PL
SQL coue as they wiite it. They stuuy theii tiace
files to ensuie that the coue they'ie wiiting is
efficient.
2. Bevelopeis become fluent in unueistanuing what
the tiace files mean, which iesults in moie
efficient application coue.
S. Application uevelopeis coue applications to set
theii usei session hanule attiibutes.
4. Bevelopeis anu uatabase auministiatois
paiticipate togethei uuiing integiation testing to
tiace the application coue (which is easy with the
usei session hanule attiibutes all set) anu stuuy
the tiace files to ensuie that the coue is efficient
unuei loau anu that theie aie no unwanteu
peifoimance inteiuepenuencies among competing
sessions.
S. Piouuction system auministiatois (uatabase
auministiatois, foi example) tiace the application
coue peiiouically in piouuction when the
application is behaving well, to iecoiu a baseline
of how each application task is supposeu to
peifoim (which is easy with the usei session
hanule attiibutes all set).
6. Buiing peifoimance pioblem episoues,
piouuction opeiatois tiace pioblem business
tasks (session hanule attiibutes make this easy)
anu woik togethei with application uevelopeis
who aie alieauy familiai with what the tiace file
foi the given task noimally looks liketo solve
pioblems quickly.
With habitual tiacing thioughout a softwaie lifecycle,
applications tenu to have fewei peifoimance
pioblems, anu they tenu to get fixeu moie quickly on
the iaie occasion when they uo have a pioblem. They
tenu to evolve towaiu being fast, optimizeu systems
that can last a long time. .Nuch like the 0iacle
Batabase, which is itself heavily peifoimance
instiumenteu.
@E :A;>4 =C94 DICB4B :2IA
0nce you have youi tiace file in hanu, the ieal fun
begins. Now you have a call-by-call uesciiption of what
the 0iacle Batabase uiu thioughout youi entiie
iesponse time.
If youi only inteiface to the 0iacle Batabase has been
SQL oi PLSQL executeu thiough client softwaie like
%F9695% oi 0iacle SQL Bevelopei, you may not know
about the uistinct steps that 0iacle uses to piocess a
2u11 Nethou R Coipoiation. All iights ieseiveu. 7
6
Baiiy Boehm, 1981, C081?72" O4;$4""2$4; O3040/$3%, Englewoou Cliffs N}: P T R Pientice Ball, p4u.
single statement. Exhibit 2 shows the steps fiom the
peispective of the 0iacle Call Inteiface.
OB.$:$1 (> C1"6% $4 6203"%%$4; 7 CDE %171"/"41L
820/ 1." @RM V20;27//"2W% X5$&">
You can see this sequence of activity whenevei you
tiace even a simple SQL statement like this:
select 'Hello world' from dual
An 0iacle 11.2.u.2.u tiace file foi an execution of this
SQL statement is shown in its entiiety in section 24.
Beie aie some highlights.
0n line S4, you can see the "Bello woilu" queiy itself.
Lines S6-4S show how 0iacle piocesseu the queiy:
32.=====================
33.PARSING IN CURSOR #7285244 len=30 dep=0 uid=5 oct=3
lid=5 tim=1294688002747560 hv=1604528588 ad='3d526cd8'
sqlid='guthms1gu6afc'
34.select 'Hello world' from dual
35.END OF STMT
36.PARSE ,e=70,
37.EXEC ,e=112,
38.WAIT 'SQL*Net message to client' ela= 2
39.FETCH ,e=8,,r=1,
40.STAT op='FAST DUAL ()'
41.WAIT 'SQL*Net message from client' ela= 375
42.FETCH ,e=1,,r=0,
43.WAIT 'SQL*Net message to client' ela= 1
44.WAIT 'SQL*Net message from client' ela= 1627
45.XCTEND rlbk=0,
Beie, you can see the queiy being piepaieu (the
VS!CO line), which consumeu 7u s.
7
Next, it was
executeu (iegaiuing Exhibit 2, theie was no binu
necessaiy, since my queiy uiun't contain any
placeholueis), which consumeu 112 s. Then, on
line S8, the 0iacle keinel piocess issueu an 0S ?2$1"
call to senu the iesult of the 62"672"="B"351" call paii
back to the client (which 0iacle calls CDEYZ"1 /"%%7;"
10 39$"41).
0n line S9 is a 8"13. call, which consumeu 8 s anu
ietuineu 1 iow. Line 4u shows the execution plan that
0iacle useu to piocess my queiy. Line 41 shows the
0iacle keinel blockeu on an 0S 2"7& call foi S7S s,
awaiting anothei uatabase call fiom the client (which
0iacle calls CDEYZ"1 /"%%7;" 820/ 39$"41).
0n line 42, the keinel has piocesseu a seconu 8"13.
call, consuming 1 s anu fetching no iows. Line 4S
shows that the keinel issueu anothei 0S ?2$1" call to
senu the zeio-iow iesult set back to the client (which
of couise infoims the client that theie aie no moie
iows to be hau fiom the queiy). Line 44 shows the
keinel blockeu foi 1,627 s on anothei 0S 2"7& call,
awaiting instiuctions foi what to uo next. Line 4S
shows that the next action was a 30//$1.
The iemainuei of the tiace file (shown in its entiiety
in section 24) is inteiesting, because it shows
something you might not have expecteu: the session
inseiteu a iow into %G%>75&N. Fiom this, I can tell that
my instance is set up foi session-level auuiting. I uiun't
have to woik my way uown a checklist to figuie that
out; plus, my tiace uata will show me exactly the cost
of that oveiheau upon my iesponse time.
0n line 48, you can see the SQL statement. Line Su
shows that the 62"672" call foi that statement
consumeu 9S s. 0n lines S1-1S6, you can see the
values that weie bounu into the statement's
placeholuei vaiiables (:1, :2, ., :22) within that
statement. Line 1S7 shows that the "B"351" call foi the
inseit took 1,7SS s. Line 1S8 shows the execution
plan foi the inseit, anu line 1S9 shows that the 390%"
call foi the inseit took S s. Line 16u shows that the 0S
call 0iacle uses to implement its 90; I$9" %G43 event
consumeu 1SS s, anu the final two lines of the tiace
file show two moie 390%" calls, which consumeu 9 s
anu Su s.
I hope you can see fiom just this little toui how it is
possible to use tiace uata to see exactly wheie youi
2u11 Nethou R Coipoiation. All iights ieseiveu. 8
7
Call uuiations in tiace files aie expiesseu in micioseconus, abbieviateu s. 1 s = .uuuuu1 seconus, oi 11,uuu,uuu
th
of a seconu.
time has gone.
8
You just neeu to know how to ieau
them. Let's begin with the alphabet.
@J :A;>4 =C94 ;9+G;<4:
Tiace files look plenty ugly when you fiist meet them,
but it's not too uifficult to get a hang of what they
mean. Theie aie ieally only two types of lines in the
tiace file that tell you wheie time has gone:
Batabase call lines
Batabase call lines begin with the keywoiu VS!CO,
O[OR, \O-R], RE@CO, ^ZTSV, oi C@!- ^ZTSV.
Such a line inuicates that an application client has
maue a uatabase call, anu that the 0iacle Batabase
has iesponueu to it.
System call lines
System call lines begin with the keywoiu _SM-.
Such a line inuicates that the 0iacle keinel
piocess has maue an 0S call, anu that the 0S has
iesponueu to it.
That's it; just two.
@K :A;>4 =C94 ;9+G;<4:U B< >;997
With exception of the RE@CO call, which uoesn't
contain all the fielus that the othei uatabase calls uo,
all contain the same fielus. They look like this:
FETCH #12390720:c=8000,e=1734,p=1,cr=7,cu=0,mis=0,r=1,
dep=2,og=4,plh=3992920156,tim=1294438818783887
Beie's what the fielus mean:
FETCH
This was a 8"13. call (not a 672%" oi "B"3 oi some
othei kinu).
#12390720
The call's cuisoi hanule iu is 12S9u72u. You can
seaich backwaius (towaiu the beginning) in youi
tiace file to finu the SQL oi PLSQL statement foi
which this call was maue.
c=8000
This call consumeu 8,uuu 1u,uuu s of CP0 time.
Yes, 0iacle measuieu the call as having consumeu
|faij moie CP0 time than the call's actual elapseu
uuiation.
9
e=1734
This call consumeu 1,7S4 1 s of elapseu time.
p=1
This call obtaineu 1 0iacle block via 0S ieau calls.
cr=7
This call executeu 7 "consistent moue" accesses
upon the 0iacle uatabase buffei cache.
cu=0
This call executeu u "cuiient moue" accesses upon
the 0iacle uatabase buffei cache.
mis=0
This call encounteieu u misses upon the 0iacle
libiaiy cache.
r=1
This call ietuineu 1 iow.
dep=2
Theie weie 2 levels of uatabase calls below this
call in the uatabase call stack. That is, this call is a
chilu of anothei ub call, which in tuin is a chilu of
anothei ub call.
og=4
This call's 0iacle optimizei goal is 3.00%". The
foui choices, in oiuei, aie 799, I$2%1, 259", anu
3.00%".
plh=3992920156
This call's execution plan hash value is
S99292u1S6.
tim=1294438818783887
This call completeu at time 1,294,4S8,818,78S,887
1 s. The 1$/ value is given in s, but the answei
to "s since ?.71." vaiies by platfoim. All that
ieally matteis is foi the 1$/ values to be consistent
within a given tiace file, which they usually aie.
1u
2u11 Nethou R Coipoiation. All iights ieseiveu. 9
8
You can see anothei tiace file guiueu toui in "Foi Bevelopeis: Naking Fiienus with the 0iacle Batabase foi Fast, Scalable
Applications."
9
Foi a uetaileu uesciiption of why the CP0 consumption statistic is accuiate to only 1u,uuu s , see @61$/$`$4; @2739"
V"2802/743", by Caiy Nillsap anu }eff Bolt (0'Reilly 2uuS), pp161-16S. Theie is a mistake in the book that I shoulu clean up,
howevei. 0ntil veiy iecently, I believeu that 0iacle Batabase keinels iunning on Solaiis with miciostate accounting tuineu on
woulu iesult in 3 statistic values accuiate to 1 s. Bowevei, since 0iacle on Solaiis (at least thiough 11;R2) uses the 1$/"% system
call insteau of ;"125%7;", even with miciostate accounting available fiom the 0S, 0iacle's 3 statistic is accuiate to only 1u,uuu s.
1u
They'ie most likely not to be consistent within a given file on Niciosoft Winuows systems, wheie 1$/ values wiap (that is, get as
laige as they can go anu then stait ovei at u) moie fiequently than on othei systems.
@L :A;>4 =C94 ;9+G;<4:U 27 >;997
All 0S call lines iesemble this:
WAIT #12397272: nam='db file sequential read' ela= 221
file#=1 block#=2735 blocks=1 obj#=423
tim=1294438818791494
Beie's what the fielus mean:
WAIT
This call is an 0S call, not a uatabase call.
11
#12397272
The call's cuisoi hanule iu is 12S97272.
nam='db file sequential read'
This call was a ieau of uatabase file that ietiieveu
one oi moie blocks anu stoieu them contiguously
somewheie in memoiy. 0iacle Coipoiation gives
each 0S call its keinel makes a uistinct name, anu
the company uocuments the meanings of those
names so you can moie easily unueistanu what's
going on.
ela= 221
This call consumeu 221 1 s of elapseu time.
file#=1 block#=2735 blocks=1
This ieau call (piobably a 62"7& callyou can tell
by using %1273") ieau 1 block fiom 0iacle file iu 1,
beginning at block iu 27SS. Each call has thiee
paiameteis like these, which aie nameu
uiffeiently uepenuing on the call name. These
thiee paiameteis give you contextual infoimation
about what the call was meant to accomplish.
obj#=423
This call opeiateu upon the 0iacle object with
iu 42S. When a call opeiateu upon no object in
paiticulai, the 0iacle keinel will emit obj#=-1.
tim=1294438818791494
This call completeu at time 1,294,4S8,818,791,494
1 s.
@M :A;>4 =C94 =92F
The 0iacle keinel emits tiace lines as calls complete.
You have to know this to ueteimine the paient-chilu
ielationships among calls. Foi example, what can you
know about the following (simplifieu) tiace uata
stieam.
PARSE #2:,e=1,,tim=5
EXEC #2:,e=2,,tim=8
FETCH #2:,e=4,,tim=15
PARSE #1:,e=15,,tim=17
Fiom just the uata shown heie, you can ueteimine the
stait anu enu time of each call, anu fiom that, you can
woik out the paient-chilu ielationships among these
calls:
tim call
--- --------------
2 PARSE #1 {
4 PARSE #2 {
5 }
6 EXEC #2 {
8 }
11 FETCH #2 {
15 }
17 }
Befoie 0iacle 1u.2, when theie weie no 1$/ values on
the 0S call lines, we infeiieu call paient-chilu
hieiaichy fiom the &"6 values on uatabase call lines
anu fiom the lines' ielative locations. We attiibuteu
each &"6a* call as the chilu of the &"6a) call that
followeu it in the tiace uata stieam. With 1$/ values on
all oui call lines, it is no longei necessaiy to uo that.
You can ueteimine all the paient-chilu infoimation
you neeu now just by noticing at what time each call
begins anu enus.
@O :A;>4 =C94 ;AC:G!4:C>U +;A: @
Now you know the tiace file alphabet anu how tiace
file lines flow foith at the completion of the uatabase
calls youi client application makes anu the 0S calls
youi 0iacle keinel makes. The next step is to
unueistanu a few uetails about how time woiks in
0iacle tiace files.
Fiist, a uatabase call's total uuiation (") is
appioximately the sum of the time it spenus using CP0
(3) plus the uuiation of the 0S calls maue by the call.
The equation is only appioximate, piimaiily because
of the 1u,uuu s on the CP0 time measuiement.
Expiesseu as a piecise equality, we have, foi a given
uatabase call $:
e
i
= c
i
+ ela
j
j !Children(i )
"
+ #
i
(1)
The teim A$ theie at the enu is the time within the
known uuiation of the call that is unaccounteu foi
(unexplaineu, if you piefei) by 0iacle. It can be
positive oi negative oi zeio. As much as you'u like foi
this unaccounteu-foi uuiation to be zeio, it just won't
be. Bowevei, in the aggiegate, unaccounteu-foi time
won't usually be a mateiial contiibutoi to total
iesponse time. When it is, it's an inuication that:
2u11 Nethou R Coipoiation. All iights ieseiveu. 1u
11
I've hau goou fun explaining why the woiu ?7$1 is the wiong woiu to use in this context. See foi example my essay, "Bang it,
people, they'ie syscalls, not 'waits'..."

Eithei you have foigotten to tiace the 0S calls (as

with waits=>false);

0i you have an un-patcheu 0iacle bug;

0i you have a piocess pieemption pioblem

(peihaps too much paging oi swapping).
Beie's how the aiithmetic woiks out in a two-line
tiace file exceipt:
1. WAIT #124367516: nam='Disk file operations I/O'
ela= 124 FileOperation=2 fileno=4 filetype=2 obj#=-1
tim=1294702369858445
2. FETCH #124367516:c=0,e=386,p=0,cr=6,cu=0,mis=0,r=14,
dep=0,og=1,plh=3956160932,tim=1294702369858666
Beie, you have an 0S call on line 1 that is a chilu of the
uatabase call on line 2. You can piove that by
compaiing 1$/ anu uuiation values:
tim call
---------------- ---------------------
1294702369858280 FETCH #124367516 {
1294702369858321 WAIT #124367516 {
1294702369858445 }
1294702369858666 }
Foi the uatabase call (a 8"13.), we have " = S86, anu
3 = u. The sum of the 0S call uuiations is simply 124.
0sing the equation above, this leaves an unaccounteu-
foi uuiation of A = 262. Beie's what the call's iesponse
time looks like in the foim of a piofile table:
b5271$04 Jc%K b5271$04 Jc%K R799
262 67.9% unaccounteu-foi within calls
124 S2.1% Bisk file opeiations I0
u u.u% CP0 seivice, 8"13. calls
S86 1uu.u% Response time
OB.$:$1 ,> !"%604%" 1$/" 620I$9" 802 1." 8"13.
&717:7%" 3799L 5%$4; OF571$04 *>
This is only a simple example. You can extenu the
same logic to uatabase calls that aie paients of many
moie 0S calls, oi even calls that aie paients of one oi
moie levels of othei uatabase calls.
@P :A;>4 =C94 ;AC:G!4:C>U +;A: E
The seconu ielationship you neeu to unueistanu is
that the total iesponse time uesciibeu by a tiace file is
simply the time elapseu fiom the beginning of the fiist
call to the enu of the final call. Remembei that the
0iacle keinel uoesn't emit call begin timesonly call
enu times. But you can compute the begin time of a call
by subtiacting its uuiation fiom its enu time:
R = tim
end
! tim
begin
! e
begin
( )
(2)
Note that although Equation 2 iefeis to the uuiation of
the fiist call as ", if the fiist call in the file is an 0S call,
you'll neeu to use its "97 fielu.
Be awaie that the values 1$/:";$4 anu ":";$4 iefei to
fielus associateu with the fiist call to begin in the
piocess being tiaceu. Because the 0iacle keinel emits
tiace lines when calls "4& (not when they begin), the
fiist call to begin is not necessaiily the fiist call listeu
in the tiace file.
@Q :A;>4 =C94 ;AC:G!4:C>U +;A: J
The final categoiy of iesponse time to uesciibe is the
time that is unaccounteu-foi between the top-level
uatabase anu 0S calls. This one is simple:
!
between
= R " e
i
i #TopLevelCalls
$
(S)
That is, the total between-call unaccounteu-foi
uuiation is the total iesponse time that we know has
passeu minus the time accounteu foi by top-level calls
(both uatabase anu 0S) in the tiace file. So Equation 1
gives us most of the uetail that you neeu to make up
youi piofile. Equation 2 gives you the total iesponse
time that you neeu to account foi. Anu Equation S
gives you the label to which to assign the iemainuei of
the unexplaineu uuiation. These aie the thiee
elements you neeu to cieate a complete iesponse time
piofile.
ER :A;>C?D :A;+7
When peifoimance is the question, tiacing is usually
my answei. Bowevei, it's not always easy. Beie aie
some of the pitfalls to avoiu.
!"#" %&''(%#)&*+ By fai, the most common pioblem we
encountei in the fielu is the tiace file whose iesponse
time uoesn't match the iesponse time of the enu usei's
expeiience that you'ie tiying to uiagnose. We get a lot
of tiace files whose oveiall iesponse times aie
uominateu by eithei a small numbei of long-uuiation
CDEYZ"1 /"%%7;" 820/ 39$"41 calls oi a small numbei of
unaccounteu-foi gaps in the action. The most
impoitant thing to iemembei about collecting tiace
uata is you want the iesponse time of youi tiace file to
exactly match the iesponse time of youi enu usei's
expeiience.
,-"%'( .)%-&/(%&*0/+ A micioseconu is
11,uuu,uuu
th
of a seconu, oi 1,uuu nanoseconus. It
may suipiise you to leain that, on some platfoims, an
0iacle micioseconu is 1,u24 nanoseconus, oi
1976,S6S
th
of a seconu. Being off by 2.4% may not
sounu like too big of a pioblem, but if you want to
2u11 Nethou R Coipoiation. All iights ieseiveu. 11
iegaiu an 0iacle 1$/ value as a numbei of
micioseconus that have elapseu since the 0nix epoch
(197u-u1-u1Tuu:uu:uu.uuuuuuZ in IS0 86u1 foim),
you neeu to multiply youi 1$/ value by 1.u24 befoie
calculating the uate anu time it iepiesents. 0theiwise,
foi 1$/ values geneiateu in 2u11, you will be off by
almost a yeai.
,-"%'( 123/+ The 0iacle 11.1 eia has not been kinu to
tiace file useis. Bug 7S22uu2 causes uatabase call time
values to iun off a 1,u24-nanoseconu-pei-micioseconu
clock anu 0S call time values to iun off a 1,uuu-
nanoseconu-pei-micioseconu clock. This causes the
sequence of calls in a tiace file to zigzag back anu foith
in time. This pioblem is coiiectible, but then bug
8S42S29 isn't.
12
It causes the 0iacle keinel to emit the
same 1$/ value iepeateuly foi multiple uistinct
uatabase calls, which is uebilitating. Bappily, these
bugs in 11.1 have patches anu aie fixeu in veision
11.2.u.2.u.
E@ >2!!4A>C;9 :A;>4 =C94 :2297
Theie's geneially too much uetail in a tiace file to look
at all at once. To make sense out of what's theie
iequiies softwaie assistance. The most famous tiace
file softwaie tool of all is 0iacle Coipoiation's 1H6208
utility, which 0iacle ships with the 0iacle Batabase
(all ieleases, all euitions). -H6208 aggiegates by SQL
statement, anu it allows you to soit those SQL
statements in its output iepoit by any of seveial
ciiteiia. This can woik out ok, as long as:

You caie about inefficient SQL moie than you caie

about iesponse time (which is often a mistake).

You soit by the iight thing

(sort=prsela,exeela,fchela is almost always the
iight thing).

Youi pioblem uoesn't involve paient-chilu

ielationships among uatabase calls (so-calleu
iecuisive SQL).
I useu 1H6208 extensively foi the fiist ten yeais of my
caieei. Bowevei, I spent a lot of time looking at iaw
tiace uata, too, because theie's a lot of infoimation
insiue a tiace file that 1H6208 ignoies, anu the foimat of
the 1H6208 output is not paiticulaily fiienuly to my
iesponse time focus. So, I've helpeu to uesign anu
wiite a few softwaie tools in the past ten yeais,
incluuing:
Nethou R Piofilei
The Nethou R Piofilei is softwaie that }eff Bolt
anu I wiote about in oui book @61$/$`$4; @2739"
V"2802/743" (0'Reilly 2uuS). It piouuces an BTNL
iepoit that unambiguously explains the iesponse
time accounteu in a tiace file, no mattei what the
cause (whethei SQL oi not), using coloi to uiiect
youi attention.
/2%H"?
The Nethou R skew analyzei is a uata-mining tool
foi tiace files. It can uo neaily eveiything the
Piofilei can uo, anu moie. With /2%H"?, you can
gioup by pietty much anything in a tiace file to
uiill with piactically any analysis peispective you
want. Foi example, you can gioup a whole
uiiectoiy full of tiace files by SQL statement hash
values, oi uatabase file ius, oi hash latch auuiess
values, oi libiaiy cache miss counts, oi block
counts pei ieau call, oi seivice-mouule-action
names, oi 62"672" call uuiations, .anything you
like.
EE ;>V?2F94BD!4?:7
Thank you to }eff Bolt, Ken Feilita, Baiolu Palacio, anu
Ron Ciisco at Nethou R Coipoiation foi theii help in
piepaiing this woik, anu to Anuiew Zitelli of Thales
Raytheon Company foi shaiing his ieseaich with us.
EJ ;<2I: :G4 ;I:G2A
Caiy Nillsap is the founuei anu piesiuent of Nethou R
Coipoiation, a small business that builus anu
optimizes softwaie all ovei the woilu (.116<++/"1.0&=
2>30/). Caiy uesigns anu wiites softwaie anu
euucational mateiial. Be is the authoi of 0ptimizing
0iacle Peifoimance (0'Reilly 2uuS), foi which he anu
Nethou R colleague }eff Bolt weie nameu @2739"
T7;7`$4"'s Authois of the Yeai. Be has piesenteu at
hunuieus of confeiences anu couises woiluwiue, anu
he is also publisheu in R0//54$371$04% 08 1." SRT.
Caiy spent the 199us as a consultant anu leauei of
consulting teams within 0iacle Coipoiation. In 1999,
Caiy iesigneu as vice piesiuent of 0iacle's System
Peifoimance uioup to stait a caieei as a business
ownei. Caiy is an 0iacle ACE Biiectoi anu a founuing
paitnei of the 0ak Table Netwoik, an infoimal
association of "0iacle scientists" that aie well known
thioughout the 0iacle community. Caiy blogs at
.116<++372G/$99%76>:90;%601>30/ anu tweets at .116<++
1?$11"2>30/+R72GT$99%76.
2u11 Nethou R Coipoiation. All iights ieseiveu. 12
12
See also patches 941S42S foi 0iacle 11.1.u.7.1 on Solaiis anu 9941786 foi 0iacle 11.1.u.7.4 on Solaiis.
EK A4TC7C2? GC7:2AH
2u11u11S
Initial ievision.
2u11u22S
Incoipoiateu uiscoveiy by Alex uoibachev,
ieseaicheu by }eff Bolt anu Anuiew Zitelli, noting
that even with Solaiis miciostate accounting, the
3 statistic is accuiate only to 1u,uuus.
Incoipoiateu feeuback fiom Naik Fainham into
the footnote about tiacing "to youi heait's
content."
2u11 Nethou R Coipoiation. All iights ieseiveu. 1S
EL ;++4?BCSU WG4992 F2A9BX :A;>4
1. Trace file /opt/oracle/diag/rdbms/v11202/V11202/trace/V11202_ora_21089.trc
2. Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
3. With the Partitioning, OLAP, Data Mining and Real Application Testing options
4. ORACLE_HOME = /opt/oracle/product/11.2.0.2
5. System name: Linux
6. Node name: rhel01.dev.method-r.com
7. Release: 2.6.18-92.1.13.el5xen
8. Version: #1 SMP Thu Sep 4 04:20:55 EDT 2008
9. Machine: i686
10. VM name: Xen Version: 3.1 (PVM)
11. Instance name: V11202
12. Redo thread mounted by this instance: 1
13. Oracle process number: 30
14. Unix process pid: 21089, image: oracle@rhel01.dev.method-r.com (TNS V1-V3)
15.
16.
17. *** 2011-01-10 13:33:22.746
18. *** SESSION ID:(19.1247) 2011-01-10 13:33:22.746
19. *** CLIENT ID:() 2011-01-10 13:33:22.746
20. *** SERVICE NAME:(SYS$USERS) 2011-01-10 13:33:22.746
21. *** MODULE NAME:(SQL*Plus) 2011-01-10 13:33:22.746
22. *** ACTION NAME:() 2011-01-10 13:33:22.746
23.
24. =====================
25. PARSING IN CURSOR #7285244 len=60 dep=0 uid=5 oct=47 lid=5 tim=1294688002745892 hv=1548041990 ad='3d51df50'
sqlid='31wf1v9f4ags6'
26. BEGIN dbms_monitor.session_trace_enable(binds=>true); END;
27. END OF STMT
28. EXEC #7285244:c=0,e=845,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=1294688002745889
29. WAIT #7285244: nam='SQL*Net message to client' ela= 2 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=1294688002746391
30. WAIT #7285244: nam='SQL*Net message from client' ela= 931 driver id=1650815232 #bytes=1 p3=0 obj#=-1
tim=1294688002747354
31. CLOSE #7285244:c=0,e=55,dep=0,type=0,tim=1294688002747449
32. =====================
33. PARSING IN CURSOR #7285244 len=30 dep=0 uid=5 oct=3 lid=5 tim=1294688002747560 hv=1604528588 ad='3d526cd8'
sqlid='guthms1gu6afc'
34. select 'Hello world' from dual
35. END OF STMT
36. PARSE #7285244:c=0,e=70,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=1294688002747559
37. EXEC #7285244:c=0,e=112,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=1294688002747793
38. WAIT #7285244: nam='SQL*Net message to client' ela= 2 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=1294688002747865
39. FETCH #7285244:c=0,e=8,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=1388734953,tim=1294688002747910
40. STAT #7285244 id=1 cnt=1 pid=0 pos=1 obj=0 op='FAST DUAL (cr=0 pr=0 pw=0 time=4 us cost=2 size=0 card=1)'
41. WAIT #7285244: nam='SQL*Net message from client' ela= 375 driver id=1650815232 #bytes=1 p3=0 obj#=-1
tim=1294688002748453
42. FETCH #7285244:c=0,e=1,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=1388734953,tim=1294688002748493
43. WAIT #7285244: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=1294688002748520
44. WAIT #7285244: nam='SQL*Net message from client' ela= 1627 driver id=1650815232 #bytes=1 p3=0 obj#=-1
tim=1294688002750168
45. XCTEND rlbk=0, rd_only=1, tim=1294688002750230
46. =====================
47. PARSING IN CURSOR #7386176 len=447 dep=1 uid=0 oct=2 lid=0 tim=1294688002750383 hv=1097020010 ad='3d7e77d4'
sqlid='f711myt0q6cma'
48. insert into sys.aud$( sessionid,entryid,statement,ntimestamp#, userid,userhost,terminal,action#,returncode, logoff
$lread,logoff$pread,logoff$lwrite,logoff$dead, logoff$time,comment$text,spare1,clientid,sessioncpu,proxy$sid,user$guid,
instance#,process#,auditid,dbid) values(:1,:2,:3,SYS_EXTRACT_UTC(SYSTIMESTAMP), :4,:5,:6,:7,:8, :9,:10,:11,:12,
cast(SYS_EXTRACT_UTC(systimestamp) as date),:13,:14,:15,:16,:17,:18, :19,:20,:21,:22)
49. END OF STMT
50. PARSE #7386176:c=0,e=93,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=1294688002750382
51. BINDS #7386176:
52. Bind#0
53. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
54. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
55. kxsbbbfp=0070ac04 bln=22 avl=04 flg=05
56. value=63843
57. Bind#1
58. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
59. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
60. kxsbbbfp=0070abe0 bln=24 avl=02 flg=05
2u11 Nethou R Coipoiation. All iights ieseiveu. 14
61. value=2
62. Bind#2
63. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
64. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
65. kxsbbbfp=0070abbc bln=24 avl=02 flg=05
66. value=1
67. Bind#3
68. oacdty=01 mxl=32(06) mxlc=00 mal=00 scl=00 pre=00
69. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
70. kxsbbbfp=006df248 bln=32 avl=06 flg=09
71. value="SYSTEM"
72. Bind#4
73. oacdty=01 mxl=32(23) mxlc=00 mal=00 scl=00 pre=00
74. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
75. kxsbbbfp=006df1c2 bln=32 avl=23 flg=09
76. value="rhel01.dev.method-r.com"
77. Bind#5
78. oacdty=01 mxl=32(05) mxlc=00 mal=00 scl=00 pre=00
79. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
80. kxsbbbfp=635ffd04 bln=32 avl=05 flg=09
81. value="pts/0"
82. Bind#6
83. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
84. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
85. kxsbbbfp=0070ab98 bln=24 avl=03 flg=05
86. value=101
87. Bind#7
88. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
89. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
90. kxsbbbfp=0070ab74 bln=24 avl=01 flg=05
91. value=0
92. Bind#8
93. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
94. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
95. kxsbbbfp=0070ab40 bln=24 avl=02 flg=05
96. value=44
97. Bind#9
98. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
99. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
100. kxsbbbfp=0070ab1c bln=24 avl=01 flg=05
101. value=0
102. Bind#10
103. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
104. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
105. kxsbbbfp=0070aaf8 bln=24 avl=02 flg=05
106. value=4
107. Bind#11
108. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
109. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
110. kxsbbbfp=0070aad4 bln=24 avl=01 flg=05
111. value=0
112. Bind#12
113. oacdty=01 mxl=32(00) mxlc=00 mal=00 scl=00 pre=00
114. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
115. kxsbbbfp=00000000 bln=32 avl=00 flg=09
116. Bind#13
117. oacdty=01 mxl=32(06) mxlc=00 mal=00 scl=00 pre=00
118. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
119. kxsbbbfp=635ffc98 bln=32 avl=06 flg=09
120. value="oracle"
121. Bind#14
122. oacdty=01 mxl=32(00) mxlc=00 mal=00 scl=00 pre=00
123. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
124. kxsbbbfp=00000000 bln=32 avl=00 flg=09
125. Bind#15
126. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
127. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
128. kxsbbbfp=0070aab0 bln=24 avl=02 flg=05
129. value=1
130. Bind#16
131. oacdty=02 mxl=22(00) mxlc=00 mal=00 scl=00 pre=00
132. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
2u11 Nethou R Coipoiation. All iights ieseiveu. 1S
133. kxsbbbfp=0070aa8c bln=24 avl=00 flg=05
134. Bind#17
135. oacdty=01 mxl=32(00) mxlc=00 mal=00 scl=00 pre=00
136. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
137. kxsbbbfp=00000000 bln=32 avl=00 flg=09
138. Bind#18
139. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
140. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
141. kxsbbbfp=0070aa58 bln=24 avl=01 flg=05
142. value=0
143. Bind#19
144. oacdty=01 mxl=32(05) mxlc=00 mal=00 scl=00 pre=00
145. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
146. kxsbbbfp=bfc1b840 bln=32 avl=05 flg=09
147. value="21089"
148. Bind#20
149. oacdty=01 mxl=32(00) mxlc=00 mal=00 scl=00 pre=00
150. oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
151. kxsbbbfp=00000000 bln=32 avl=00 flg=09
152. Bind#21
153. oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
154. oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
155. kxsbbbfp=0070a7f8 bln=24 avl=06 flg=05
156. value=518865609
157. EXEC #7386176:c=0,e=1733,p=0,cr=1,cu=2,mis=0,r=1,dep=1,og=4,plh=0,tim=1294688002752264
158. STAT #7386176 id=1 cnt=0 pid=0 pos=1 obj=0 op='LOAD TABLE CONVENTIONAL (cr=1 pr=0 pw=0 time=155 us)'
159. CLOSE #7386176:c=0,e=5,dep=1,type=0,tim=1294688002752341
160. WAIT #0: nam='log file sync' ela= 153 buffer#=5673 sync scn=1046841 p3=0 obj#=-1 tim=1294688002752632
161. CLOSE #7285244:c=0,e=9,dep=0,type=0,tim=1294688002752690
162. CLOSE #7248564:c=0,e=30,dep=0,type=0,tim=1294688002752740
2u11 Nethou R Coipoiation. All iights ieseiveu. 16