USING STATSPACK TO TRACK DOWN BAD CODE

Michael R. Ault – Burleson Consulting

Introduction
Many times a developer may be given the task of helping the DBA find and resolve an applications poorly performing code.
Of course first one must define what constitutes poor performance. Is poor performance a certain number of logical or
physical IO’s? Is it a certain number of consistent reads? I believe it needs to be defined in the context of the specific
application. For example, in an order entry situation inserts should be expected to be sub-second. Another example would be
that in a customer service application the customer information screen should be expected to be populated within less than
seven seconds. Yet another would be that a decision support system needs to return results within 15 minutes. Each of these
applications has different expectations for performance and each must be tuned using that expectation set in mind.
Another key concept when tuning is the concept of enough is enough. This concept means to set specific tuning goals and
when you reach them, go on to the next problem. Tuning Oracle has been likened to a video game with infinite levels, there is
always a way to get a few more milliseconds or microseconds of performance from Oracle, you have to know when to quit!
In this presentation we will look at using the statspack, and by extension the AWR, tool for finding and correcting bad SQL
in an application.

Setting Up Statspack
Before you use statspack it must be installed. This is fairly simple to do. To install statspack you must be on release 8.1.7 of
Oracle or later (you can go back as far as 8.1.5 but results can be interesting with these older releases) also, be very careful
about what you have running, we have seen issues with early versions of ColdFusion and Statspack colliding. Anyway, the
steps to install are:
1. Make sure dbms_jobs and dbms_shared_pool are installed in the system.
2. Review the spcreate.sql series of called scripts and eliminate the calls to install the packages in step 1.
3. Create a perfstat tablespace
4. Run the spcreate.sql script, usually in the $ORACLE_HOME/rdbms/admin directory
5. Use the statspack.snap procedure to test the install
6. Start automated statistics runs with spauto.sql
The spcreate.sql script runs a series of other scripts that creates the appropriate user, gives them the proper grants and builds
the proper tables and packages for the statspack process. By default it create the perfstat user with the password perfstat. Note
in step 2 we suggest editing the appropriate script to remove the build/rebuild of the dbms_jobs and dbsm_shared_pool. This
is to prevent locking issues should these packages already be installed. It is strongly suggested that these be prebuilt so you
can ensure there are no issues.
Once statspack is installed, you can begin testing.

AWR Usage
In 10g we now have AWR. The Automatic Workload Repository (AWR) defaults to a collection interval every 30 minutes
and collects data that is the foundation for all of the other self-tuning features. AWR is very much like STATSPACK,
especially the level-5 STATSPACK collection mechanism where top SQL is collected every hour, based on your rolling
thresholds for high-use SQL. In addition to the SQL, AWR collects detailed run-time statistics on the top SQL (disk reads,
executions, consistent gets) and uses this information to adjust the rolling collection threshold. This technique ensures that
AWR always collects the most resource intensive SQL. The AWR system provides reports that can be run against the AWR
tables that provide the same type of data a statspack report used to, and, much more. AWR takes advantage of the ADDM
(Adam) to gather its statistics using non-SQL based collection techniques which is much more efficient than the old SQL

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

based statspack and less intrusive on the database. The awrrpt.sql script is used to generate the reports that take the place of
the Statspack reports in 10g.

Testing Using Statspack or AWR

Statspack and AWR are simply statistics capture tools that also provide a report that shows you performance related statistics.
Part of the statistics are several listings of SQL statements sliced and diced by number of consistent gets, number of physical
reads, number of parses, number of executions and number of versions. Of course for a statement to appear there, it must
have been run!
The prime thing to remember when testing with statspack is that the statspack process will only capture what is currently
going on in the database. It takes snapshots of the current state and allows you to choose two snapshots to compare. IN the
case of the SQL statements, if they are still in the shared pool between the two statspack runs, then your changes may be
masked by old code runs. Therefore for testing purposes you need to flush the shared pool, execute a snapshot, then test your
changes and execute another snapshot:
1. Use the “ALTER SYSTEM FLUSH SHARED_POOL;” command to flush old SQL form the pool.
2. Execute the command “execute statspack.snap;” to begin a statspack capture window.
3. Run your test code
4. Execute the command “execute statspack.snap;” to end the statspack capture window.
5. Run the spreport.sql script to generate a report based on the time interval between steps 2 and 4.
If on the other hand, you don’t already know what SQL is causing issues, which may be the case in a large system with the
general user complaint “Everything is Slow!” use the automated statspack gathering (spauto.sql) to capture a profile of
statspack runs across the time periods when the users have the problems. Running statspack off hours probably won’t tell you
a lot, you need to run it when the problem is occurring!

Evaluating Statspack or AWR for Code Issues

The statspack and AWR reports have a number of sections devoted entirely to code. There are also sections devoted to waits
and one to latches. Using the cross-mix between waits, latches and the reported SQL the developer can isolate and then
correct the problem SQL. Generally you will start with the major hitters, SQL that shows up in more than one of the major
areas relegated to SQL. For example, if a SQL shows up in the top ten in both the consistent gets and disk reads area then it is
probably a good candidate for optimization.
For this paper we will be using example outputs from Statspack reports.
However, if you are looking at code in the statspack report that appears to be a problem but it is not used all the time or is an
ad hoc query, then it may not be an issue unless it has the potential to become a major use SQL. Let’s look at some actual
statspack outputs and see what we can determine from them about the code they contain. Look at Figure 1.
Snap Id Snap Time Sessions Curs/Sess Comment
------- ------------------ -------- --------- -------------------
Begin Snap: 1240 26-May-05 15:00:02 34 5.0
End Snap: 1241 26-May-05 16:00:02 33 4.9
Elapsed: 60.00 (mins)

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
db file sequential read 320,448 4,588 32.25
direct path read 58,652 2,988 21.00
CPU time 2,182 15.34
PX Deq: Execute Reply 1,428 1,257 8.83
db file scattered read 11,406 1,020 7.17
-------------------------------------------------------------

SQL ordered by Gets for DB: TEST10 Instance: TEST10 Snaps: 1240 -1241
-> End Buffer Gets Threshold: 10000
-> Note that resources reported for PL/SQL includes the resources used by

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

all SQL statements called within the PL/SQL code. As individual SQL
statements are also reported, it is possible and valid for the summed
total % to exceed 100

CPU Elapsd
Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
1,340,065 1 1,340,065.0 3.3 27.85 27.75 2183036294
Module: SQLNav5.exe
SELECT /*+ FIRST_ROWS */ KEY||','||WRTN_PREM||','||WRTN_EXPSR||'
,'||EARNED FROM ( SELECT /*+ INDEX(pcoopt PK_POL_COVG_ON_OF
F_PREM_TRAN) */ pcoopt.pol_id||','||pcoopt.pol_tran_id
||','||pcoopt.veh_unit_nbr||','|| pcoopt.covg_mp_cd||'
,'||pcoopt.covg_cd KEY, SUM(pcoopt.veh_covg_wrtn_prem_

1,278,490 1 1,278,490.0 3.2 11.15 12.26 656581566

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo
pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_
POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||
pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||

972,208 1 972,208.0 2.4 10.10 13.08 3656618493

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

796,391 20,322 39.2 2.0 90.85 3506.03 2624188903

Module: SQL*Plus
SELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt
FROM pol p1, pol_tran t1 WHERE p1.p
ol_nbr = :b2 AND p1.ign_sys_cd =
:b1 AND t1.pol_id = p1.pol_id A
ND t1.pol_tran_typ_cd = 'CN' AND

707,029 1 707,029.0 1.7 7.01 6.84 324622422

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

113,134 19,300 5.9 0.3 2.96 3.06 1584996842

Module: SQL*Plus
SELECT /*+ INDEX(p3 iu_pol_isc) */ MAX(p3.pol_expr_dt)
FROM pol p3 WHERE p3.pol_n
br = :b2 AND p3.ign_sys_cd = :b1

55,151 6 9,191.8 0.1 3.62 85.42 2433730562

Module: jre@testit (TNS V1-V3)
/* OracleOEM */ SELECT d.status "Status", d.tablespace_name "Nam
e", d.contents "Type", d.extent_management "Extent Management",
TO_CHAR(NVL(a.bytes / 1024 / 1024, 0),'99,999,990.900') "Size (M
)", TO_CHAR(NVL(a.bytes - NVL(f.bytes, 0), 0)/1024/1024,'9999999
9.999') ||'/'||TO_CHAR(NVL(a.bytes/1024/1024, 0), '99999999.999'

47,895 1 47,895.0 0.1 27.18 577.44 1017018803

Module: jre@testit (TNS V1-V3)
/* OracleOEM */ SELECT DL.SEGMENT_FILEID, DL.SEGMENT_BLOCK, F.FI
LE# FNO, DL.BLOCK BNO, DL.LENGTH, DL.EXTENT_ID FROM SYS.DBA_LMT_
USED_EXTENTS DL, SYS.FILE$ F WHERE DL.TABLESPACE_ID = :1 AND DL.
TABLESPACE_ID = F.TS# AND DL.FILEID = F.RELFILE# UNION ALL SELEC
T -1, -1, F.FILE# FNO, DL.BLOCK_ID BNO, DL.BLOCKS, -1 FROM SYS.D

38,004 4 9,501.0 0.1 0.45 0.49 4287766396

Module: SQLNav5.exe

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

select a.column_name,a.column_length,a.column_position,b.column_
expression,decode(a.descend,'ASC','N','Y') descend,a.index_owner
object_owner,a.index_name object_name from sys.dba_ind_columns
a,sys.dba_ind_expressions b where a.index_owner=:schema and a.in
dex_name=:object_name and b.index_owner(+)=a.index_owner and b.i

-------------------------------------------------------------

SQL ordered by Reads for DB: TEST10 Instance: TEST10 Snaps: 1240 -1241
-> End Disk Reads Threshold: 1000

CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
90,051 20,322 4.4 1.9 90.85 3506.03 2624188903
Module: SQL*Plus
SELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt
FROM pol p1, pol_tran t1 WHERE p1.p
ol_nbr = :b2 AND p1.ign_sys_cd =
:b1 AND t1.pol_id = p1.pol_id A
ND t1.pol_tran_typ_cd = 'CN' AND

9,752 1 9,752.0 0.2 27.18 577.44 1017018803

Module: jre@testit (TNS V1-V3)
/* OracleOEM */ SELECT DL.SEGMENT_FILEID, DL.SEGMENT_BLOCK, F.FI
LE# FNO, DL.BLOCK BNO, DL.LENGTH, DL.EXTENT_ID FROM SYS.DBA_LMT_
USED_EXTENTS DL, SYS.FILE$ F WHERE DL.TABLESPACE_ID = :1 AND DL.
TABLESPACE_ID = F.TS# AND DL.FILEID = F.RELFILE# UNION ALL SELEC
T -1, -1, F.FILE# FNO, DL.BLOCK_ID BNO, DL.BLOCKS, -1 FROM SYS.D

1,619 6 269.8 0.0 3.62 85.42 2433730562

Module: jre@testit (TNS V1-V3)
/* OracleOEM */ SELECT d.status "Status", d.tablespace_name "Nam
e", d.contents "Type", d.extent_management "Extent Management",
TO_CHAR(NVL(a.bytes / 1024 / 1024, 0),'99,999,990.900') "Size (M
)", TO_CHAR(NVL(a.bytes - NVL(f.bytes, 0), 0)/1024/1024,'9999999
9.999') ||'/'||TO_CHAR(NVL(a.bytes/1024/1024, 0), '99999999.999'

1,466 1 1,466.0 0.0 10.10 13.08 3656618493

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

852 1 852.0 0.0 11.15 12.26 656581566

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo
pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_
POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||
pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||

443 1 443.0 0.0 1.65 4.51 2892151655

Module: SQL*Plus
BEGIN statspack.snap(i_snap_level=>6); END;

75 4 18.8 0.0 0.30 1.47 2243218285

Module: SQLNav5.exe
select :schema owner,a.object_name,a.object_id,a.created,a.last_
ddl_time,decode(a.status,'VALID',0,'INVALID',1,2) status,decode(
b.partitioned,'YES','Y','NO','N') partitioned,'N' object_table,'
N' external_table,decode(b.nested,'YES','Y','N') nested,decode(b
.IOT_Type,'IOT',1,'IOT_OVERFLOW',2,0) IOT_Type,b.IOT_Name,b.temp

62 326 0.2 0.0 0.81 1.46 1604489463

Module: jre@testit (TNS V1-V3)
select file#, block#, type#, nvl(spare1,0), hwmincr, cachehint
from seg$ where ts# = :1

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

54 1 54.0 0.0 0.27 0.65 4119687667

Module: jre@testit (TNS V1-V3)
/* OracleOEM */ SELECT -1, RELATIVE_FNO, HEADER_BLOCK, SEGMENT_T
YPE, OWNER, SEGMENT_NAME, PARTITION_NAME, EXTENTS, BLOCKS, MIN_E
XTENTS, MAX_EXTENTS, (INITIAL_EXTENT/1024), (NEXT_EXTENT/1024),
PCT_INCREASE FROM SYS.DBA_SEGMENTS WHERE TABLESPACE_NAME = :1 OR
DER BY 2, 3

21 19 1.1 0.0 0.04 0.60 4080861370

select owner#,name,namespace,remoteowner,linkname,p_timestamp,p_
obj#, d_owner#, nvl(property,0),subname from dependency$,obj$ wh
ere d_obj#=:1 and p_obj#=obj#(+) order by order#

Figure 1: Basic Sections from Statspack

From the data in Figure 1, we first need to determine if there are problems. We can see that the duration is good (one hour)
and we can see that our majority waits are also IO related (db file sequential and scattered reads, direct reads) these type of
waits show we are seeing large numbers of single and multiblock IOs as well as probable sorts (the direct IO events.) This
clues us to look at the physical reads to see what SQL is occurring there. Look at Figure 2.
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
90,051 20,322 4.4 1.9 90.85 3506.03 2624188903
Module: SQL*Plus
SELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt
FROM pol p1, pol_tran t1 WHERE p1.p
ol_nbr = :b2 AND p1.ign_sys_cd =
:b1 AND t1.pol_id = p1.pol_id A
ND t1.pol_tran_typ_cd = 'CN' AND

1,466 1 1,466.0 0.0 10.10 13.08 3656618493

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

852 1 852.0 0.0 11.15 12.26 656581566

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo
pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_
POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||
pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||

Figure 2: SQLs with Large IOs

The three SQL statements in Figure 2 are application SQL statements, the other of the top 10 are all from monitoring tools.
Of the three, two are limited to only a single execution (whether it is from our period or fell outside of it, is hard to tell)
however we have one SQL that was executed a whopping 20,322 times. Even though it only does 4.4 reads per execution, it
does a great number of these. By tuning this SQL we can reduce the physical IO requirements of the application. That we are
getting lots of small reads (4.4 IOs per execution) indicates that this may be the source of many of our sequential read waits
reported in this period. Generally sequential read waits can be mitigated by more cache memory, we may not need to tune
this query.
The scattered read waits are probably due to the monitoring SQL being generated by the monitoring reads. The problem here
may also be memory related as the memory may not be large enough to hold the data needed by the application.
Let’s look further into our SQL examples here by looking at the application code that is doing the most consistent gets next.
Look at Figure 3.
Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
1,340,065 1 1,340,065.0 3.3 27.85 27.75 2183036294
Module: SQLNav5.exe
SELECT /*+ FIRST_ROWS */ KEY||','||WRTN_PREM||','||WRTN_EXPSR||'
,'||EARNED FROM ( SELECT /*+ INDEX(pcoopt PK_POL_COVG_ON_OF

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

F_PREM_TRAN) */ pcoopt.pol_id||','||pcoopt.pol_tran_id
||','||pcoopt.veh_unit_nbr||','|| pcoopt.covg_mp_cd||'
,'||pcoopt.covg_cd KEY, SUM(pcoopt.veh_covg_wrtn_prem_

1,278,490 1 1,278,490.0 3.2 11.15 12.26 656581566

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo
pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_
POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||
pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||

972,208 1 972,208.0 2.4 10.10 13.08 3656618493

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

796,391 20,322 39.2 2.0 90.85 3506.03 2624188903

Module: SQL*Plus
SELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt
FROM pol p1, pol_tran t1 WHERE p1.p
ol_nbr = :b2 AND p1.ign_sys_cd =
:b1 AND t1.pol_id = p1.pol_id A
ND t1.pol_tran_typ_cd = 'CN' AND

707,029 1 707,029.0 1.7 7.01 6.84 324622422

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

113,134 19,300 5.9 0.3 2.96 3.06 1584996842

Module: SQL*Plus
SELECT /*+ INDEX(p3 iu_pol_isc) */ MAX(p3.pol_expr_dt)
FROM pol p3 WHERE p3.pol_n
br = :b2 AND p3.ign_sys_cd = :b1
Figure 3: Extracted Consistent Gets SQL
From this SQL in Figure 3, we see one SQL statement that is also repeated in our physical gets top SQL codes:
90,051 20,322 4.4 1.9 90.85 3506.03 2624188903
Module: SQL*Plus
SELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt
FROM pol p1, pol_tran t1 WHERE p1.p
ol_nbr = :b2 AND p1.ign_sys_cd =
:b1 AND t1.pol_id = p1.pol_id A
ND t1.pol_tran_typ_cd = 'CN' AND

796,391 20,322 39.2 2.0 90.85 3506.03 2624188903

Module: SQL*Plus
SELECT /*+ INDEX(p1 iu_pol_isc) */ t1.pol_tran_eff_dt
FROM pol p1, pol_tran t1 WHERE p1.p
ol_nbr = :b2 AND p1.ign_sys_cd =
:b1 AND t1.pol_id = p1.pol_id A
ND t1.pol_tran_typ_cd = 'CN' AND

We know it is the same code, even though we can’t see all of it by the hash value: 2624188903 being identical. This then is
probably again, the code we should look at optimizing because of this. We can obtain the entire SQL statement by extracting
it from the V$SQLTEXT view using the hash value provided. Using this hash code, we can also (if we are in 9i or greater)
extract the explain plan for the code from the V$SQL_PLAN table as well.
Another statement that appears in both sets is this one:
1,466 1 1,466.0 0.0 10.10 13.08 3656618493

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

972,208 1 972,208.0 2.4 10.10 13.08 3656618493

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ INDEX(pcoopt POL_COVG_ON_OFF_PRE
M_TRAN_IDX1) */ --/*+ INDEX(pcoopt POL_COVG_ON_OFF_PREM_TRAN_I
DX1) */ --- /*+ INDEX(pcoopt PK_POL_COVG_ON_OFF_PREM_TRAN) */
pcoopt.pol_id||','||pcoopt.pol_tran_id||','||pcoop

And the next one as well:

852 1 852.0 0.0 11.15 12.26 656581566
Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo
pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_
POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||
pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||

1,278,490 1 1,278,490.0 3.2 11.15 12.26 656581566

Module: SQLNav5.exe
Select /*+ FIRST_ROWS */ Key||','||WRTN_PREM||','||WRTN_EXPSR||
','||EARNED From ( Select /*+ FIRST_ROWS */ --/*+ INDEX(pcoo
pt POL_COVG_ON_OFF_PREM_TRAN_IDX1) */ --- /*+ INDEX(pcoopt PK_
POL_COVG_ON_OFF_PREM_TRAN) */ pcoopt.pol_id||','||
pcoopt.pol_tran_id||','||pcoopt.veh_unit_nbr||','||

This pretty much tells us that these statements are the ones we should concentrate our tuning efforts on them first, then move
on to the other application based statements.
But how about clear indicators of troubled SQL? One issue I see over and over again, is recursion. Recursion occurs because
a statement doesn’t use bind variables. For example, look at Figure 4.
Snap Id Snap Time Sessions Curs/Sess Comment
------- ------------------ -------- --------- -------------------
Begin Snap: 12 07-Jun-04 17:53:55 117 8.2
End Snap: 13 07-Jun-04 18:03:18 107 7.4

Elapsed: 9.38(mins)

Instance Efficiency Percentages (Target 100%)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 99.98
Buffer Hit %: 98.55 In-memory Sort %: 100.00
Library Hit %: 99.51 Soft Parse %: 98.80
Execute to Parse %: 63.14 Latch Hit %: 99.90
Parse CPU to Parse Elapsd %: 68.58 % Non-Parse CPU: 99.45

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
------------------------------------ ------------ --------- --------
CPU time 1,570 75.13
latch free 13,348 193 9.21
SQL*Net more data to client 327,015 147 7.03
log file sync 3,263 91 4.34
db file scattered read 191,897 44 2.13
-------------------------------------------------------------

CPU Elapsd
Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

19,607,674 245 80,031.3 12.9 150.15 206.17 1654560632

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT COUNT(recruit_id) count, MAX(created) max FROM recruit WH
ERE internet_app = 1 AND stage = 'application' AND status = 'unr
eached' AND has_phone = 1 AND office_id = :office_id

19,448,207 244 79,705.8 12.8 149.57 203.95 219061387

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT COUNT(recruit_id) count FROM recruit WHERE internet_app =
1 AND stage = 'application' AND status = 'unreached' AND has_ph
one = 1 AND office_id = :office_id AND TO_CHAR(created, 'DD-MON-
YY') = :office_today

3,507,015 44 79,704.9 2.3 26.22 33.46 922577606

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT MAX(created) max FROM recruit WHERE internet_app = 1 AND
stage = 'application' AND status = 'unreached' AND has_phone = 1
AND office_id = :office_id

3,139,718 264 11,892.9 2.1 29.10 32.12 2158914250

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT COUNT(r.recruit_id) count FROM recruit r INNER JOIN inter
view_roster ir ON (r.recruit_id = ir.recruit_id) INNER JOIN inte
rview i ON (i.interview_id = ir.interview_id) WHERE i.office_id
= :office_id AND r.status IN ('scheduled', 'confirmed') AND r.ha
s_phone = 1 AND ir.last = 1 AND (i.start_time BETWEEN SYSDATE AN

2,381,778 54 44,107.0 1.6 143.82 198.66 992807648

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT mailer_id, first_name, last_name, zip, tracking_number, c
lass, school_name, year FROM mailer WHERE tracking_number = :tra
cking_number ORDER BY year DESC

1,392,943 108 12,897.6 0.9 34.14 43.47 1151498535

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT recruit_id FROM recruit WHERE mailer_id = :mailer_id

1,275,420 16 79,713.8 0.8 10.27 14.01 1265818810

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT COUNT(r.recruit_id) count FROM recruit r INNER JOIN offic
e o ON (o.office_id = r.office_id) WHERE r.internet_app = 1 AND
r.stage = 'application' AND r.status = 'unreached' AND r.has_pho
ne = 1 AND (r.nhlm_date < '07-JUN-04' OR r.nhlm_status != r.stat
us) AND r.office_id = '96' AND TO_CHAR(r.created, 'J') <= TO_CHA

1,275,416 16 79,713.5 0.8 10.68 12.89 3079473164

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT COUNT(recruit_id) count, MIN(created) min, MAX(created) m
ax FROM recruit WHERE internet_app = 1 AND stage = 'application
' AND status = 'unreached' AND has_phone = 1 AND office_id IN (9
6)

1,275,414 16 79,713.4 0.8 10.67 13.53 3981525210

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT COUNT(r.recruit_id) count FROM recruit r INNER JOIN offic
e o ON (o.office_id = r.office_id) WHERE r.internet_app = 1 AND
r.stage = 'application' AND r.status = 'unreached' AND r.has_pho
ne = 1 AND (r.nhlm_date < '07-JUN-04' OR r.nhlm_status != r.stat
us) AND r.office_id = '96' AND TO_CHAR(r.created, 'J') = TO_CHAR

1,275,409 16 79,713.1 0.8 10.51 14.99 569188530

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT COUNT(r.recruit_id) count FROM recruit r INNER JOIN offic
e o ON (o.office_id = r.office_id) WHERE r.internet_app = 1 AND
r.stage = 'application' AND r.status = 'unreached' AND r.has_pho
ne = 1 AND (r.nhlm_date < '07-JUN-04' OR r.nhlm_status != r.stat
us) AND r.office_id = '96' AND TO_CHAR(r.created, 'J') = TO_CHAR

SQL ordered by Reads for DB: LINEAR Instance: LINEAR Snaps: 12 -13
-> End Disk Reads Threshold: 1000

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ----------
2,081,753 54 38,551.0 89.7 143.82 198.66 992807648
Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT mailer_id, first_name, last_name, zip, tracking_number, c
lass, school_name, year FROM mailer WHERE tracking_number = :tra
cking_number ORDER BY year DESC

63,977 176 363.5 2.8 9.98 12.74 2919827978

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT city, state FROM zip WHERE zip = :zip

6,542 4 1,635.5 0.3 1.94 3.58 1792306764

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',
r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.
has_notes, o.name office_name, r.created FROM recruit r INNER JO
IN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI
N office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

6,396 12 533.0 0.3 8.47 1244.88 3633465809

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT * FROM (SELECT r.recruit_id, CONCAT(r.first_name, CONCAT(
' ', r.last_name)) name, r.source_main, r.has_notes, r.stage, r.
status, o.office_id, o.name office_name, i.start_time, ir.create
d, ir.reminded, ir.prior_qual, ir.user_id, ((TO_CHAR(TO_DATE('07
-JUN-04', 'DD-MON-YY'), 'J') - TO_CHAR(i.start_time, 'J')) + 1)

3,432 2 1,716.0 0.1 1.07 1.09 413550429

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',
r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.
has_notes, o.name office_name, r.created FROM recruit r INNER JO
IN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI
N office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

3,398 2 1,699.0 0.1 0.58 0.57 1480883621

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',
r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.
has_notes, o.name office_name, r.created FROM recruit r INNER JO
IN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI
N office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

3,328 7 475.4 0.1 4.98 549.37 4234417328

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT r.recruit_id, CONCAT(r.first_name, CONCAT(' ', r.last_nam
e)) name, r.source_main, r.has_notes, r.stage, r.status, o.offic
e_id, o.name office_name, i.start_time, ir.created, ir.reminded,
ir.prior_qual, ir.user_id, ((TO_CHAR(TO_DATE('07-JUN-04', 'DD-M
ON-YY'), 'J') - TO_CHAR(i.start_time, 'J')) + 1) day_lapse FROM

3,163 2 1,581.5 0.1 1.07 1.37 775361915

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',
r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.
has_notes, o.name office_name, r.created FROM recruit r INNER JO
IN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI
N office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

3,043 2 1,521.5 0.1 0.69 0.65 305117496

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',
r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.
has_notes, o.name office_name, r.created FROM recruit r INNER JO
IN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI
N office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

2,986 2 1,493.0 0.1 0.92 0.93 3796993827

Module: ? @wsrv1.linearlive.com (TNS V1-V3)

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

SELECT DISTINCT(r.recruit_id), CONCAT(r.first_name, CONCAT(' ',

r.last_name)) name, r.source_main, r.stage, r.status, r.city, r.
has_notes, o.name office_name, r.created FROM recruit r INNER JO
IN recruit_office ro ON (r.recruit_id = ro.recruit_id) INNER JOI
N office o ON (o.office_id = ro.office_id) WHERE r.status != 'ac

% Total
Parse Calls Executions Parses Hash Value
------------ ------------ -------- ----------
4,025 8,043 8.72 3300435981
Module: ? @wsrv1.linearlive.com (TNS V1-V3)
SELECT val FROM storage WHERE sid = :sid AND name = :name

1,694 3,387 3.67 602811079

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
SELECT CONCAT(stage, CONCAT('|', status)) return FROM recruit WH
ERE recruit_id = :bind_item_id

1,385 3,318 3.00 1013713935

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
SELECT phone, extension, name, rank FROM recruit_phone WHERE rec
ruit_id = :recruit_id ORDER BY rank

1,324 2,648 2.87 1553378658

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
SELECT id, start_time FROM history_user_collated WHERE user_id =
:user_id AND end_time >= (SYSDATE - 15/1440)

1,321 1,322 2.86 1553758124

Module: ? @wsrv1.linearlive.com (TNS V1-V3)
INSERT /*+ IDX(0) */ INTO "LINEAR"."MLOG$_HISTORY_USER_COLLATE"
(dmltype$$,old_new$$,snaptime$$,change_linear$$,"ID") VALUES (:d
,:o,to_date('4000-01-01:00:00:00','YYYY-MM-DD:HH24:MI:SS'),:c,:1
)

1,299 1,299 2.81 188214805

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
UPDATE history_user_collated SET end_time = :end_time, seconds =
:second_count WHERE id = :huc_id

1,254 2,508 2.72 706506348

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
SELECT COUNT(*) count FROM interview_roster WHERE interview_id =
:interview_id AND cancelled = 0 AND hide = 0

1,254 5,013 2.72 3547551472

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
SELECT COUNT(*) count FROM receptionist_nh_lm WHERE recruit_id =
:recruit_id AND type = :type

1,252 2,505 2.71 1604412695

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
SELECT * FROM (SELECT created, type FROM receptionist_nh_lm WHER
E recruit_id = :recruit_id ORDER BY created DESC) WHERE rownum =
1

1,252 5,007 2.71 3600969427

Module: ? @wsrv2.linearlive.com (TNS V1-V3)
SELECT COUNT(*) count FROM receptionist_nh_lm WHERE recruit_id =
:recruit_id AND type = :type AND TO_CHAR(created, 'DD-MON-YY')
= :date_today

-------------------------------------------------------------
Figure 4: Problem SQL Indications
In looking at the waits, events and SQL in the above excerpts, we see a lot of CPU time being used, lots of buffer gets and
lots of physical reads. We also see that our key waits are for SQL area (latch free) events. This usually indicates issues with
recursion. We can see that the various parse related ratios are very much less than 100% which is the ultimate tuning goal.
This would indicate no re-parsing (at least hard parsing) was occurring. Reparsing is generally caused by lack of bind

www.odtug.com ODTUG 2005

Using Statspack to Find Bad SQL… Ault

variables. As we look through the SQL in Figure 4 we see a number of SQL statements that are not using bind variables and
some that use a mix of literal and bind variables.
In this situation we can apply the comparisons we used for the buffer reads and the physical reads as well as non-use of bind
variables to find and repair the problem SQL statements. If you are unable to correct the non-use of bind variables, using the
CURSOR_SHARING initialization parameter will help with the parse situation.

Identify SQL using Comments

It can be very difficult to pull your PL/SQL procedure code out of the background code in an instance shared pool, large
tkprof, trace output or from v$sql_plan. I suggest placing a comment in each SQL statement that identifies the SQL. An
example of this is:
CURSOR get_latch IS
SELECT /* DBA_UTIL.get_latch */
a.name,100.*b.sleeps/b.gets
FROM
v$latchname a, v$latch b
WHERE
a.latch# = b.latch# and b.sleeps > 0;
Now to find all SQL code in the shared pool from the DBA_UTILITIES package I can simply query the V$SQLAREA or
V$SQLTEXT, or the V$SQL_PLAN VPT to find code entries with '%DBA_UTIL%' in the SQL_TEXT column. In addition,
in any Statspack output, the code identifies itself.

Summary
In this paper I have tried to convey the importance of using statspack to help find and isolate SQL statements that need
tuning. We have covered the installation and use of Statspack and have discussed AWR and its use along with statspack.
Developers should utilize statspack or AWR for point monitoring of development environments and for continued
monitoring of production environments.

www.odtug.com ODTUG 2005