ITNM Best Practices For DB2 - v1.2
ITNM Best Practices For DB2 - v1.2
ITNM Best Practices For DB2 - v1.2
n
Note: Before using this information and the product it supports, read the information in “Notices” on page 33.
Contents
About this Guide...................................................................................................iii
Chapter 4: Troubleshooting................................................................................17
Troubleshooting Tips.....................................................................................................................................17
Buffer Pool....................................................................................................................................................17
DB2 commands useful in troubleshooting..................................................................................................18
Diagnostic queries..........................................................................................................................................19
Appendix A: Using the Network Manager GUI to set the Cognos datasource
passwords............................................................................................................23
Appendix D: References....................................................................................31
Notices..................................................................................................................33
Trademarks......................................................................................................................................................35
This edition applies to 3.9, 4.1,4.1.1 of IBM Tivoli Network Manager and to all subsequent releases and modifications
until otherwise indicated in new editions.
© Copyright IBM Corporation 2013.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
About this Guide
The database is central to the functionality of IBM Tivoli Network Manager
IP. It stores all the data discovered and processed by the discovery engine to
build the network topology views, historical data collected by the poller, as
well as reference and operational data used by the Network Manager
processes. Network Manager 3.9 supports the use of DB2, Oracle, MySQL,
and Informix databases. We will focus on DB2 (9.7,10.1,10.5) specifics in this
article, although most of the information referring to the use of databases with
Network Manager applies to all databases.
This document is intended to help you tune and maintain the DB2 database. In
many cases the default values for DB2 are sufficient and very little attention is
required. For customers with large environments it is common to use the
services of a central Database Administration, and that can affect performance
as the DB administrators attempt to optimize their resources by sharing the
DB2 server with other applications. Knowing how Network Manager uses the
database can help avoid heavy traffic problems. Using the information in this
article, you can optimally set up DB2 with Network Manager to maintain
optimum performance levels.
The total amount of discovered data is generally not an issue for the DB2
database. However, as you increase the number of domains and entities,
beyond say five domains, or if you are storing a lot of historical polled data,
you will need to pay attention to the performance of the DB2 database.
Poller
When each poller starts up, and at the end of discovery, there occurs many
minutes of intensive database queries and updates on the NCIM,
NCMONITOR, and NCPOLLDATA tables. If configured, the poller stores
historical polled data in NCPOLLDATA and this can put a steady, heavy load
on the database if large numbers of data points are collected and stored at high
frequency. The poller runs a pruning service every hour to maintain the most
recent data points to a configured limit – by default, 5 million. Refer to the
About Historical Polled Data Collection documentation especially if you plan
on storing a lot of data. See:
http://www.ibm.com/support/knowledgecenter/SSSHRK_4.1.1/com.ibm.netw
orkmanagerip.doc_4.1.1/itnm/ip/wip/common/welcome.html
Under normal circumstances the pruning takes only a few minutes, but the
time can be affected by poor database performance caused by inefficient
transaction log maintenance.
The more concurrent GUI users with network views that are open, the more
load on the database. Recent refactoring of the Network View caching for
Network Manager 3.9 FP3 has improved performance and lightened the load
on the NCP GUI and NCIM schemas for large topology databases
considerably, both for large network views and the number of active users.
2 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
Chapter 2: Network Manager Configuration for DB2 Database
This chapter explains how and where each Network Manager component’s
database connection details are configured. This helps you confirm the details
of the DB2 instance, the database name within the instance, and the user name
specified.
Network Manager Backend Processes
The following files hold all the database connection data for the different
schemas used by the Network Manager backend processes and many Perl
scripts.
$NCHOME/etc/precision/DbLogins.<domain>.cfg
$NCHOME/etc/precision/MibDbLogin.<domain>.cfg (NCMIB schema only)
These files contain the following information that you will need when
accessing the database or changing its settings:
• Database name: The name that the administrator specified when creating the
database instance for Network Manager. An administrator creates the database,
before installing Network Manager, using the create_db2_database.sh (UNIX) or
create_db2_database.bat (WINDOWS) script.
• Schema: The following schemas are created in the database: NCIM, NCPGUI,
NCMONITOR, NCPOLLDATA, and NCMIB. Note that there is a separate record
for each schema in these files. All the schemas are expected to be co-located,
except NCPOLLDATA, which can be in a separate database.
• Username: Specifies the account used for read-write access to the database
instance. By default, the same account is configured for all the schemas.
• Password: Specifies the password for the user associated with Username. By
default, the password is encrypted. You can use the ncp_crypt utility to handle
encryption/decryption of this password.
Note that the passwords in these files, when encrypted, use a different format
from those in DbLogins and cannot be copied from DbLogins, nor can they be
encrypted/decrypted using the ncp_crypt utility. The best way to set or change
these passwords is by using the Network Manager GUI. See DB2 User
Accounts
The database connections for the Tivoli Common Reporting (TCR) reports are
set up separately for BIRT and Cognos reports. The TCR trcmd.sh utility is
used to configure TCR. For convenience, the Network Manager
configureTCR.sh script uses this utility to deploy the Network Manager BIRT
The following sections explain how to configure the passwords for the
Network Manager backend, Network Manager GUI, TCR Reports, and
Network Manager Topology API.
Note: The Topology API is only used by the IBM Tivoli Network
Configuration Manager when integrated with Network Manager.
Configuring the passwords for the Network Manager Backend
Follow these steps to configure the password for the Network Manager
Backend. The DbLogins file (in step 2) has a separate record for each database
schema. Note the username defined for the ncim schema – the new password
should be for this account. Normally the same account is used for all the
schemas, and you can replace the same password in each of these records.
1. For the database account name being used, encrypt the new password
using the ncp_crypt utility, for example, if the new password is netcool:
> ncp_crypt –password netcool
'@44:O2aJesKCAX6Af0aOKeTBMRWk2ru8soEOe9PFEv6smwc=@' netcool
2. Cut and paste the encrypted string (without the single quotes) into the
following files for each schema record using this database account name:
$NCHOME/etc/precision/DbLogins.<domain>.cfg
$NCHOME/etc/precision/MibDbLogin.<domain>.cfg
4 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
Configuring the passwords for the Network Manager GUI
Use the Network Manager GUI to change the password for the database
account. The GUI will handle encryption, if configured, automatically. Select
the following task:
Change the password on both NCIM and Historical Polling panels and press
the Save button for each.
If you are not using Tivoli Data Warehouse for the historical polled data, then
you can simply use the configTCR.sh script located here:
$NCHOME/precision/products/tnm/bin/configTCR.sh
For later versions of Network Manager, unless you add the new
command line option to install the reports, this command will only set
the passwords.
Run the following script to modify all the data sources for the BIRT and
Cognos reports:
./configTCR.sh –p tipadmin_password –d new_db_password
However, if you are using Tivoli Data Warehouse, you must use the TCR
trcmd.sh utility to set the passwords for the data sources separately. Note:
Executing the configTCR.sh script resets the connection details to the local
If you choose not to use the configTCR.sh script, see the BIRT and Cognos
sections below, which describe how to use the TCR script (trcmd.sh) to set the
database readonly account and password for the BIRT and Cognos reports.
BIRT Reports
To set the account name and password, run the trcmd utility once for each of
the data sources: NCIM, NCPOLLDATA, and PARAMETERS. The trcmd.sh
script is located here:
$TIPHOME/../tipv2Components/TCRComponent/bin/trcmd.sh
For example, using the account name db2inst and password netcool, run each
of these commands:
./trcmd.sh -modify -dataSources -reports -username tipadmin -password
tipadmin_password -dataSource name=NCIM -setDatasource odaUser=db2inst
odaPassword=netcool
If you are using Tivoli Data Warehouse, run the first command for the NCIM
data source against the local DB2 database and the other two commands for
the PARAMETERS and NCPOLLDATA data sources to change the password
for the Tivoli Data Warehouse database account.
To set the account name and password, run the trcmd utility once for each of
the data sources: NCIM, NCMONITOR, NCPGUI, NCPOLLDATA, and
PARAMETERS. The trcmd.sh script is located here:
$TIPHOME/../tipv2Components/TCRComponent/bin/trcmd.sh
For example, if the database account is db2inst and the new password is
netcool, the trcmd.sh script would be executed as follows:
6 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
./trcmd.sh –dataSource –add NCIM –dbType DB2 –dbName NCIM
-user tipadmin -password tipadmin_password
-openSessionSql "SET CURRENT SCHEMA = NCIM"
–dbLogin db2inst –dbPassword netcool –force
./trcmd.sh –dataSource –add NCMONITOR –dbType DB2 –dbName NCIM
-user tipadmin -password tipadmin_password
–openSessionSql "SET CURRENT SCHEMA = NCMONITOR"
–dbLogin db2inst –dbPassword netcool -force
./trcmd.sh –dataSource –add NCPGUI –dbType DB2 –dbName NCIM
-user tipadmin -password tipadmin_password
–openSessionSql "SET CURRENT SCHEMA = NCPGUI"
–dbLogin db2inst –dbPassword netcool -force
If using Tivoli Data Warehouse, the database details must match the TDW
database only for the NCPOLLDATA and PARAMETERS data sources.
Follow these steps to configure the password for datasource used by the
Network Manager topology API.
1. Run the following command to refresh the password used by the API after
you have changed the password from the Network Manager GUI as described
above.
$NCHOME/precision/bin/createITNMDatasource.sh
<tip_administrator_username> <tip_administrator_password>
3. Test the datasource after the change by executing the following command:
$TIPHOME/bin/wsadmin.sh -lang jython -username tipadmin
-password <TIP ADMIN Password> -f
$NCHOME/precision/bin/createITNMDatasource.jy -test
8 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
Chapter 3: IBM DB2 Configuration Settings for Network
Manager
The DB2 default configuration settings are usually adequate for a standard
DB2 9.7, 10.1, or 10.5 deployment, with the default settings made by Network
Manager when creating the database instance. However, if you have several
domains, or store a lot of historical polled data, or share the database server
with other products, then you may need to be more aware of the database
performance. If you upgraded your DB2 database, it may still have some of
the previous settings, so it would be wise to check your settings according to
the suggestions below. This chapter assumes you are familiar with issuing
DB2 commands.
If the DB2 server is shared with other products, you might consider increasing
the number of connections to 200 for the Network Manager database instance,
as in the following example:
In either case, restart the database instance for the changes to take effect.
If the transaction log size is insufficient, you may see errors during periods of
many uncommitted row changes such as:
The previous command gives you an idea of how often the active log id is
changing.
Configuration Settings
The following sections have been taken from the Best Practices for System
Performance for IBM Tivoli Service Management Products for settings that
have been found useful for Network Manager.
https://www.ibm.com/developerworks/community/blogs/a9ba1efe-b731-4317-
9724-
a181d6155e3a/entry/new_best_practices_for_system_performance_7_5x_whi
te_paper9?lang=en
Registry keys
Set the following registry keys individually. Using the database instance
owner account (or one with administrative privileges) run the db2set
commands:
db2set DB2_SKIPINSERTED=ON
db2set DB2_SKIPDELETED=ON
db2set DB2_EVALUNCOMMITTED=YES
10 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
db2set DB2_USE_ALTERNATE_PAGE_CLEANING=ON
db2set DB2_INLIST_TO_NLJN=YES
db2set DB2_MINIMIZE_LISTPREFETCH=YES
db2set DB2_FMP_COMM_HEAPSZ=65536
Note that while the recommended setting for SOFTMAX has a positive
impact on runtime performance, crash recovery time will increase. If you use
DB2 HADR, this would not be cause for concern. In addition, if you use
online backup, there may be a spike in I/O while data is being flushed to disk
when the backup starts.
While these settings are generally the default settings in DB2 9.7 and higher,
you must ensure that they are set, especially if you upgraded from a previous
version of DB2 to DB2 9.7.
Note that when using DB2 9.7 with the statement concentrator
(STMT_CONC), IBM recommends applying DB2 fixpack level 5 or later to
avoid a potential performance issue with the statement concentrator.
In addition, DB2 fenced processes can cause excessive memory use in some
cases. Set the following parameter:
db2 update dbm cfg using keepfenced NO
If you need fenced processes and decide to use this feature of DB2, monitor
the db2fmp process closely to ensure that memory usage does not become
excessive.
When running DB2 on AIX 5.2 or above, set the AIX network parameter
lo_perf=0 to avoid an issue where loopback traffic becomes slow, and ping
responds with out-of-order packets. See APAR IY63671 for more details.
-----------------------------------------------------------------------------------------
12 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
Maximum query degree of parallelism (MAX_QUERYDEGREE) = ANY 1
-----------------------------------------------------------------------------------------
Percent log file reclaimed before soft chckpt (SOFTMAX) = 520 360
Sort heap thres for shared sorts (4KB) (SHEAPTHRES_SHR) = AUTOMATIC AUTOMATIC
-----------------------------------------------------------------------------------------
Database Maintenance
Database maintenance consists of tasks such as reorganizing tables and
indexes, and archiving and deleting historical data.
Note: Network Manager 3.9 runs statistics on the NCIM schema tables in the Oracle
and Informix databases after discovery is finished. However, starting with Network
Manager 4.1, Network Manager now runs statistics on the NCIM schema tables in
the DB2 database. You should not need to run statistics on NCIM schema tables if
Network Manager is doing it for you.
If you are storing historical polled data in the NCPOLLDATA schema of any
database type, consider reorganizing and running statistics regularly on the
polldata table.
1. REORG
Should run at least weekly
2. RUNSTATS
Should run at least weekly
3. LOG file pruning
Should run at least every three days in a large environment if you are
storing the historical polled data in Tivoli Data Warehouse.
You can run the first command to first generate an input file for all the tables,
and then run the second command to execute them:
14 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
uses when executing queries.
When there is any data modification to the user tables, the catalog statistics
tables are not automatically modified. A RUNSTATS command has to be
performed on tables and indexes so that columns in the catalog tables are
updated with the latest information.
Run the first command to generate the RUNSTATS commands for all the
tables, and the second command to execute them, as follows:
db2 -x "select 'runstats on table',substr(rtrim(tabschema) ||'.'||
rtrim(tabname),1,60),' with distribution and detailed indexes all
ALLOW WRITE ACCESS;'from syscat.tables where type = 'T' order by
tabschema asc" |tr -s " " |tee runstats_all.sql
Check the log history file and purge it, if necessary, as in the following
example:
If you see more than approximately 5000 records, then that could start to
impact database performance.
DB2 keeps a list of all the log files that have been created, but you really only
need the ones that relate to the last few backups, so as long as you have recent
backups, you can purge the log files before a point in time (for example, the
last 3 backups).
Purge the log files by running the following command:
db2 'prune logfile prior to <time>'
Troubleshooting Tips
When you see a DB2 error message in one of the Network Manager log files
and the meaning is not obvious, execute the following command:
Another place to look is the DB2 log, db2diag.log, located under the DB2
instance account directory (similar to
/home/db2inst1/sqllib/db2dump/db2diag.log) and see what the error
messages in context suggest.
Buffer Pool
This section describes how to troubleshoot a buffer pool failure. For instance
you might see a failure of a buffer pool, for example, BUF32K, to get the
memory it needs. DB2 offers automatic allocation but this feature may not be
active for all buffer pools and this can result in unexpected issues.
To fix this, reset the buffer pool to automatic. Create a file that contains the
following select statement:
Execute this statement from the instance owner account, or one with
appropriate privileges:
su – db2inst1
db2 connect to NCIM
db2 –tvf buffer.sql
7 record(s) selected.
Connect to database:
db2 connect to <DBNAME> user <username> using <pwd>
18 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
ipclean -a
db2support . –d <dbname> -c –s
Diagnostic queries
Here are some useful queries that can help you identify a problem from the
recent executions of SQL queries against the DB2 database.
List the last 10 queries by number of executions. Some of the queries that
run in the normal operation of Network Manager can be very repetitive. If
these are also showing longer than normal execution times, it can expose a
problem. Run this command to show the most frequently executed queries:
Identify poor sorting. Use this command to understand the queries that had
Chapter 4: Troubleshooting 19
more than X number of executions and by SORTS_PER_EXECUTION to
know which queries might indicate missing or poor indexing. Poor indexing
may indicate a need to run REORG to rebuild the indexes, or indicate where
an additional index may be useful. If poor indexing is really the problem,
rather than corrupted indexes, it may be useful to discuss with IBM Support so
that if it does not impact other areas (such as inserts), the indexing in the
product can be improved.
cd ~/sqllib/misc
db2 -tvf EXPLAIN.DDL
db2advis -d <dbname> -n <schema> -s "SELECT DISTINCT tunnel.entityId TUNNELID,
COALESCE(tunnel.displayLabel,tunnel.entityName) TUNNELNAME, dm.domainMgrId
DOMAINMGRID FROM ncim.entityData tunnel INNER JOIN ncim.domainMembers dm ON
dm.entityId = tunnel.entityId INNER JOIN ncim.networkPipe outerPipe ON
outerPipe.entityId = tunnel.entityId INNER JOIN ncim.connects
outerPipeConnects ON outerPipeConnects.connectionId = outerPipe.connectionId
INNER JOIN ncim.mplsTETunnel headend ON headend.entityId = tunnel.entityId
LEFT OUTER JOIN ncpgui.pathView pv ON pv.pathId = tunnel.entityId WHERE
tunnel.entityType = 36 AND pv.viewId IS NULL "
/opt/IBM/tivoli/netcool/platform/linux2x86/db2client/sqllib/db2dump/
For diagnostic information like memory usage and problem queries, check
these files:
<instance_name>.nfy
20 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
db2diag.log
db2evmon -path
/opt/IBM/tivoli/netcool/platform/linux2x86/db2client/ncim/NODE0000/SQL
00001/MEMBER0000/db2event/db2detaildeadlock > db2detail_dead_log.txt
Use the following commands to monitor locking and to identify any long
running locks, including deadlocks:
Chapter 4: Troubleshooting 21
Appendix A: Using the Network Manager GUI to set the
Cognos datasource passwords
1. Select Reporting->Common Reporting, and click on Launch at the top left and
select Administration.
3. Repeat the following steps for each of the data sources: NCIM,
NCPOLLDATA, PARAMETERS, NCPGUI, and NCMONITOR.
6. Click on the Signon tab, and then click on “Edit the signon...”.
7. Enter the database username and password on the following panel. Click Ok to
save it.
24 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
Appendix A: Using the Network Manager GUI to set the Cognos datasource passwords 25
Appendix B: DB2 deprecates System Managed Spaces for
permanent tablespaces
System Managed Spaces (SMS) for permanent table spaces have been deprecated
in DB2 10.1. For details, see:
http://www.ibm.com/support/knowledgecenter/api/content/SSEPGG_10.1.0/com.ibm.
db2.luw.wn.doc/doc/i0058748.html
Network Manager 3.9 and 4.1.1 use DB2 10.5. SMS for permanent tables spaces is
still supported in DB2 10.5, but may be dropped in a future release.
http://www.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.sql.
ref.doc/doc/r0000929.html?cp=SSEPGG_10.5.0%2F2-9-7-102
For new databases, permanent table spaces should use Automatic Storage Table
(AST) managed space.
Tests for Network Manager in the Lab using AST showed some efficiency during
intense database activity, for example moving the discovery data to the NCIM
database after discovery. It did not show measurable improvements working with
topology views in the GUI. While not tested as I write this, we would expect some
similar improvement in cases where the ncpolldata database is heavily used.
If you are deploying a new instance of Network Manager you can take advantage of
the new efficiency and create the new database using AST tablespace. Follow these
steps to edit the script used to create the DB2 database:
with,
create tablespace userspace2 pagesize 32k managed by automatic storage
bufferpool buf32k
3. Create the DB2 database using this script prior to installing Network Manager as
described here:
http://www.ibm.com/support/knowledgecenter/SSSHRK_4.1.1/com.ibm.networkmana
gerip.doc_4.1.1/itnm/ip/wip/install/task/nmip_ins_installdb2unix.html
Visit the home page and download this free tool from:
http://ibm.biz/BdxDbg
To get started, run the PerfAnalyst tool and select File..Open from the menu to
display a list of plugins. Select the DB2::Snapshot plugin and you will see the
commands to use to generate the snapshot. Click OK and it will prompt you
for the snapshot. After loading the snapshot into the tool, the Statement tab
shows the queries sorted by total CPU time. In this example, you can see that
a significant portion of the database CPU utilization (total_cpu_util 53%) was
being consumed by a SELECT statement which throttled the delete processing
from the same pollData table.
34 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2
All IBM prices shown are IBM's suggested retail prices, are current and are
subject to change without notice. Dealer prices may vary.
This information is for planning purposes only. The information herein is
subject to change before the products described become available.
This information contains examples of data and reports used in daily business
operations. To illustrate them as completely as possible, the examples include
the names of individuals, companies, brands, and products. All of these names
are fictitious and any similarity to the names and addresses used by an actual
business enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language,
which illustrate programming techniques on various operating platforms. You
may copy, modify, and distribute these sample programs in any form without
payment to IBM, for the purposes of developing, using, marketing or
distributing application programs conforming to the application programming
interface for the operating platform for which the sample programs are
written. These examples have not been thoroughly tested under all conditions.
IBM, therefore, cannot guarantee or imply reliability, serviceability, or
function of these programs. The sample programs are provided "AS IS",
without warranty of any kind. IBM shall not be liable for any damages arising
out of your use of the sample programs.
Each copy or any portion of these sample programs or any derivative work,
must include a copyright notice as follows:
© (your company name) (year). Portions of this code are derived from IBM
Corp. Sample Programs. © Copyright IBM Corp. _enter the year or years_.
If you are viewing this information softcopy, the photographs and color
illustrations may not appear.
Trademarks
IBM, the IBM logo, ibm.com, Netcool, Netcool/OMNIbus,and Tivoli are
trademarks or registered trademarks of International Business Machines
Corp., registered in many jurisdictions worldwide. Other product and service
names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information"
at www.ibm.com/legal/copytrade.shtml.
Adobe, Acrobat, and Portable Document Format (PDF) are trademarks or
registered trademarks of Adobe Systems Incorporated in the United States,
other countries, or both.
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Oracle and/or its affiliates.
Linux is a registered trademark of Linus Torvalds in the United States, other
countries, or both.
UNIX is a registered trademark of The Open Group in the United States and
other countries.
Notices 35
Microsoft and Windows are trademarks of Microsoft Corporation in the
United States, other countries, or both.
36 IBM Tivoli Network Manager: IBM Tivoli Network Manager with DB2