RMF Overview

RMF Development Edition
z/OS
Resource Measurement Facility
RMF Technical Overview
March, 6th 2010
2010 IBM Corporation
Trademarks
The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.
APPN*
CICS*
DB2*
DB2 Connect
DirMaint
e-business logo*
ECKD
Enterprise Storage Server*
ESCON*
FICON*
GDPS*
Geographically Dispersed Parallel Sysplex
HiperSockets
HyperSwap
IBM*
IBM eServer
IBM e(logo)server*
IBM logo*
IMS
Language Environment*
MQSeries*
Multiprise*
NetView*
On demand business logo
OS/390*
Parallel Sysplex*
PR/SM
Processor Resource/Systems Manager
RACF*
Resource Link
RMF
S/390*
Sysplex Timer*
System z
System z9
TotalStorage*
Virtualization Engine
VM/ESA*
VSE/ESA
VTAM*
WebSphere*
z/Architecture
z/OS*
z/VM*
z/VSE
zSeries*
z9
* Registered trademarks of IBM Corporation
The following are trademarks or registered trademarks of other companies.

Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.
* All other products may be trademarks or registered trademarks of their respective companies.
Anything else?
Postprocessor
ZZ Session
Monitor III
Monitor II
Mintime &
Range
Interval &
Cycle Time
% Workflow
Sysplex
Data Server
Distributed
Data Server
Spreadsheet
Reporter
CIM Data
Provider
% Using / Delay
The RMF product consists of several components for all kinds of performance related disciplines. This session introduces
the main RMF components, sheds light on their primary purpose and discusses the most common RMF terms
Even though its impossible to become an RMF expert within one hour: lets get started and lets see how far we get!
Agenda

Product Structure
Address Spaces
Controlling the Data Gatherers
Historical Reporting
Postprocessor
Spreadsheet Reporter
Realtime Reporting
Monitor III
Monitor II
WTO Alerts
RMF Performance Monitoring
Data Portal

4
RMF Performance Data APIs
The following topics will be covered in this session:

What are the basic functions and what is the primary purpose of these functions?
What address spaces have to be started? What are their names, underlying procedures and responsibilities?
What are the options which determine how the gatheres work in detail?
What are the functions for historical reporting and after the facts analysis?
What are the functions for realtime reporting and problem determination?
Which API's can be user by components or user written programs to retrieve z/OS performance data from RMF
RMF Product Overview

Monitor III Data Portal & RMF PM
RMF Spreadsheet Reporter
Windows, Linux
Windows
RMF Distributed Dataserver (DDS)
RMF Postprocessor
Historical Reporting,
Analysis and Planning
RMF Monitor II and III

Real-Time Reporting,
Problem Determination
RMF Sysplex Data Server and APIs

SMF
SMF
RMF
Data Gatherer
RMF
Monitor I
RMF
Monitor II
background
RMF
Monitor III
VSAM
VSAM
z/OS Resource Measurement Facility (RMF) is an optional priced feature of z/OS. It supports installations in performance
analysis, capacity planning, and problem determination. For these disciplines, different kinds of data collectors are needed:
Monitor I long term data collector for all types of resources and workloads. The SMF data collected by Monitor I is
mostly used for capacity planning and performance analysis
Monitor II snap shot data collector for address space states and resource usage. A subset of Monitor II data is also
displayed by the IBM SDSF product
Monitor III short-term data collector for problem determination, workflow delay monitoring and goal attainment
supervision. This data is also used by the RMF PM Java Client and the RMF Monitor III Data Portal
Data collected by all three gatherers can be saved persistently for later reporting (SMF records or Monitor III VSAM
datasets)
While Monitor II and Monitor III are realtime reporters, the RMF Postprocessor is the historical reporting function for Monitor
I data
RMF Address Spaces / Procedures
SYS1.PROCLIB(RMF)
SYS1.PROCLIB(RMFGAT)
SYS1.PROCLIB(GPMSERVE)
RMF
Control
RMF
Monitor III
Gatherer
s RMF
Monitor I
(zz Session)
SY
S
( E 1 .P A
RB
RM RML
F 0 IB
4)
SY
S
(E 1.PA
RB
RM RML
F0 IB
0)
RMF
Distributed
Data
Server
RMF
Monitor III
Batch
Reporting
s GPMSERVE
s RMFM3B
SY
S
(G 1.PA
PM
R
SR ML
V0 IB
0)
f RMF, s III
f RMF,s ZZ
SYS1.PROCLIB(RMFM3B)
In order to make all RMF functions working, a couple of address spaces have to be activated:
RMF is the root address space. It is required for any further activities
The Monitor I gatherer is a subtask in the RMF address space. It is started immediately together with RMF (by default) or
later by means of the modify command
RMFGAT is the Monitor III data gatherer. It cannot be started as standalone address space. Like Monitor I, it can only be
activated with the modify command. Within a sysplex, one instance per system is needed for RMFGAT
The RMF Distributed Data Server is the data source for the RMF Performance Monitoring java client and the Monitor III
Data Portal. Only one instance is needed per sysplex
RMFM3B is needed for the generation of console messages in case a certain threshold is exceeded
RMF Control Address Space

keeps Configuration Tables and other Control Information
provides the Command Interface to set of modify Options:
f RMF,f ZZ,MEMBER(99)
maintains the SMF Data Buffer:

s RMF,,,SMFBUF(SPACE(12M),RECTYPE(70:78))
SMF
Exits
IEFU83x
INSTALL
COPY
Sysplex
Data Server
SMFBUF
(DataSpace)
RMF
CALL
SMF
WRITE
Write SMF Record
SMF
SMFWTM
Component
all SMF Record Types can be maintained by the RMF Sysplex Data Server !
The RMF control address space has 3 major responsibilities:

Builds up configuration tables, reads and remembers the parmlib options
Provides the command interface to set options or to start and stop the gathering functions
Maintains the SMF Buffer of the RMF Sysplex Data Server
Data Gathering Methods
exact measurement counts

pick up consecutive counters
calculating the difference at the end of an interval
Interval Gathering
eg. CPU seconds, device connect time...
sampling counts
inspect variable counters continuously
building the average at the end of an interval
Cycle Gathering
eg. queue counts, frame counts...
In order to collect all the information that is displayed in the various reports, RMF uses two different techniques of data
gathering
The preferred and cheap method is to retrieve exact measurement counts: this can be timers (e.g. dispatch times) or
accumalitve counts like the Start Subchannel count for a device
The costly cycling technique is needed for counters which are changing rapidly and frequently:
how long is the dispatcher queue?
how many I/O queue elements are chained to a UCB?
what is the actual enqueue situation?
Monitor I Data Gathering

1. Measurements
CACHE
CHANNEL
CPU
CRYPTO
DEVICE
ENQ
IOQ
FCD
ESS
PAGESP
PAGING
TRACE
VSTOR
WKLD
SMF 74.5
SMF 73
SMF 70.1
SMF 70.2
SMF 74.1
SMF 77
SMF 78.3
SMF 74.7
SMF 74.8
SMF 75
SMF 71
SMF 76
SMF 78.2
SMF 72.3
SY
S
(E 1.PA
RB
RM RML
F 0 IB
0)
SMF
SMF 74.2
SMF 74.3
SMF 74.4
SMF 74.6
XCF
OMVS
CF
HFS
2. Timing
CYCLE(1000)
NOSTOP
SYNC(SMF)
gathered by
Monitor III
3. Reporting / Recording
RECORD
REPORT(REALTIME)
SYSOUT(A)
4. User Exits
NOEXITS
The directives to setup Monitor I data gathering are kept in SYS1.PARMLIB(ERBRMF00)

There are four different categories of control statements:
Measurements: specifies the kind of resources, which are the candidates for data gathering
The collected data can be saved to SMF records for later reporting
A subset of SMF records is gathered by Monitor III: since the same data is needed for Monitor III reports,
duplicate gathering is avoided
Timing: determines the frequency for interval gathering and cycle gathering
Reporting/Recording: defines the type and the creation time of the output
User Exits: specifies routines which can be executed during session processing to gather or report additional data:
ERBMFIUC (initialization)
ERBMFxxx (cycle gatherer, loaded by ERBMFIUC)
ERBMFDUC (interval gatherer)
ERBMFRUR (report writer)
ERBMFTUR (termination)
Monitor III Data Gathering

SY
S
(E 1.PA
RB
RM RML
F0 IB
4)
1. Measurements
IOSUB
CFDETAIL
CACHE
VSAMRLS
OPD
HFSNAME
zFS
SGSPACE
2. Timing
CYCLE(1000)
MINTIME(60)
NOSTOP
SYNC(00)
RMFGAT Address Space

In Storage Buffer
WSTOR(32)
3. Recording
DATASET(ADD(RMF.M3G.&SYSNAME..DS1))
DATASET(ADD(RMF.M3G.&SYSNAME..DS2))
DATASET(START)
DATASET(NOSWITCH)
DATASET(WHOLD(7))
DATASET(WHOLD(7))
4. Buffer
WSTOR(32)
ERBVSDEF vsam_ds VSAMVOL(volser)
DATASET(ADD(vsam_ds))
DATASET(DEL(vsam_ds))
ERBV2S vsam_ds seq_ds
ERBS2V seq_ds vsam_ds
10
The directives to setup Monitor III data gathering are kept in SYS1.PARMLIB(ERBRMF04)
Only a few measurements can be turned ON or OFF. The data collection cannot be controlled on resource granularity:
almost all resources must be monitored in order to get the complete picture of the address spaces which are using or
delayed for certain resources. However, a subset of gathering activities which does not interfere with address space states
can be switched on or off
In order to extend the report range for historical data, Monitor III VSAM Datasets can be used. In this case, the buffersize
which has been specified by the WSTOR option can be reduced to the WHOLD size (since the data which does not fit any
more in the buffer is kept at least in the datasets)
For a smart handling of the VSAM datasets, the following procedures are provided:
ERBVSDEF
defines a Monitor III VSAM dataset
ERBV2S
unloads the VSAM data to a sequential dataset
ERBS2V
reconverts the sequential dataset to VSAM
RMF Distributed Data Server

has Sysplex-wide scope
exploited by RMF PM, Monitor III Data Portal,
CIM Provider
talks XML over HTTP
no configuration needed!
just enter S GPMSERVE or F RMF,DDS on one
TCP/IP
image of the Sysplex
GPMSERVE
(RMF Distributed Data Server)
RMF Sysplex Data Server (ERB3XDRS Service)
RMF M III Data Gatherer

11
The RMF Distributed Data Server (DDS) is the component, which serves as data source for the RMF client applications
There is just one additional address space (GPMSERVE) needed per Sysplex. The DDS uses the ERB3XDRS service to
retrieve Sysplex-wide Monitor III data.
In a mixed release configuration, the DDS should run on system with the highest RMF release
Historical Reporting
Activities measured by Monitor I
Other Activities
Lotus Domino Server
HTTP Server
12
Lotus 1-2-3 and Excel
Spreadsheet Applications
XCF
OMVS
Coupling Facility
HFS
SMF
Activities measured by Monitor III
RMF Data
Buffer
RMF Postprocessor
Cache Subsystem
Channel Path
CPU / Crypto
Device
Enqueue
Ficon Director
I/O Queuing
Page/Swap Data Set
Paging
Trace
Virtual Storage
Workload
The tool of choice for historical reporting and after the facts analysis is the RMF Postprocessor
Accordingly to the Monitor I data collection options (there is one SMF record type for each resource), the reporting structure
is resource oriented (exception: the WLMGL record (SMF 72.3) holds performance data and resource consumption on
workload granularity)
Some SMF record types (74.x) are written by the Montior III data gatherer address space RMFGAT. However, this address
space should be active anyway to guarantee access to the Monitor III reports
With OS/390 R10 the Postprocessor has been enabled to accept foreign SMF records:
Type 103 HTTP Server
Type 108 Domino Server
While the RMF Postprocessor generates tabular reports, the RMF Spreadsheet Reporter can create graphical charts based
on Postprocessor data
Postprocessor:
Postprocessor: Preparing SMF Data
//SMFDUMP EXEC PGM=IFASMFDP

//IDD1
DD
DISP=SHR,DSN=<input_smfdata_system1>
//IDD2
DD
//SMFDATA DD
DISP=(NEW,PASS),SPACE=(CYL,(10,10),RLSE),
//
UNIT=SYSDA,DCB=(RECFM=VBS,LRECL=32760,BLKSIZE=0)
//SYSIN DD *
INDD(IDD1,OPTIONS(DUMP))
INDD(IDD2,OPTIONS(DUMP))
OUTDD(SMFDATA,TYPE(70:78))
//RMFSORT EXEC PGM=SORT

//SORTIN
DD
//
DD
//SYSIN
DD
*
SORT FIELDS=(11,4,CH,A,7,4,CH,A),EQUALS
MODS E15=(ERBPPE15,36000,,N),E35=(ERBPPE35,3000,,N)
13

SMF data is kept in VSAM datasets

Postprocessor requires sequential format
use SMF dump utility IFASMFDP to unload the data
usuallly GDGs are the prefered target:
RMF.SMFDATA.SYSNAME(0)
RMF.SMFDATA.SYSNAME(-1)
SMF reords must be sorted by date and time

SORT step is required for sysplex-wide reporting
Workload Activity Report
Coupling Facility Report
Shared DASD Report
Before the raw SMF data can be feeded to the Postprocessor, two preparation steps are needed:
The data has to be transformed from VSAM to Sequential format
This is achieved with the SMF Dump Utility IFASMFDP
In most installations, the SMF data is dumped by default to Generation Data Groups (GDGs)
For Sysplex-wide reporting, the records arriving from the various systems must be sorted by ascending interval time
Therefore, a preceeding SORT step must be executed
RMF provides the exit routines ERBPPE15 and ERBPPE35 to enable the sorting by RMF interval start time
Postprocessor JCL
DD Names
MFPMSGDS
MFPINPUT
PPRPTS
PPSUMnnn
PPXSRPTS
PPORPnnn
PPOVWREC
Message Output
SMF Input Datasets
combined Interval Reports
Summary Report Output
Sysplex Report Output
Overview Report Output
Overview Record Output
Control Statements
14
DATE
RTOD
DINTV
REPORTS
OVERVIEW
OVW
Start / End Date

Start / End Time
Duration Interval Length
Report Types
Report or Record
Overview Control Statement
JCL can be generated

by ISPF Application or
//RMFPP
EXEC PGM=ERBRMFPP
//MFPMSGDS DD
DISP=SHR,DSN=*.ALLOC.MSG
//MFPINPUT DD
DISP=(OLD,DELETE),DSN=*.RMFSORT.SORTOUT
//PPRPTS
DD
DISP=SHR,DSN=RMF.INTERVAL.REPORTS
//PXSRPTS DD
DISP=SHR,DSN=RMF.SYSPLEX.REPORTS
//SYSIN
DD
*
DATE(04212003,04232003)
RTOD(0000,2400)
DINTV(0100)
REPORTS(ALL)
Different DD names and control statements must be specified for the different kinds of report output
However, the user doesn't need to take care about any syntax: the JCL to execute the Postprocessor can be generated with
an ISPF dialog or with the RMF Spreadsheet Reporter workstation frontend
Postprocessor:
Postprocessor: Standard Reporting
C P U
//RMFPP EXEC PGM=ERBRMFPP

//SYSIN
DD
*
DATE(10142002,10142002)
RTOD(1100,1300)
REPORTS(CPU)
SYSRPTS(WLMGL(SCPER))
SYSOUT(H)
z/OS V1R8
CPU
2084
SYSTEM ID SYSD
RPT VERSION V1R8 RMF
MODEL
314
H/W MODEL
---CPU--- ONLINE TIME

NUM TYPE PERCENTAGE
0
CP
100.00
1
CP
100.00
2
CP
100.00
3
CP
100.00
CP
TOTAL/AVERAGE
LPAR BUSY
TIME PERC
3.23
2.77
2.58
2.54
2.78
W O R K L O A D
z/OS V1R8
SYSPLEX SYSDPLEX
A C T I V I T Y
DATE 02/13/2007
TIME 11.00.00
INTERVAL 15.00.048
CYCLE 1.000 SECONDS
B16
MVS BUSY
TIME PERC
3.61
3.18
2.93
2.97
3.17
CPU SERIAL
NUMBER
066F7A
066F7A
066F7A
066F7A
I/O TOTAL
INTERRUPT RATE
14.60
22.53
25.61
29.42
92.16
% I/O INTERRUPTS
HANDLED VIA TPI
0.69
0.49
0.43
0.46
0.50
A C T I V I T Y
START 02/13/2007-11.00.00 INTERVAL 000.15.00

END
02/13/2007-11.15.00
MODE = GOAL
POLICY ACTIVATION DATE/TIME 02/12/2007 08.43.05

------------------------------------------------------------------------------------------------------------ SERVICE CLASS PERIODS
REPORT BY: POLICY=DEFAULT
-TRANSACTIONSAVG
23.36
MPL
23.36
ENDED
0
END/S
0.00
#SWAPS
106
EXCTD
0
AVG ENC
0.00
REM ENC
0.00
MS ENC
0.00
15
WORKLOAD=SYSTEM
TRANS-TIME HHH.MM.SS.TTT
ACTUAL
0
EXECUTION
0
QUEUED
0
R/S AFFIN
0
INELIGIBLE
0
CONVERSION
0
STD DEV
0
SERVICE CLASS=SYSTEM
CRITICAL
=NONE
--DASD I/O-SSCHRT 30.3

RESP
1.1
CONN
0.6
DISC
0.0
Q+PEND
0.3
IOSQ
0.1
---SERVICE---IOC
15835
CPU
5183K
MSO
0
SRB
1548K
TOT
6747K
/SEC
7496
ABSRPTN
TRX SERV
321
321
RESOURCE GROUP=*NONE
--SERVICE TIMES-CPU
26.498
SRB
7.913
RCT
0.017
IIT
0.275
HST
0.000
AAP
0.000
IIP
N/A
PROMOTED
PERIOD=1 IMPORTANCE=SYSTEM
---APPL
CP
AAPCP
IIPCP
%--3.86
0.00
0.00
-----STORAGE----AVG
2871.20
TOTAL
67066.09
SHARED
11.00
AAP
IIP
0.00
N/A
--PAGE-IN RATES-SINGLE
0.0
BLOCK
0.0
SHARED
0.0
HSP
0.0
0.000
In most cases, the RMF Postprocessor is used to produce tabular reports (one per interval).
The chart shows an example for a single system report and for a Sysplex-wide report:
The CPU Activity Report shows the processor utilization from the systems perspective (on logical processor level)
The Workload Activity Report shows Sysplex-wide transaction and resource consumption statistics (on service class period
level)
Postprocessor:
Postprocessor: Overview Reporting
OVERVIEW(RECORD,REPORT)
OVW(PROCS(NUMPROC))
OVW(CPUBSY(CPUBSY))
OVW(APPL(APPLPER(POLICY)))
OVW(TRANS(TRANS(POLICY)))
OVW(RTIME(RTIME(POLICY)))
DINTV(0100)
(via Spreadsheet Reporter)
R M F
z/OS
V1R8
NUMBER OF INTERVALS 4
DATE
MM/DD
08/07
08/07
08/07
08/07
TIME
HH.MM.SS
11.00.00
12.00.00
11.00.00
12.00.00
INT
HH.MM.SS
01.00.00
01.00.00
01.00.00
01.00.00
O V E R V I E W
SYSTEM ID SYSF
R E P O R T
START 08/07/2006-11.00.00
END
08/07/2006-15.00.00
INTERVAL 01.00.00
CYCLE 1.000 SECONDS
TOTAL LENGTH OF INTERVALS 04.00.00

PROCS
CPUBSY
APPL
TRANS
RTIME
5.0
5.0
5.0
5.0
9.2
8.5
8.3
8.8
33.9
33.0
30.6
24.5
16.8
28.1
38.9
24.5
40.56
35.92
25.66
44.99
Format of the INTERVAL column

changes from MM.SS to HH.MM.SS
16
When the user wants to focus on specific metrics without analyzing an entire report, the Postprocessor Overview feature
provides a smart solution: by means of overview control statements, he can extract specific values and produce his own
customized reports
Multiple intervals are visible at a glance: Thus, Overview reporting is well suited for trend analysis over long periods
For all SMF 70-78 record types, the user can choose from a rich set of Overview Control statements. For more details, refer
to the RMF Users Guide (SC33-7990), Chapter 17, Overview and Exception Conditions
17
converts SMF Data to Spreadsheet Format

creates graphical Views for Trend Analysis
can be downloaded from the RMF Homepage
In extension to the RMF Postprocessor, RMF offers a graphical workstation frontend for historical reporting:
the RMF Spreadsheet Reporter
The visualization of the data is performed by spreadsheet macros, which are shipped as samples together with the
Spreadsheet Reporter main application.
The Spreadsheet Reporter is a basic part of the RMF product and doesnt require additional charges. The installable image
is shipped with the host file SYS1.SERBPWS(ERB9R2SW) or the most recent version can be downloaded from the RMF
homepage: http://www.ibm.com/servers/eserver/zseries/zos/rmf/. You will find the Spreadsheet Reporter in the Tools
category of the homepage.
SMF raw Data

RMF Postprocessor
Report Listings
Overview Records
Download
Report Listings
Overview Records
Extract
*.RPT Files
*.OVW Files
Convert
CPU Contention: System SYS1
*.WK1 Files
Generate
Working Sets
Display
Spreadsheet
18
The transition from the SMF raw data to graphical charts is a complex process with multiple steps.
The good news - the RMF Spreadsheet Reporter does it all for you by magic:
Generates Report Listings or Overview Records from the SMF data with the RMF Postprocessor: on the Resource
notebook page of the Spreadsheet Reporter, specify as input the SMF dump data sets from all the systems of your
sysplex. Then check the desired reports on the Reports selection dialog. Now you are ready to exploit the Spreadsheet
Reporters remote job execution engine, which allows you to run the Postprocessor from your workstation!
Downloads the Postprocessor output to your workstation
Extracts the reports or records
Convert the reports or records to spreadsheet format and generates the Working Set
Macros
for all Types of
Performance relevant
Areas !!
19
Once a Working Set has been created from SMF data, the user can select from a rich set of macros for all performance
relevant areas
Three basic types of macros can be distinguished:
Macros based on data contained in one Postprocessor Report (for detailed analysis of one single interval)
Macros based on data contained in Overview Records (for trend analysis with multiple intervals)
Macros based on data contained in multiple Postprocessor Reports (hybrid macros)
The new XCF Trend Report macro belongs to the family of hybrid macros
Realtime Reporting
CF

covers all Sysplex related aspects

two monitors and a workstation extension
Monitor
III, best suited for
short-term, real-time and historical reporting

online performance analysis
goal attainment supervision
sysplex-wide and single-system reporting
monitoring of exceptional conditions
WLM
Monitor
II, best suited for
snapshot reporting
single job and resource monitoring
RMF
PM / Data Portal
enterprise-wide reporting of z/OS systems

based on RMF Monitor III data
20
RMF offers three different Monitors for realtime reporting:

Monitor III Reporter (ISPF):
Is designed for workflow analysis and goal attainment supervision
Can be used in realtime mode, but supports historical reporting as well
Monitor II (ISPF and native TSO):
Is a snapshot monitor. The interval length can be determined by the user
Consists of a set of address space and resource oriented reports
RMF Performance Monitoring / Monitor III Data Portal:
Allows to manage multiple Sysplexes from the workstation
The raw data is identical with Monitor III ISPF, but the GUI is totally different
Monitor III Reporting

Monitor III Delay Monitoring

Processor
Storage
Device
Enqueue
Operator
Message
Tape Mount
Subsystem
HSM - JES - XCF
Monitor III Features

Monitor III Activity Monitoring

21
Common Storage
Page/Swap Data Sets
Storage Frames
Device
Data Set Level by Job and Volume
Cache
Coupling Facility
Goal Attainment
VSAM RLS
UNIX System Services
Enclaves
zFS
Diskspace
Spin/Suspend Locks

Cursor-Sensitive Navigation
Workflow/Exceptions Monitoring
Automatic Customization
Support of WTO Messages
Continuous Monitoring
Hardcopy Reports
On-Line Tutorial
On-Line Help
Adaptive Reports
User Reports
Sysplex-wide Reports
Remote Reporting
The Monitor III ISPF Reporter provides two basic types of monitoring:
WorkloadDelay monitoring
The state samplers are inspecting all hard- and software resources and assign those resources on address space
level
Possible states for an address space are:
Delayed for Processor, Storage, Device, Enqueue, Operator, Subsystem
Using Processor or Device
Idle (timer wait or terminal input wait)
Unknown (e.g. using an unmonitored waiting mechanism)
ActivityResource Monitoring
In addition to the workloaddelay concept, there are lots of resource oriented reports with similar statistics than the
RMF Postprocessor reports
Together with hardware or software related innovations new reports have been added with each z/OS release
States of a Job
Proc
Idle
Dev
Dev
Dsn
Dsn
JES
Enq
Oper
HSM
Stor
XCF
Proc
?
Unknown
22
Using(%) =
using samples
number of samples
100
= 50%
Delay(%) =
delay samples
number of samples
100
= 33%
Workflow (%) =
using samples
using samples + delay samples
100
Address spaces are either using resources or they are delayed for resources or they are idle
The RMF state samplers are trying to monitor continuously all kinds of hard- and software resources
Depending on the result, all address spaces can be associated with the following qualities:
Using%
Delay%
Workflow%
Since not all address spaces can be catched up every cycle, the remaining states are counted as unknown. Possible
reasons for unknown are:
The address space is delayed for unmonitored resource (other than DASD or Tape)
The address space is using an unmonitored waiting mechanism (most Started Tasks!)
statistical reasons
= 60%
Example: Using and Delay
D
B
Job
A
B
C
D
Res
D
*
*
Why
C
C
*
*
Job
A
B
C
D
Res
D
*
Why
B
*
*
*
Job
A
B
C
D
U
*
Why
D
*
Why
C
*
*
RMF Monitor III Resource Delay Report

Samples: 4
Jobname
WFL USG DLY IDL Primary

%
%
%
% Reason
WFL
Resource %
C
C
A
C
Job
A
B
C
D
Time: 06.28.20 Range: 4 Sec
25 25 75 0
50 25 25 50
66 50 25 25
0 0 25 75
Res
Samples: 4
A
B
C
D
Res
RMF Monitor III Delay Report
23
Res
Time: 06.28.20 Range: 4 Sec

ADU
40 1.5
Jobname USG DLY Reason

%
%
A
B
C
D
25
25
50
0
75
25
25
25
C
C
A
C
The example helps to understand what is calculated and reported by RMF Monitor III
From the jobs perspective:
Job A
1x Using(25%)
3x Delayed(75%) 0x Idle(0%)
Workflow: 1/4 = 25%
Job B
1x Using(25%)
Workflow: 1/2 = 50%
Job C
2x Using(50%)
Workflow: 2/3 = 66%
Job D
0x Using(0%)
Workflow: 0/1 = 0%
Workflow = 4/10 = 40 %
Average Delayed User (ADU): 6/4 = 1.5
From the resource perspective:

4x using, 6x delay
Monitor III: Job Delays

RMF V1R7
Samples: 100
System: AQTS
Delay Report
Date: 07/18/06
Line 1 of 326
Time: 06.28.20
Range: 100
Sec
24
Name
Service
CX Class
SUSANK
CONSOLE
RRSSERVQ
BHBE
MORABIT
RONDA2A
D24JAP1
RRSSERVQ
GRSARTSQ
RRSSERVQ
CATALOG
ANFWPROC
GRSARTSQ
SMFDRS
JES2
GRSARTSQ
ARTXESQ
DFRMM
T
S
B
T
T
B
T
B
B
B
S
SO
B
S
S
B
B
S
Cr
WFL USG DLY IDL UKN ---- % Delayed for ---- Primary
%
%
%
%
% PRC DEV STR SUB OPR ENQ Reason
TSOPRIME
SYSTEM
WLMSHORT
TSOPRIME
TSOPRIME
COMBUILD
TSOPRIME
WLMSHORT
WLMSHORT
WLMSHORT
SYSTEM
SYSSTC
WLMSHORT
STCMED
SYSSTC
WLMSHORT
WLMSHORT
SYSSTC
0
0
0
40
41
42
49
50
50
50
63
67
71
71
73
80
80
83
0 100
0 15
0
1
4
6
37 56
29 42
22 23
2
2
1
1
1
1
57 39
2
1
5
2
5
2
16
6
4
1
4
1
81 18
0
0
0
90
0
0
56
0
0
0
0
0
0
0
0
0
0
0
0
85
1
0
7
29
0
0
0
0
12
2
0
93
79
0
0
1
0
0
0
6
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
56
41
23
2
0
0
39
1
1
0
6
0
1
18
0 100
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
1
0
1
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0
0
15
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0
Address Space
Performance at a Glance !
sorted by ascending Workflow
Delay Type Breakdown
Delay Reason Information
HSM
Message
JES
JHUGO
COMPK5
COMPK5
PRIPK5
SPOL1J
JES
JES
MCATTS
SPOL1J
SPOL1J
SYSZVVDS
SPOL1J
JES
SPOL1L
SL3061
The Address Space Delay report is the basic report that visualizes the overall Monitor III concept
The report is sorted by ascending workflow: thus, the critical candidates with low workflow appear on top of the address
space list
Beyond the basic statistics (workflow, using, delay, idle, unknown), a delay type breakdown for the following resources is
provided:
PRC
processor
DEV
device
STR
storage
SUB
software subsystem
OPR
operator
ENQ
enqueue
Additionally, the delay reason information is displayed in the rightmost column:

For DEV the critical volume
For PRC the competing address space with the highest usage
For SUB the subsystem name
For ENQ the enqueue resource name
Monitor III: Workflow/Exceptions

RMF V1R7
Samples: 100
System: AQTS
Workflow/Exceptions
Date: 07/18/02
Time: 06.28.20
Line 1 of 32
Range: 100
Sec
--------------------------- Speed (Workflow) --------------------------------Average CPU Util: 81 %

Speed of 100 = Maximum, 0 = Stopped
Name
Users Active
Speed
Name
Users Active
Speed
*SYSTEM
560
35
42
*DEV
53
15
46
ALL TSO
19
4
17
*MASTER*
1
1
100
ALL STC
485
6
52
ALL BATCH
39
24
37
ALL ASCH
Not avail
ALL OMVS
17
1
92
*PROC
292
8
43
------------------------------ Exceptions ------------------------------------Name
Reason
Critical val. Possible cause or action
*SLIP*
SLIP PER TRAP
SLIP ID=U427 is active.
ALL TSO
DEV -RMFLIB
1.5 users
Short block size or excessive seeks.
*SYSTEM
PROC-SUSANK
12.1 users
BHBE
SUBS-JES
100.0 % delay Processing TSO OUTPUT command request.
MORABIT
PROC-SUSANK
75.0 % delay
GANDALF
DEV -SMS001
20.0 % delay Device may be overloaded.
CONSOLE
OPER-Message
13.0 % delay Awaiting reply to operator request 52.
HSM
Not avail
Job HSM is not running.
SYSPAG
Not avail
Volume SYSPAG is not mounted.
25
Bottleneck Detection
at a Glance !
Workflow for Groups and
Resources
Exception Lines for individual
Conditions
The Monitor III Workflow Exception Report consists of two independent sections:
The Workflow section provides an overview about groups of work, individual address spaces or resources
The Exceptions section displays alerts when critical thresholds are exceeded
The content of both sections can be customized individually:
For the Workflow section, up to 14 candidates for continuous display of workflow can be selected
For the Exceptions section, alerts for almost all Monitor III metrics (as well as combination of metrics) can be defined
Monitor III: Goal Attainment

RMF 2.10.0 Sysplex Summary - SCLMPLEX
Command ===>
WLM Samples: 240
Systems: 3
Date: 05/15/02 Time: 13.00.00 Range: 60
Service Definition: SCLM

Active Policy: STANDARD
Name
STC
STCCMD
SYSTEM
SYSSTC
SYSTEM
TSO
PRDTSO
W
S
W
S
S
W
S
1
2
3
R
MASTER
26
1
1
1
40
0.46
N/A
N/A
1.000 AVG
1.500 AVG
2.000 AVG
N/A
Sec
Installed at: 12/06/00, 10.07.24

Activated at: 12/06/00, 10.07.33
------- Goals versus Actuals -------Exec Vel --- Response Time --- Perf
Goal Act ---Goal--- --Actual-- Indx
88
88
69
N/A 68
N/A 70
84
84
60
0.0
85
N/A 47
Line 1 of 14
Scroll ===> CSR
0.080
0.109
2.928
AVG
AVG
AVG
0.08
0.07
1.46
Trans --Avg. Resp. TimeEnded WAIT EXECUT ACTUAL

Rate
Time
Time
Time
0.000
0.000
0.000
0.000
0.000
2.100
2.100
1.150
0.567
0.383
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.608
0.608
0.080
0.109
2.928
0.000
0.000
0.000Sysplex Performance at a Glance
0.000 80 Intervals in GO Mode
0.000
0.000 colored Indication for PI > 1
0.608Importance = 1+2
0.608
0.080Importance > 2
0.109
2.928
0.000
The Sysplex Summary Report is the standard report for goal attainment supervision
The report provides Goals vs. Actuals on Service Class Period level. In case the Performance Index (PI) for a certain
Service Class Period is higher than 1, coloured lines are generated:
A red line for Service Class Periods with high Importance (1, 2)
A yellow line for Service Class Periods with low Importance (3, 4, 5)
In GO Mode, a coloured bar shows the state for the last 80 intervals: when at least one line for the current interval becomes
red or yellow, a red or yellow square will be added. Otherwise the bar is extended with a green square. Thus, the Sysplex
performance is visible at a glance for an 80-minute period (proposed a 60 second reporting range)
For Service Classes with response time goals, the report provides a direct link to Response Time Distribution Report by
means of cursor sensitivity
Monitor III: Session Setup

VSAM Archives
alloc f(rmfds00) da(rmf.m3.sys1.data)
alloc f(rmfds01) da(rmf.m3.sys2.data)
TSO Address Space

VSAM Datasets
SYS1.SERBCLS
(ERBRMF3X)
Monitor III
Reporter
Type text
Type text
ERBTLIB
&USER.&REL.ISPTABLE
SYS1.SERBTENU
SYS1.SERBT
Type text
RMFGAT
In Storage Buffer
&USER.&REL.ADMGDF
27
The options for a Monitor III reporter session are stored in tables (e.g exception definitions). The initialization procedure
ERBRMF3X performs the table allocations when a reporter session is going to be started
By default, a reporter session accesses directly the actual data collected by the RMFGAT (via Cross Memory)
However, it is also possible to establish a standalone reporter session independent from the Monitor III gatherer: in order to
analyze historical data, up to 99 VSAM datasets can be preallocated (DDNAME = RMFDSxx)
Monitor III Reporter Usage

RMF
Session
Color
etc.
RMF
Main Menu
Report
Options
Session
Options
RO
F3
M III
Main Menu
Tutorial
Help
Report
Selections,
Short
Commands
F1
System Information
Samples: 100 System: AQTS
Group
WFL
*SYSTEM 30
*TSO
20
DMN001
28
PG002
20
User
250
22
16
22
...
...
...
...
...
Enter
F3
Scroll
Monitor II
Enter
Tutorial
F7
Postprocessor
Selection Menu
F8
BREF/FREF
F3
F10
F11
Find, Color,
ICU, Current,
GO,
and more ...
28
The Monitor III reporter session can be invoked directly from the RMF Main Menu
Various session options (SO command) can be defined and stored persistently, for example:
initial mode (GO or STOP)
initial screen
range length
By means of report options (RO command), the content of almost all reports can be customized
A very detailed tutorial provides lots of information about the Monitor III concepts and features
Cursor sensitive HELP (including detailed metric descriptions) is always available via the PF1 key
Monitor III Report Overview
Sysplex Reports
Coupling Facility
SYSSUM
CFSYS
SYSRTD
CFOVER
SYSWKM
CFACT
Cache Subsystem
VSAM RLS
RLSSC
CACHSUM
User Reports
SYSENQ
RLSDS
CACHDET
DSINDEX
RLSLRU
RG
System/Workload Reports
Monitoring Reports
GROUP
SYSINFO
CPC
WFEX
Delay Reports
Job/Workload oriented
PROC
DEV
DSND
DSNV
STORS
HSM
ENCLAVE
MSI
OPD
SYSTREND
DELAY
Resource Utilization Reports
DEVN
STORCR
DEVT
STORC
DSD
Resource oriented
ENQR
ENQ
STOR
Other
DEVR
STORR
STORF
STORCR
JES
CHANNEL
PROC
XCF
Job
Reports
IOQUEUE
Modified panel
User-written report
Sortable report
Detailed Job Analysis
29
JOB
Monitor III offers a complete report suite which covers all performance relevant areas. The reports can be grouped by
categories:
A family of reports with sysplex wide scope
System level reports for various kinds of measurements
Job/address space oriented reports
Reports from the perspective of the major hard-and software resources
User reports which can be adapted or customized
In order to improve the Monitor III capabilities, new reports have been added frequently, e.g.
OS/390 R4
DSND DSNV
OS/390 R6
CFSYS CFOVER CFACT
OS/390 R7
CACHESUM CACHDET ENCLAVE
OS/390 R10
RLSSC RLSDS RLSLRU OPD
z/OS V1R2
CPC
z/OS V1R7
ZFSSUM ZFSACT SPACEG SPACED
z/OS V1R8
XCFOVW XCFSYS XCFPATH XCFGROUP (Data Portal only)
Monitor III: Setup for WTO

WTOs
Monitor III Batch Address Space creates Reporting Tables
Monitor III
Gatherer
Thresholds can be defined via

Workflow Exception Options Dialog
Reporter Phase Exit Module
Console Message is generated by Exit Module (e.g. WLM Capping)
$HASP100 BMAI
ON TSOINRDR
$HASP373 BMAI
STARTED
IEF125I BMAI - LOGGED ON - TIME=12.36.20
+RMF300I 3B: Processing CPC Report...
+RMF301I 3B: Local Partition Capping State:
+RMF303I 3B: Time until Capping (sec):
40 (WTO Limit:
+RMF304I 3B: MSU Consumption of critical LPARs:
+RMF305I 3B: SYS1 :
64 (WTO Limit: 60)
+RMF305I 3B: SYS4 :
48 (WTO Limit: 32)
IEF126I BMAI - LOGGED OFF - TIME=12.38.00
$HASP395 BMAI
ENDED
$HASP250 BMAI PURGED
+RMF300I 3B: Processing CPC Report...
+RMF301I 3B: Local Partition Capping State:
+RMF302I 3B: WLM Capping %: 24.2 (WTO Limit: 10.0)
+RMF304I 3B: MSU Consumption of critical LPARs:
+RMF305I 3B: SYS1 :
82 (WTO Limit: 60)
+RMF305I 3B: SYS2 :
12 (WTO Limit: 10)
$HASP100 BMGU
ON TSOINRDR
$HASP373 BMGU
STARTED
IEF125I BMGU - LOGGED ON - TIME=12.39.40
30
Set of
Samples
Monitor III
Batch
Reporter
600)
WTO
Exit
Module
Sample Exits provided for:

Workflow Exception Report
Sysinfo Report
CPC Capacity Report
The procedure RMFM3B is supplied to run a Monitor III reporter session in batch mode. This environment is well suited,
when a threshold is exceeded and a corresponding message should be displayed on the console
For threshold processing, the user can select from all metrics contained in one of the Monitor III reports: the procedure
accepts any report name as input parameter. In other words, the specified report is the one and only report that is produced
by the batch reporter session (multiple reports are not supported)
The scanning of the report tables and the comparison against thresholds is performed by exit routines. Depending on the
result of this comparison, a WTO can be generated
RMF ships the following REXX exit routines as examples (in SYS1.SERBCLS):
ERBR3WFX
processes the WFEX report
ERBR3SYS
processes the SYSINFO report
ERBR3CPC
processes the CPC report
Monitor II: Overview

Monitor II is a Snapshot Reporter
collects the status of system resources (CPU, devices, paging activity, ...)
collects the status of address spaces (resource usage, state information)
use Monitor II to
continuously monitor resource usage

determine the state of any address space in the system
track CPU usage of problem address spaces
collect supplemental information when analyzing performance problems with Monitor III
choose Background Session

to collect SMF records for archiving and later postprocessing
to automate snapshot reporting
choose Display Session
Postprocessor
for immediate feedback

for online analysis
f rmf,s aa
10:12:08
JOBNAME
*MASTER*
PCAUTH
31
DEV
CONN
2618
1264
SMF 79.x
SMF 79.x
Monitor II complements the host based RMF reporting suite. It is the RMF snapshot reporting feature
Three basic groups of reports can be distinguished:
Address space oriented reports (e.g. ARD, ASD, ASRM)
I/O subsystem related reports (e.g. DEV, CHANNEL)
Resource reports (e.g. SENQ, SPAG)
Monitor II is the appropriate function, when immediate feedback about address spaces or resources is required
However, by means of a background session, Monitor II can also produce SMF Type 79 records. These records can be
feeded to the RMF Postprocessor for after the facts analysis
Monitor II Reporting
Activities measured by Monitor II:

Address Space Data

32
Resource
State
SRM
Channel Path
Device
I/O Queuing
Enqueue
HFS
IRLM Long Locks
Paging
Page/Swap Data Set
SRM Resource Data
Sysplex Data Server
Library Display
ISPF
for most comfortable usage

supports sorting and finding
started from TSO READY or
from RMF main menu
TSO
don't use it anymore!
3270
dropped with z/OS V1R2
Monitor II collects statistics about activities and resources (but no delays like Monitor III)
The most heavily used Monitor II reports are the address space oriented reports
Address Space Resource Data(ARD): Resource consumption of address spaces (CPU, I/O Storage)
Address Space State Data(ASD): State Information for address spaces (Service Class, Dispatching Priority, Location
e.g IN or OUT)
Address Space SRM Data(ASRM): Details from the SRM point of view (Transaction Active Time, Service Units...)
ISPF is the recommended Monitor II reporting environment (although Monitor II reports can be invoked from native TSO as
well)
Monitor II Commands
TSO-Command: RMF MON2
e.g. ASD report options

Find string
RFind
Cancel
Report Options
e.g. ASD
Monitor II Primary Menu
Enter selection number or command
RMF - ASD
e.g. ARD
RMF
Selection ===>
Reset
Selection
RO
RMF - ARD
Command ===>
report-name
F7
Scroll up
F8
Scroll down
MIG=63.1K
1 Address Spaces
2 I/O Subsystem
3 Resource
L Library Lists
U User
Return
F4
11:15:27 DEV
JOBNAME CONN
*MASTER*
PCAUTH
TRACE
DUMPSRV
364.3
0.000
0.000
1.113
Other commands:
Sort
H
D
Print
Sys
Keys
33
Prints all reports

Delta-mode
Prints current screen
Remote reporting
View/Assign PF-keys
F6
FF PRIV LSQA
BEL
FF CSF
0
0
0
0
455
2
3
2
106
32
105
62
GO nn for auto update
<Enter> for manual update
Monitor II reports can be selected either by menu choice or by report name

With the report options command RO, the report contents can be customized (e.g. Workloads TSO, Batch....)
Use the PF7/PF8 keys for scrolling (in case not all report lines are visible)
Note, that all reports are sortable (place the cursor on the desired coulmn header and press PF6)
The GO command provides the automatic refresh of the report data
With the SYS command, the reports from all remote systems in the Sysplex can be accessed
Monitor II: ARD Report

RMF - ARD Address Space Resource Data
Line 1 of 48
Scroll ===> PAGE
Command ===>
CPU= 45/ 30 UIC=2540 PR=
16:04:11 DEV
JOBNAME CONN
HSM
ANTMAIN
*MASTER*
XCFAS
IOSAS
RMFGAT
JES2
SMSVSAM
SMS
OMVS
CATALOG
HSMMON
NET
SMF
LLA
NETVSCLM
IXGLOGR
79920
28880
8616
4485
3774
3144
773.4
771.2
697.1
548.4
376.5
81.79
66.93
52.73
27.73
24.02
23.65
FF PRV LSQA LSQA X

SRM
16M FF CSF ESF M CR ABS
1
4 136
2 23 160
0 731 102
0 2K 1059
0 29 109
1 20
65
9 66 185
1 205 764
0
2
53
5 131 630
0
2 234
0
3
53
0 36
79
0
2
56
0 40
67
1
5 148
0
5
87
6
12
18
459
10
6
43
85
12
60
1
9
67
8
14
31
48
X
X
X
X
X
X
X
X
X
X
X
X
X
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
TCB
TIME
8508
360.2
296.6
409.7
46.07
5160
360.8
434.8
166.1
85.77
111.7
99.15
771.0
0.20
3.65
115.1
38.28
System= SCLM Total
CPU EXCP SWAP LPA CSA NVI V&H

TIME RATE RATE RT RT RT RT
10575
427.7
2652
796.5
46.89
5236
455.3
550.2
174.2
103.0
117.4
101.8
1556
3.70
4.09
125.9
45.04
0.01
0.00
0.04
2.41
0.02
0.67
2.19
1.55
2.88
0.00
0.21
0.19
0.00
0.00
0.06
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Address Space Resource

Consumption at a Glance !
I/O Activity
Frame Counts
CPU Time
all Table Reports

are sortable !
34
The ARD report provides a quick check about address spaces and their resource usage (e.g. Device Connect Time or TCB
Time)
The report header displays the following information:
MVS view of the CPU Utilization
LPAR view of the CPU utilization
Unreferenced Interval Count
Pagein Rate
The tabular part of the report contains similar information than the SDSF DA screen
Monitor II Session Setup

H | Print
RMF - ARD
Command ===>
1. ALLOC F(RMFDMTSO)
DA('BHBE.M2.REPORTS') disp
DCB=(RECFM=VBA,BLKSIZE=1693,LRECL=137)
2. RMF MON2
MIG=63.1K
11:15:27 DEV
JOBNAME CONN
*MASTER*
PCAUTH
TRACE
DUMPSRV
364.3
0.000
0.000
1.113
FF PRIV LSQA
BEL
FF CSF
0
0
0
0
455
2
3
2
106
32
105
62
ERBROPT2
Report options
BHBE.RMFOS260.ISPTABLE
BHBE.M2.REPORTS
35
In case the user wants to save the reports for later analysis he can use the Hardcopy and Print commands
The reports are written either to SYSOUT or to the preallocated dataset RMFDMTSO
For most of the reports you can specify options or filters, e.g.
ARD:
address space types BATCH or TSO only
DEV:
volume names, device numbers or storage groups
For persistence, the report options are written to the dataset &uid.RMF&rel.ISPTABLE
Monitor II Display Modes

Specific resource or job, e.g. job
BHOL
All resources or jobs at a specific

point in time
RMF - ARDJ Address Space Resou

Command ===>
RMF - ARD Address Space Resour

Command ===>
CPU= 19/ 19 UIC=2540

BHOL
TIME
DEV
CONN
13:16:46
13:16:47
13:16:48
13:16:49
13:16:50
13:16:51
13:16:52
13:16:53
13:16:54
16.87
16.88
16.89
16.90
16.91
16.92
16.93
16.94
16.95
FF PRIV LSQA LSQA X SRM

BEL
FF CSF ESF M ABS
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
79
79
79
79
79
79
79
79
79
0
0
0
0
0
0
0
0
0
65K
65K
65K
65K
65K
65K
65K
65K
65K
TCB
TIME
7.20
7.20
7.20
7.21
7.21
7.21
7.22
7.22
7.22
CPU=
13:20:26 DEV
JOBNAME CONN
*MASTER*
PCAUTH
RASP
TRACE
DUMPSRV
XCFAS
GRS
SMXC
SYSBMAS
36
Current status in highlighted line
8 UIC=2540
FF PRIV LSQA LSQA X SRM

BEL
FF CSF ESF M ABS
409.1
0.000
0.000
0.000
1.113
273.9
0.000
0.000
0.000
Row Report
9/
0 503
0
2
0
4
0
3
0
2
0 1420
0
40
0
2
0
41
106
32
12
105
62
749
692
33
73
0
0
0
0
2
5
0
0
0
TCB
TIME
0.0 34.00 2
X 0.0
0.00
0.0
0.00
X 0.0
0.00
0.0
0.05
X 0.0 28.64
X 0.0 241.01 3
0.0 30.28
0.0
3.94
Table Report
Current status in whole report
The address space and volume oriented reports are available in two flavours:
Table reports:
one sample with all address spaces or all volumes at a glance is displayed
Row reports:
multiple samples for one address space or one volume are displayed
Monitor II Report Overview

Name
Mon I
Row
ISPF
ARD
Address space resource data
ARDJ
ASD
Address space state data
ASDJ
ASRM
Address space SRM data
ASRMJ
CHANNEL
Channel path activity data
DEV
DEVV
Device activity data

Y
HFS
ILOCK
IOQUEUE
I/O queuing activity data

Library lists
Page/swap data set activity
SDS
Sysplex data server statistics
SENQ
Enqueue contention
SENQR
37
HFS statistics
IRLM locking data
LLI
PGSP
Explanation
Enqueue reserve activity
SPAG
Paging activity
SRCS
Central storage, processor, SRM
A subset of Monitor II reports is based on Monitor I gatherer data (DEV, IOQUEUE, PGSP).
Those reports are not available when the ZZ session is inactive (reason: Supervisor state and Key zero is required for
certain gathering activities)
The SORT capability as well as the HFS report is only available in the ISPF environment
RMF Performance Monitoring
Enterprise-wide performance monitoring
38
of z/OS hosts
Platform independent Java Edition
Linux gathering support
Graphical user interface
Flexible definition of data
Persistent definition of views
Powerful data reduction
Analysis support
RMF Performance Monitoring (RMF PM) is the graphical workstation frontend for Monitor III data. It allows to manage
multiple Sysplexes from one single point of control
RMF PM is free of charge. The most recent version can be downloaded from the RMF homepage:
http://www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/pmweb/pmweb.html
RMF PM: Perfdesk Concept
Perfdesk Folder
Perfdesk 1
Dataview 1
Dataview 2
Series 1
Series n
Series 1
Series n
Dataview 3
Dataview 4
Series 1
Series n
Series 1
Series n
Perfdesk 2
39
Dataview 1
Dataview 2
Series 1
Series n
Series 1
Series n
The main window is of the RMF PM application is divided into 2 parts:

The left side contains the PerfDesk notebook page, which allows to control the content of the right side of the window
The right side contains the Open Perfdesk window: one Perfdesk consists of one or more views with the actual data
(Data Views). The data itself is represented by the so called Series
On the host side, the Distributed Data Server (DDS) address space must be started in order to provide the data for the PM
client by means of a TCP/IP connection
RMF PM: Resources and Metrics
Sysplex
MVS Image
I/O Subsystem
All SSIDs
SSID
All LCUs
LCU
All Channels
Channel
All Volumes
Volume
Processor
Storage
Auxiliary Storage
Central Storage
CSA, SQA,
ECSA, ESQA
Expanded Storage
Enqueue
Operator
Subsystems
JES, XCF, HSM
CPC
LPAR
Coupling Facility
CF Structure
40
Resources are components of a computing system. The Monitor III data model maps those resouces in terms of a
hierarchical scheme
Each resource is associated with a set of metrics. Individual metrics (or counters) can be selected for data collection and
continous display
RMF PM: Intelligent Analysis

data views with click-sensitve bars
link to predefined Analysis PerfDesks
41
One of the most powerful features of RMF PM is the Intelligent Analysis: once the user has detected an abnormal value, he
can navigate to context information or additional views in order to detect the reason of a problem. This is achieved by clicksensitive bars whereby the analysis menu choice offers a selection of various pathes for further drill-down
Linux Performance Monitoring
RMF PM Java TM
Technology Edition - Linux
support available on Internet
Common View
for
z/OS and Linux
Data
42
RMF PM has been extended to support the Linux operating system using the existing RMF PM infrastructure. Thus, the PM
capabilities (spreadsheet export, filter, historical data collection, graphical trend reports, etc.) are also available for Linux
environments
For the RMF PM client, it is transparent whether the host is a z/OS or Linux system
On the server side, two additional components are needed:
The Linux compiled version of the Distributed Data Server infrastructure
The Linux data gatherer backend (collects data out of the procfile system)
The Linux support can be downloaded free of charge: http://www.ibm.com/servers/eserver/zseries/rmf/pmlin.htm
Please note that the Linux support is not an official part of z/OS RMF
RMF Monitor III Data Portal

direct connection to the RMF Distributed Data Server
just specify http://<hostname>:8803
all resources and metrics supported (same than RMF PM)
43
Since the RMF Distributed Data Server talks XML over HTTP, the client can be either a fat client like RMF PM or the data
can also be accessed directly by means of a web browser. Thats what we call the RMF Monitor III Data Portal
In other words, the RMF DDS server behaves like a webserver: the data can be requested by means of the standard HTTP
protocol. In this case, the presentation is managed by associated formatting instructions in terms of Style Sheets (XSL
documents)
The chart shows the Overview Performance Desktop (=PerfDesk) with the following metrics:
% processor utilization by MVS image
% total physical utilization (CP) by partition
performance index by important WLM service class period
% CSA utilization by MVS image

Portal
select favorite
Metric from List
click
Add this Metric to My View
for Persistence
select any Resource

from Configuration View
Coupling Facility PerfDesk with:

Request rates
Service times
44
Beyond the Overview PerfDesk, the user can customize his own private PerfDesk (aka MyView). Thereby he can select
from a rich set of metrics for any kind of resource he wants to monitor contionuously. The definitions are stored persistently
in terms of browser cookies
The example shows a Coupling Facility PerfDesk with the following metrics:
sync request rate by CF structure
sync service time by CF structure
async request rate by CF structure
async service time by CF structure

Portal
Combine strengths of Monitor III with state-of-the-art frontend
View full Reports and exploit the SORT capability
Report name
Sortable!
Scrollable!
All columns
visible!
Resizeable!
45
In the past, the Data Portal was limited to display single metrics for selected resources
This concept has been extended: with z/OS V1.8 RMF, complete Monitor III reports can be displayed in the browser window
Thus, the strengths of the Monitor III ISPF reporting are combined with a state of the art frontend:
The window is resizable and scrollable
All report columns can be displayed (including ISPF hidden fields and the content of popup windows)
The columns can be sorted by clicking on the column header
Support of automatic refresh (GO mode)
Multiple reports can be displayed in parallel (tabbed browsing)
Print facility and Spreadsheet export
RMF Performance Data API

APIs
ERBSMFI - Monitor II Data (SMF Type 79)
RMF Sysplex Data Server (SDS)

SMF
Data:
Monitor III Data:
Monitor II Data:
46
ERBDSQRY, ERBDSREC
ERB3XDRS
ERB2XDGS
RMF Provider for z/OS CIM Server
RMF offers a couple of powerful APIs to retrieve z/OS performance data. The exploiters can be:
Other z/OS components or systems management applications
User written applications
All RMF APIs are documented in the RMF Programmers Guide (SC33-7994)
Sysplex Data Server: Services
Sysplex-wide
ERBDSQRY
ERBDSREC
access to SMF data
requests a directory of available SMF data in the sysplex

requests SMF record data in the sysplex
Sysplex-wide
access to Monitor III data
ERB3XDRS
requests a set_of_samples of Monitor III data

does not require an ISPF and Monitor III reporter environment
provides data reduction features to transfer only the necessary data
Sysplex-wide
access to Monitor II data
ERB2XDGS
requests Monitor II data according to the specified SMF

type 79 subtype
returns Monitor II snapshot data
provides data reduction features like ERB3XDRS
all Services are available as High-Level-Language APIs

47
The RMF Sysplex Data Server provides a set of callable services to retrieve performance data from all RMF Monitors
Please note, that the SMF data related services are not limited to the SMF 70-78 record types: once the user has specified
any SMF record type for buffering by the Sysplex Data Server (via the SMFBUF parameter), the record images can be
obtained with the RMF services ERBDSQRY and ERBDSREQ
Groovy!
Postprocessor
ZZ Session
Monitor III
Monitor II
Mintime &
Range
Interval &
Cycle Time
% Workflow
48
Sysplex
Data Server
Distributed
Data Server
Spreadsheet
Reporter
CIM Data
Provider
% Using / Delay
Thats the freshly baked RMF expert! Probably he feels a bit more skilled now. At least he came out of his hiding-place!
Information and Tools

RMF homepage: www.ibm.com/systems/z/os/zos/features/rmf/
Product information, newsletters, presentations, ...
Downloads

RMF PM Java Edition
RMF data collector for Linux
RMF email address: rmf@de.ibm.com

Documentation and news
RMF Performance Management Guide, SC33-7992
RMF Report Analysis, SC33-7991
RMF Users Guide, SC33-7990
Latest version of PDF files can be downloaded from:

http://www.ibm.com/systems/z/os/zos/bkserv/
49
RMF Redbook !!!
50

RMF Overview

Uploaded by

Copyright:

Available Formats

RMF Overview

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

RMF Overview

Uploaded by

Copyright:

Available Formats

RMF Development Edition

RMF Technical Overview

March, 6th 2010

2010 IBM Corporation

Resource Measurement Facility

* Registered trademarks of IBM Corporation

The following are trademarks or registered trademarks of other companies.

RMF Technical Overview

2010 IBM Corporation

Resource Measurement Facility

RMF Technical Overview

2010 IBM Corporation

Resource Measurement Facility

RMF Performance Data APIs

RMF Technical Overview

2010 IBM Corporation

The following topics will be covered in this session:

Resource Measurement Facility

RMF Product Overview

RMF Spreadsheet Reporter

RMF Distributed Dataserver (DDS)

RMF Monitor II and III

RMF Sysplex Data Server and APIs

RMF Technical Overview

2010 IBM Corporation

Resource Measurement Facility

RMF Address Spaces / Procedures

RMF Technical Overview

2010 IBM Corporation

Resource Measurement Facility

RMF Control Address Space

maintains the SMF Data Buffer:

Write SMF Record

RMF Technical Overview

The RMF control address space has 3 major responsibilities:

2010 IBM Corporation

Resource Measurement Facility

Data Gathering Methods

exact measurement counts

eg. CPU seconds, device connect time...

eg. queue counts, frame counts...

RMF Technical Overview

2010 IBM Corporation

Resource Measurement Facility

Monitor I Data Gathering

RMF Technical Overview

2010 IBM Corporation

The directives to setup Monitor I data gathering are kept in SYS1.PARMLIB(ERBRMF00)

Resource Measurement Facility

Monitor III Data Gathering

RMFGAT Address Space

ERBS2V seq_ds vsam_ds

RMF Technical Overview

2010 IBM Corporation

defines a Monitor III VSAM dataset

unloads the VSAM data to a sequential dataset

reconverts the sequential dataset to VSAM

Resource Measurement Facility

RMF Distributed Data Server

image of the Sysplex