ON EVALUATING CONTROL PERFORMANCE ON LARGE DATA SETS

Alexander Horch 1, Friedrun Heiber 2

1
ABB Corporate Research, Germany
2
University of Stuttgart, Germany

Abstract: The evaluation of control performance by means of performance indices from

large amounts of measurement data is investigated. The focus is twofold: Firstly to assess
information that can be deduced from many data sets and secondly to investigate the
usefulness of simple performance measures. Established methods and some useful new
ideas are evaluated on many industrial data sets and the results are discussed.

Keywords: Performance indices, control loops, signal processing, data collection.

1 INTRODUCTION base a performance assessment on much more than

only a few data sets.
The evaluation of control performance mostly deals
with the computation of performance indices. These The first issue discussed in this paper is: can useful
indices range from very simple ones, e.g. the control information be found in simple control performance
error mean and variance (e.g. [Salsbury, 1999] up to indices when evaluated on many data sets? Some
rather complex ones involving e.g. subspace identifi - indices (both simple and more advanced ones) will
cation [Bezer gianni and Georgakis, 2000]. be evaluated for 20 control loops from a pulp mill.
For each of the control loops more than 400 data
Obviously, simple indic es are very appealing since batches were ana lysed.
for performance monitoring, typically many control
loops are assessed regularly. Unfortunately, com - A second aspect discussed is the continous collection
plexity and information are usually correlated such of information that can be combined to new
that simple indices mostly imply simple answers. knowledge about a control loop. The availability of
One exception is the mini mum-variance control many data sets can be used to build a nonlinearity
performance index by Harris [Harris, 1989]. It map of the process. Such information is of grea t use
combines low computational effort with important for tuning procedures.
information about the current loop performance. This
fact contributed to its popularity over the last decade. The paper is organised as follows: In Section 2 the
data used in this study is described and presented.
Traditionally, control loop performance assessment Section 3 presents the evaluation of some
deals with the kind of information that can be performance indices for all data sets and conclusions
deduced from the evaluation of a specific data set. thereof. Section 4 discuss es the use of data in order
This approach is sound and valid but carries some to build a knowledge database that grows with each
pitfalls: the data set may reflect unusual behaviour of data set that was analysed.
the control loop in questio n. In the industrial
practice, single data batches are often erroneous Examples accompany both sections and the pa per
since plant shut downs or other unusual events might concludes with a summary and require ments for
have been reflected in the data. industrial control performance moni toring tools.
Such data sets will in general not be able to givea fair
picture of the control loop per formance. They should
ideally be excluded from performance evaluation; 2 INDUSTRIAL DATA SETS
however, a mechanism to automatically discard all
such cases seems to be very difficult to achieve. This study makes intensive use of more than 400 data
sets containing control loop data from a stock
This paper attacks control performance assess ment preparation section in a pulp mill. Each data set
from a slightly different point of view: Given the fact contains about 20 minutes of data at a sampling rate
that the time constant of good control is in the range of 1 second. The loop setpoint (SP), the process
of months rather than days, it would make sense to
variable (PV) and the controller output (OP) were A typical data batch containing 20 control loops (11
logged. flow loops, 3 level loops and 6 composition loops) is
shown in Figure 1.

Figure 1: Example data (SP and PV respectively) for 20 loops from one data collection occasion. The signals
are scaled such that they have equal standard deviation and are plotted on top of each other.

The data collection was done autmatica lly, once per

day for more than a year resulting in 424 data sets.

index description
3 PERFORMANCE INDICES CE mean [%] mean of control error
CE std [%] standard dev. of control err.
Quite a number of performance measures for ass ess- OP std [%] st. dev. of controller ouptut
ing controller performance have been pro posed in the CE skewness skewness of control error
literature, especially during the last decade. Most of CE kurtosis kurtosis of control error
them targeted to be computed from normal operating ratio of std of control error
data only. It is the constraint of not allow ing std ratio
and controller output
experiments that outperforms the computation of maximum bic max. bicoherence
similar perfor mance measures that are typically used correlation coefficient bet -
in controller design (e.g. the loop overshoot or rise correlation
ween control error and
time). coefficient
controller output
3.1 Simple statistics Table 1: Simple performance indices that were
evaluated for all data sets. The units [%] refer to the
The term 'simple indices' refers to indices that can be operating ranges of OP and PV.
evaluated with a modest amount of computa tions and The (normalised) standard deviation of the control
that do not require any non -trivial a priori know- error can also give considerable insight into loop
ledge. Table 1 shows the simple p erformance indices behaviour, see Figure 3. It can be clearly seen that
that were evaluated in this study. loop No. 8 has a problem with increased variability
between logs no. 250 and 300. At this time, the loop
The control error mean should of course be centered had a strong oscillatory behaviour.
around zero with no off -set and a sufficiently small
standard deviation. Long or excessive deviation can
easily be identified (see loops no. 2, 12, 15 and 20 in
Figure 2.
 a cycle time of around one hour. The data batches are
too short for a detection algorithm to detect this
oscillation.

Figure 2: Trend of loop stan dard deviation.

Figure 5: Control error kurtosis trend for loop 12.

In Figure 6, two trends of the ratio of the stan dard
deviations of control error and controller out put are
shown
σ
R = SP − PV .
σ OP

Figure 3: Standard deviation of control error (loop

No. 8).
Another interesting statistics is the control error
skewness. Skew data often indicate problems of
nonlinear character. As an example see Figure 4
where it can be seen that for many data sets, the
skewness is rather large, indicating regular problems
of nonlinear source. As a matter of fact, this loop
exhibits stiction regularly resulting in the presented
non-symmetrical data distribution.

Figure 6: Example for ratio of standard devia tions of

control error and controller output. Left: case with
not moving process output, loop No.6; right:loo p
with normal behaviour, loop No. 5
It can be seen that the value is 5 orders of magnitude
smaller in the left example than in the right one. The
reason is that the process variable in the left loop
(No. 6) does not move much more than the
quantisation level, whereas the loop on the right (No.
5) moves reasonably much. In loop No. 6, either the
sensor signal is corrupt or the loop actuator does
hardly move.

Figure 4: Skewness trend of control error for loop For a last example consider Figure 7 where the
No. 2. maximum bicoherence of the control error is plotted
for all loops. In [Choudhury et al., 2004] it is shown
Consider another example in Figure 5 where the that the bicoherence plot can be used to assess signal
control error kurtosis for loop No. 12 is shown. For nonlinearity. Single evaluations may tend to contra -
Gaussian signals, the kurtosis should be centered dict this hypothesis but when considering many data
around zero. In this case, it is clearly around –1 sets, it turns out that such a measure may be able to
instead, indicating rather non - Gaussian signals. The detect loops that exhibit nonlinearity problems (in
reason in this case is a slow periodic behaviour with
this case loops No. 1, 2, 5, 6, 9, 10 and 11). This is in Oscillation detection can be done in various ways,
line with the knowledge about these loops. see [Hägglund, 1995], [Forsman and Stattin, 1999] or
[Seborg and Miao, 1999].

Figure 9: Oscillation index for all loops.

In Figure 9, trends of the oscillation index [Forsman
Figure 7: Trend of maximum bicoherence over all and Stattin, 1999] are plotted. It can easily be seen
loops. that loops No. 1, 3, 4, 5, 8, 9, 10, and 11 exhibit regu -
lar oscillatory behaviour. The human eye recognises
3.2 More advanced indices an oscillation when the index is larger than about 0.3.

Another group of indices involves more complex

4 Combination of indices
computations and eventually more prior know ledge.
As already mentioned, the most promi nent of these
indices is the Harris index that compares actual lo op Performance indices provide a good means of
variability to minimum -variance variability leading analysing plant behaviour. However, it is often the
to an index between 0 and 1 where 1 equals combination of indices that gives significant insight
minimum-variance performance, into bad plant performance.
σ 2 var
I = min . A very compact assessment would be the test if
σ SP
2
− PV certain control loops exhibit bad behaviour
simultaneously. Hence, a correlation of a spe cific
index for different loops would be a valuable source
of information.

Figure 8:Example of trends for the Harris index.

The example in Figure 8 shows the need for
interpreting the Harris index with a grain of salt. The
left loop shows a good behaviour, however, for some
batches the index is very low. The right loop on the
other side offers the complete rang e of index values,
indicating that the performance (or its assessment) is
very differ ent at different days. To rely on a few or a
single data set only can be misleading when dealing
with the Harris index.

Badly performing loops often exhibit oscillatory be- Figure 10: Correlation coefficients for oscillation
haviour. Therefore, oscillation indices for oscillation indices over all loops.
detection and assessment are the most important
quantity that should be monitored. More and more Consider Figure 10 where oscillation indices are
industrial applications start focussing on periodic correlated for each loop. Such a plot indicates which
disturbances in addition to the Harris index, which loops typically oscillate simultaneously. The plot
was mostly discussed in the mid -90s. reveals a common oscillatory behaviour between
loops no. 5, 7 and 17; see Figure 11. Note that the n Creation of static input – output maps
equality of frequency is of no importance for the n Indication of data sets suitable for model
correlation of the oscillation indices. identification

A static input output map is often of importance

when control loop tuning is performed. Many
commercial tuning tools offer ways to analyse exper -
imental data where the process input is changed step -
wise. The automatic generation of static y -u-maps
can avoid costly experiments and thus enable faster
controller tuning without disturbing current pro -
duction.

If the data within a data batch is sufficiently sta -

tionary, then an algorithm can extract stationary
values of OP and PV and store them. Figure 12
shows exampl es of static maps for all control loops
Figure 11: Three oscillating signals (with analysed. For loops where sufficiently many different
normalised variance) as indicated by correlation of operating points have been found, a quadratic fun -
their oscillation indices. ction has been fit to the data using a least -squares
method.
4.1 Increasing information
For most data sets, a linear function would be
As mentioned in the introduction, the step beyond
sufficient to describe the static input output
storing indices for single data batches is to collect
relationship well. Loops where the quadratic function
and combine assessment information from single fits the map better are No. 1, 2 and 8. However, the
evaluations for future use. Examples for such nonlinearity does not seem to be too severe, such that
applications are:
it could also be neglected.

Figure 12: Static maps for all 20 control loops. The fitted curve is quadratic. Vertical axis is the controller
output and horizontal axis is the process variable.
It was mentioned that static maps provide useful changes the process input stepwise. For both cases, a
insight into the process model when dealing with regular analysis of data could raise and store a flag if
controller tuning. A natural question is then: Could suitable identification data is available.
the regular analysis of normal operating data be used Since it may not be sufficient to flag for setpoint
to detect data sets that are suitable for model changes (or input changes in manual loop mode)
identification? This would be data sets where (a) the only, it was chosen to use a flag that indicates if an
setpoint is changed abruptly by a significant amount estimated dynamic model for the process has
or, (b) the loop is in manual mode and the operator sufficiently good quality. The quality is measured by
goodness of fit test as they are u sed in standard batches is available. From the analyses, some general
system identification pack ages. Figure 13 shows an conclusions shall be drawn:
example of a data set that typically would flag for
being suitable for system identification. n Simple statistics are most useful for fast and
overview-like scans of large amounts of data.
n More complex indices are very useful when
averages over many data sets are available.
Single evaluations may be misleading.
n Combination (e.g. correlation) of indices is
useful and gives insight into the plant dynamics
n Storage of results for later usage are very
helpful, especially for tuning (linearity, stiction).
n Trending of indices presents an extreme data and
information compression for comfortable repor -
ting.

Figure 13: Data set suita ble for model identification REFERENCES
and controller tuning.
Clearly, some loops would never generate these flags
Bezergianni, S. and C.Georgakis (2000). 'Con troller
since setpoints may never be changed or the loops
are never taking into manual mode. Note that performance assessment based on mi nimum and
disturbances alone never qualify data to be useful for open-loop output variance'. Control Engineering
Practice, 8, pp. 791-797.
process model identification. In these cases, only the
Harris, T. (1989). 'Assessment of control loop
controller can be identified.
performance'. The Canadian Journal of
Chemical Engineering, 67, pp 856-861.
Using both static maps and the described model fit
Salsbury, T.I. (1999). 'A practical algorithm for
flags, it is hence possible – at least for some loops –
diagnosing control loop problems'. Energy and
to generate the information that is usually required
for controller tuning without being forced to perform Buildings, 29, pp. 217-227.
experiments. Hägglund, T. (1995). 'A control loop performance
monitor', Control Engineering Practice, 3(11),
pp 1543-1551.
Yet another flag that is useful to store for later use in
Forsman, K and A. Stattin (1999). 'A new criterion
controller tuning is whether the loop exhibited
for detecting oscill ations in control loops.', In
stiction behaviour [Horch, 1999]. Such information
European Control Conference ECC, Karlsruhe,
should be available when tuning loops.
Germany. Session CP8 -3.
Seborg, D. and T. Miao (1999). 'Automatic detection
5 IMPLICATIONS FOR IND USTRIAL TOOLS of excessively oscillatory feedback control
loops'. In IEEE Int. Conf. on Control
Applications, Hawaii, USA, pp 359-364.
From the above results, some implications for the
Choudhury, M.A.A.S., S.L.Shah and N.F. Thornhill
controller performance tools shall be drawn. A
(2004). "Diagnosis of Poor Control Loop
senseful tool should …
Performance using Higher Order Statistics",
accepted for publication in Automatica.
n … enable analysis of performance indices
such as plot combinations, correlation, trend Horch, A. (1999). 'A simple method for detection of
plots etc., stiction in process control loops.', In Control
Eng. Practice, 7(10), pp 1221-1231.
n … enable application -dependent selection
/discarding of indices,
n … offer an index database for search
queries,
n … help to retrieve data collection dates and
– if possible – specific data sets.

6 CONCLUSIONS

The usefulness of performance indices for automa tic

controller performance assessment is an accepted
fact in the process industry. A question that has
received little attention so far is which indices to use
and what kind of infor mation can be deduced from
each of them.
The focus in this paper was to show the strengths of
some selected indices when a large amount of data