Ac16078457 1

Chair of Automation
Master's Thesis
System Identification of Nonlinear

Dynamic Systems
Stefan Oliver Fladischer, BSc

September 2020
Acknowledgements
I want to thank Paul O’Leary, the head of the Chair of Automation at the Montanuniversity
Leoben, for enabling this work. I’m thankful to my supervisor Matthew Harker for all his
support and the helpful discussions. I would like to thank Petra Hirtenlehner for always
being helpful with any organisatory matters, and all the other people working at the Chair
of Automation for providing a welcoming and friendly working environment. I’m grateful
to my parents who always supported me throughout my life, in the pursuit of my education
and through this venture.
ii
Abstract
System identification is the experimental modeling of a dynamic system whose parame-

ters or underlying physical principles are not precisely known. In particular, measurement
data in the form of input-output data sets can be used to estimate the parameters of a sys-
tem model. The goal of this work is the application of numerical methods to realize the
parametrization of a model such that it predicts the behaviour of a nonlinear dynamic sys-
tem in an optimal way.
The basis for the applied system identification algorithm is the minimization of the out-
put error of the model. A goodness-of-fit criterion, the sum of squared vertical distances
between the measurement data points and the simulated model output at these points in
time, is to be minimized.
As the model of a nonlinear dynamic system is described by nonlinear differential equa-
tions, a numerical solver for the solution of initial value problems in conjunction with
a numerical optimization method for the solution of the ensuing nonlinear curve fitting
problem are applied in the system identification procedure.
This system identification algorithm is applied to solve a set of example problems: a free
falling object that is subject to drag due to air, a nonlinear mass and spring system and a
nonlinear dynamic friction model, the LuGre model. Different numerical solutions meth-
ods for initial value problems as well as different numerical optimization techniques are
applied in the solution of these system identification problems based on synthetic mea-
surement data.
The influence of gaussian measurement noise on the identified parameters as well as the
feasibility of utilizing multiple measurement data sets in order to eliminate this distur-
bance induced variation is investigated. Furthermore, the combination of measurement
data sets corresponding to different excitation levels of the object of interest is explored –
a procedure that is of special importance in the system identification of nonlinear systems
in order to accurately identify all the model parameters.
iii
Kurzfassung
Systemidentifikation ist die experimentelle Modellierung eines dynamischen Systems,

dessen Parameter oder grundlegende physikalische Prinzipien nicht genau bekannt sind.
Im Speziellen können Messdaten in der Form von Input-Output-Datensätzen dazu ver-
wendet werden die Parameter eines Systemmodells abzuschätzen. Das Ziel dieser Arbeit
ist die Anwendung numerischer Methoden, um die Parametrisierung eines Modells zu
realisieren, sodass das Verhalten eines nichtlinearen dynamischen Systems in einer opti-
malen Weise vorausberechnet wird.
Die Basis des angewendeten Systemidentifikationsalgorithmus ist die Minimierung des
Outputfehlers des Modells. Ein Anpassungsgütekriterium, die Summe der quadrierten
vertikalen Abstände zwischen den Messwerten und dem simulierten Modelloutput zu
diesen Zeitpunkten, soll minimiert werden.
Da das Modell eines nichtlinearen dynamischen Systems durch nichtlinearen Differential-
gleichungen beschrieben wird, wird ein numerisches Lösungsverfahren für die Lösung
von Initialwertaufgaben in Verbindung mit einem numerischen Optimierungsverfahren
für die Lösung des hervorgehenden nichtlinearen Kurvenanpassungsproblems für die Sys-
temidentifikationsprozedur angewendet.
Dieser Systemidentifikationsalgorithmus wird zur Lösung einiger Beispielprobleme an-
gewendet: Ein Objekt im freien Fall unter Einfluss von Luftwiderstand, ein nichtlineares
Feder-Masse-System und ein nichtlineares dynamisches Reibungsmodell, das LuGre-
Modell.
Verschiedene numerische Lösungsverfahren für Initialwertaufgaben sowie verschiedene
numerische Optimierungsverfahren werden zur Lösung dieser Systemidentifikationsprob-
leme auf Basis von synthetisierten Messdaten angewendet.
Der Einfluss von normalverteilten Messtörungen auf die identifizierten Parameter sowie
die Möglichkeit der Nutzung mehrere Messdatensätze zur Eliminierung dieser störungs-
induzierten Variation wird untersucht. Des Weiteren wird die Kombination von Mess-
datensätzen, die zu unterschiedlichen Anregungslevels des zu untersuchenden Objekts
gehören, untersucht – eine Prozedur die von spezieller Bedeutung für die Systemidenti-
fikation von nichtlinearen dynamischen Systemen ist, um alle Modellparameter genau zu
identifizieren.
iv
Contents
1 Introduction 1
1.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Systems and Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 System Identification and its Place in Control Theory . . . . . . . . . . . 3
1.4 Why are Linear Models Preferred over Nonlinear Models? . . . . . . . . 4
2 System Identification Theory 7

2.1 System Identification as a Part of System Modeling . . . . . . . . . . . . 7
2.2 The System Identification Procedure . . . . . . . . . . . . . . . . . . . . 8
2.3 Building a Design Model . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.1 Black-Box Models . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.2 White- and Grey-Box Models . . . . . . . . . . . . . . . . . . . 10
2.4 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 The Goodness-Of-Fit Measure . . . . . . . . . . . . . . . . . . . . . . . 11
2.5.1 Vertical Distance Least Squares Fit of the Model Output . . . . . 11
2.5.2 Utilization of Multiple Data Sets . . . . . . . . . . . . . . . . . . 12
2.5.3 Weighting of the Model Output . . . . . . . . . . . . . . . . . . 13
2.5.4 Model Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.6 Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3 Numerical Solution Methods 16

3.1 Numerical Solution to the IVP . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.1 Euler-Integration . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.2 Higher Order Numerical Integration Methods . . . . . . . . . . . 17
3.2 Numerical Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.1 Topology of the Objective Function . . . . . . . . . . . . . . . . 19
3.2.2 Gradient Based Numerical Optimization . . . . . . . . . . . . . . 26
3.2.3 Numerical Optimization Without Gradient Information . . . . . . 31
3.2.4 Improving the Performance of the Numerical Optimization . . . . 32
3.3 Implementation of a System Identification Algorithm in MATLAB® . . . 35
v
CONTENTS vi
4 Case Studies 38
4.1 Free Fall with Drag due to Air . . . . . . . . . . . . . . . . . . . . . . . 38
4.1.1 The Riccati Equation . . . . . . . . . . . . . . . . . . . . . . . . 38
4.1.2 System Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.1.3 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . 40
4.1.4 System Identification Based on One Data Set . . . . . . . . . . . 42
4.1.5 System Identification Based on Two Data Sets . . . . . . . . . . 45
4.1.6 Statistical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.1.7 System Identification Based on More Than Two Data Sets . . . . 52
4.1.8 System Identification of the Model Parameter and the Initial Con-
ditions of the Experiment . . . . . . . . . . . . . . . . . . . . . . 53
4.2 The Duffing Equation and the Nonlinear Mass and Spring System . . . . 56
4.2.1 Simulation Model . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2.2 System Identification Based on Two Data Sets A and B . . . . . . 58
4.2.3 Statistical Analysis – Monte-Carlo Simulation . . . . . . . . . . 61
4.2.4 Multiple Starting Point Procedure . . . . . . . . . . . . . . . . . 65
4.3 Dynamic Friction Models – Stribeck Effect . . . . . . . . . . . . . . . . 68
4.3.1 Modeling of Friction . . . . . . . . . . . . . . . . . . . . . . . . 68
4.3.2 The LuGre Friction Model . . . . . . . . . . . . . . . . . . . . . 69
4.3.3 The Simulation Model . . . . . . . . . . . . . . . . . . . . . . . 70
4.3.4 System Identification of the LuGre Friction Model . . . . . . . . 78
5 Conclusion 85
List of Figures
1.1 Block diagram model of a system with an input u(t) and an output y(t). . 3
3.1 Fitting a polynomial to noisy data. . . . . . . . . . . . . . . . . . . . . . 21

3.2 Contour plot of the objective function for a linear curve fitting problem in
two parameters ψ1 and ψ2 . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3 The objective function for the nonlinear curve fitting problem in one pa-
rameter for y(x) = sin(ψx). . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4 The objective function for the nonlinear curve fitting problem in two pa-
rameters for y(x) = ψ1 sin(ψ2 x). . . . . . . . . . . . . . . . . . . . . . . 24
3.5 The objective function for the nonlinear curve fitting problem in one pa-
rameter for y(x) = eψx . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.6 The objective function for the nonlinear curve fitting problem in two pa-
rameters for y(x) = ψ1 eψ2 x . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.1 Local minima of the objective function for the free fall experiment A. . . 43
4.2 Position data and estimate of the object in free fall with drag due to air
resistance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Velocity estimate of the object in free fall with drag due to air resistance. . 45
4.4 Local minima of the objective functions for the free fall experiment A and
B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.5 Estimated position trajectories for the free fall experiment based in the
data sets A and B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.6 Estimated velocity trajectories for the free fall experiment based in the
data sets A and B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.7 Cumulative distribution function of the results of the Monte-Carlo simu-
lation for the free fall experiment based on one and two data sets A and
B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.8 Histogram of the results of the Monte-Carlo simulation for the free fall
experiment based on one data set A. . . . . . . . . . . . . . . . . . . . . 51
4.9 Histogram of the results of the Monte-Carlo simulation for the free fall
experiment based on two data sets A and B. . . . . . . . . . . . . . . . . 52
vii
LIST OF FIGURES viii
4.10 Expected standard deviation due to disturbance induced variation of the

parameter ψ when combining multiple data sets. The results are based on
a Monte-Carlo simulation. . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.11 Contour plot of the objective function ε(ψ) evaluated for a two dimen-
sional slice of the parameter space: ψ1 ∈ [0, 1], ψ2 ∈ [25, 35] and ψ3 = 0. . 55
4.12 Contour plot of the objective function ε(ψ) evaluated for a two dimen-
sional slice of the parameter space: ψ1 = 0.5, ψ2 ∈ [25, 35] and ψ3 ∈ [−5, 5]. 55
4.13 Spring characteristic for a linear, a hardening and a softening spring. . . . 57
4.14 System inputs uA (t) and uB (t) for the experiments A and B simulated on
the mass and hardening spring system. . . . . . . . . . . . . . . . . . . . 59
4.15 Comparison of the estimated system outputs of experiment A based on
the results ψA , ψA and ψAB of the system identification. . . . . . . . . . . 60
4.16 Comparison of the estimated system outputs of experiment B based on
the results ψA , ψA and ψAB of the system identification. . . . . . . . . . . 61
4.17 Comparison of the simulation output of experiment A corresponding to
all data sets ψA , ψA and ψAB identified in the Monte-Carlo simulation. . . 62
4.18 Comparison of the simulation output of experiment B corresponding to
all data sets ψA , ψA and ψAB identified in the Monte-Carlo simulation. . . 62
4.19 Comparison of the results of the Monte-Carlo simulation in the form of
histograms for the identified parameter k based on the identification data
generated from experiment A and B as well as both of them combined. . . 63
4.20 Histogram of the identified parameter kAB after the removal of outliers. . . 64
4.21 Comparison of the system response to the verification test case with input
uV (t). The system is parameterized according to the selected results ψA,s ,
ψB,s and ψAB,s as well as ψexct . . . . . . . . . . . . . . . . . . . . . . . . 65
4.22 Comparison of the objective function values ε(ψAB ) for each identified
parameter set ψAB based on a different initial guess that lead to a conver-
gence of the numerical optimization. . . . . . . . . . . . . . . . . . . . . 67
4.23 Comparison of the simulation output of experiment A and B for all 682
identified parameter sets corresponding to ε(ψ) < εthresh compared to the
identification data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.24 Triangular signal u(t) that is applied to the LuGre friction model in order
to analyse its properties. . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.25 System response x(t) to the triangular input signal u(t) at smaller excita-
tion levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.26 System response x(t) to the triangular input signal u(t) at larger excitation
levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.27 (x(t), Ff (t)) characteristic for smaller excitation levels u(t) < μ0 mg. . . . 74
4.28 (x(t), Ff (t)) characteristic for higher excitation levels u(t) > μ0 mg. . . . . 74
LIST OF FIGURES ix
4.29 (v(t), Ff (t)) characteristic for smaller excitation levels u(t) < μ0 mg. . . . 75
4.30 (v(t), Ff (t)) characteristic for higher excitation levels u(t) > μ0 mg. . . . . 75
4.31 Numerically approximated Jacobian for a triangular input signal u(t) with
û = 0.5μ0 mg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.32 Selected components of the numerically approximated Jacobian for a tri-
angular input signal u(t) with û = 0.5μ0 mg. . . . . . . . . . . . . . . . . 77
4.33 Numerically approximated Jacobian for a triangular input signal u(t) with
û = 1.5μ0 mg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.34 Triangular input signal uA (t) with û = 0.5μ0 mg that is applied to the Lu-
Gre friction model on order to generate identification data. . . . . . . . . 79
4.35 Identification data set generated by adding noise to the sampled response
of the exactly parameterized system to the input signal uA (t). . . . . . . . 80
4.36 Identification data set generated by adding noise to the sampled response
of the exactly parametrized system to the input signal uB (t). . . . . . . . 81
4.37 Comparison of the system response to the input signal uA (t) for the exact
and the identified parameter sets. . . . . . . . . . . . . . . . . . . . . . . 84
4.38 Comparison of the system response to the input signal uB (t) for the exact
and the identified parameter sets. . . . . . . . . . . . . . . . . . . . . . . 84
List of Tables
4.1 Results of the system identification based on the data from the free fall
experiment A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.2 Results of the system identification based on the data from the free fall
experiment A and B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3 Results of the Monte-Carlo simulation of the system identification of the
free fall experiment based on one and two data sets. . . . . . . . . . . . . 50
4.4 Results of the Monte-Carlo simulation of the system identification of the
parameter and the initial conditions of free fall experiment. . . . . . . . . 56
4.5 Results of the Monte-Carlo simulation for the nonlinear mass and spring
system after the removal of outliers. . . . . . . . . . . . . . . . . . . . . 64
x
Chapter 1
Introduction
1.1 Problem Formulation

System Identification is the determination of the properties of a system based on observa-
tions made of the behaviour of that system. For controlled dynamic systems these obser-
vations relate to measurement time series corresponding to a specific applied input signal,
i.e. input-output data sets. The goal of system identification is finding a parametrization
for a simulation model such that this model produces the same input-output behaviour as
presented by the given identification data. The main task of this work is the exploration
and application of numerical solution methods with the goal of performing the system
identification of nonlinear dynamic systems.
As a dynamic system is typically described by a model consisting of differential equa-
tions, the task of system identification is the determination of the parameters of that sim-
ulation model.
The first part of the solution process is the implementation of numerical solution meth-
ods for these differential equations. The simulation output of an arbitrarily parametrized
model is the solution of an initial values problem. Under the correct parametrization this
should be reflecting the measurement data set extracted from an experiment conducted
on the object of interest. The initial values of that experimental setup can be assumed
to be known based on the selected experiment design or estimated in the identification
procedure.
To be able to evaluate the fit of the model to the system an optimization criterion, i.e. a
goodness-of-fit measure, needs to be selected. Starting from an initial guess for the param-
eters, the second part of the solution process is the application of numerical optimization
methods. To improve the fit of the model, the parametrization is changed in an iterative
manner, optimizing the goodness-of-fit measure step by step.
Once this process converges onto a final parametrization of the model, the last step of
the system identification procedure is the verification of the results. The estimated model
1
CHAPTER 1. INTRODUCTION 2
output can be compared to the measured data set. This can be done by visual comparison
or limited statistical analysis. In addition to that, separate validation data sets from dif-
ferent experiments can be used to compare the performance of the model under different
operating conditions or the identification data can be split up in a process called cross-
validation.
When dealing with measurement signals perturbed by noise it is of interest to look at the
influence the noise has on the identified model parameters. The influence of gaussian
noise in the data sets as well as the utilization of multiple data sets in order to minimize
the influence of noise will be explored.
1.2 Systems and Models

The terms system and model are used in various disciplines. In general the term system
refers to a real or imaginary structure consisting of smaller parts or subsystems. The be-
haviour of the system is defined by the interactions and relations in between those parts.
A more comprehensive definition of a system can be found in [1]: the CESM model,
which defines a system as a structured object consisting of a composition, environment,
structure and mechanism.
A model of a system is an artificial representation of that system created for some purpose.
The purpose can be as simple as merely being a simplified or more concise depiction of
the system or a portrayal of the fundamental working principles of the system.
When models are used to perform experiments on systems that would be too dangerous,
expensive or time-consuming to be performed on the real counter part, or simply impos-
sible to be performed as the real counterpart doesn’t exist, we speak of simulation models
[2]. Models can be built in various ways. A model can be formulated verbally, as a graph-
ical model or a physical model in form of a real and possibly miniature replication. The
feasibility to perform a simulation on a model and extracting meaningful or quantitative
results out of it are dependent on the kind of model and various other factors, nonetheless
the accuracy with which the model is able to predict the behaviour of the represented sys-
tem, and the experimental design.
The idea of quantifying features brings us to mathematical models. Assuming we want
to build a simulation model, the mathematical model needs to consist of a description of
the state of the system and an output signal y(t), in order to be able to extract information
out of it. To attribute the observed output of the model to its inner workings, we assign so
called state variables x(t), which describe the state of the system at any point in time. In
many cases a system will not be fully self contained. There will be interactions with the
environment. These external influences are realized as input signals u(t), which might be
control inputs that a user or operator can define and change at will, or known or unknown
disturbances which can’t be influenced but that need to be taken into account when de-
u(t) y(t)
S
Figure 1.1: Block diagram model of a system with an input u(t) and an output y(t).
scribing the behaviour of the system. A simple block diagram model of a system S can be
seen in Fig. 1.1.
The main difference between a static and a dynamic system is that for a static system
the output only depends on the current state and input of the system. In that sense a static
system can be described by a simple algebraic equation:
y(t) = f (u(t)). (1.1)
For a dynamic system the observed output is dependent on the past history of the input,
i.e. for any sequence of the input that is applied to the system over time, the system
evolves in a different way. The tool that allows us to encode the effect that the state
of the system and the input to the system at a certain point in time have on the change
of the trajectory is the differential equation. The internal mechanisms of the system are
defined by differential equations describing the interaction of the internal properties in
between themselves as well as their interaction with the external influences. Higher order
differential equations can be transformed into first order ones by assigning additional state
variables. This results in the state space representation of the dynamic system as a system
of first order differential equations [3]:
∂x
= f (x(t), u(t),t). (1.2)
∂t
The output variables y(t) are the selected properties of the system that we are interested
in. They represent the measurements that we are making. The outputs y(t) at any point
in time are related to the external inputs u(t) at that time as well as the internal state of
the system x(t) at that time. This means that once the state space equation is solved and
we know the trajectory of the state variables for the time period that we are interested in,
the measurement variables are derived from a regular equation, the observer equation
y(t) = g(x(t), u(t),t). (1.3)
1.3 System Identification and its Place in Control Theory

The three main and integral parts of a controlled system, as seen in Fig. 1.1 are the
behaviour of the system S , the control input u(t), and the system output y(t), whereas the
behaviour of the system is described by the relations of the internal state variables x(t).
Having knowledge about any two of these three attributes of a system results in one of
three distinguished kinds of problems of determining the third property [4].
The first kind of task is the simulation problem. Given a system S and a control input u(t),
find the output y(t). As a forward problem, this is the simplest kind of the three types of
problems that were hinted at above. In general, there exists one unique solution that is the
result of solving an initial value problem (IVP) for the system of differential equations
describing the system S from the initial state of the system x(t0 ) at the initial time t0 over
the time frame of the defined control input u(t).
The second kind of problem is the control problem. Given a system S and a desired
system output, i.e. behaviour, y(t), find the input u(t). This is an inverse problem that
may not have a single unique solution. Depending on the exact problem statement, it will
result in one or multiple correct solutions. In general these kind of problems are more
loosely formulated in respect to the requirements on the desired behaviour of the system.
In many cases it is demanded that a system is brought to a specific state, but the specific
path to take and the exact time frame in which to achieve this are not exactly specified.
In fact there are infinitely many possible paths and corresponding control sequences that
move a system from one state to another [5]. In many cases there are also compromises
to be made. For example in a classical PID controlled system faster rise time needs to be
weighted against the ensuing overshoot and oscillations.
The third and last kind of problem is the system identification problem. Given an input u(t)
and an output y(t) of a system, find a system model S that is able to describe this observed
input-output behaviour. This again is an inverse problem that results in many possible
solutions. The quality of the identified model can be assessed in two ways. On the one
hand the ability to represent or interpolate the given input-output data, i.e. the goodness-
of-fit, is the determining factor in the solution of the system identification problem. On the
other hand the suitability of the identified model to be able to make accurate predictions
for different operating conditions when applied in a simulation problem might be of great
importance. Depending on the intended application of the model, shortcomings in this
suitability for extrapolation would rule out models regardless of the goodness of fit to the
given identification data.
1.4 Why are Linear Models Preferred over Nonlinear Mod-

els?
This work is aimed at the system identification of nonlinear dynamic systems. This makes
it necessary to deal with the difficulties that come with nonlinear systems compared to
linear ones. Therefore, in this section some of the differences and the available methods
to deal with linear and nonlinear dynamic systems are discussed.
The main argument as to why one should always favour modeling a system as a linear
model rather than a nonlinear one is that linear models are naturally easier to deal with.
As per definition linear systems have a more narrowly defined general structure
ẋ(t) = A(t)x(t) + B(t)u(t) (1.4)
compared to a general nonlinear model
ẋ(t) = f (x(t), u(t),t) (1.5)
Figuratively, if the space of all system models was a sea, the space of linear systems
would just be a little island. Because of both the simplicity and the generality of the
linear system, the focus of research has always been weighted on this side. Furthermore,
many nonlinear problems can only be approached by local approximation of the nonlinear
problem by a linear one. Adding to that, naturally, any approach to deal with some type
of problem may work on simple tasks but break down on more complicated ones.
This has lead to a lot of different ways to analyse and work with linear systems. For once
it is possible to look at linear systems in the time and frequency domain. This is enabled
via the Laplace- and the Z -transformation. The Laplace transform enables to solution
of a certain class of partial differential equations, which correspond to single input linear
time invariant dynamic systems:
∂ ny ∂y ∂ mu ∂u
an n + · · · + a1 + a0 y(t) = bm m + · · · + b1 + b0 u(t) (1.6)
∂t ∂t ∂t ∂t
as algebraic equations in the complex s domain:
Y (s) an sn + · · · + a1 s + a 0
H(s) = = . (1.7)
U(s) bm sm + · · · + b1 s + b0
Note that here the initial conditions are set to be zero. The transfer function H(s) is
based on the Laplace transform of the impulse response of the system. Any input u(t)
can be approximated as a sum of discrete impulses. The response of a linear system to
the input u(t) is therefore the superposition of the response to each of these impulses. For
an infinitesimally fine discretization this corresponds to a convolution integral in the time
domain
y(t) = h(t) ∗ u(t) , (1.8)
where h(t) is the response of the system to a unit impulse. In the s domain, the convolution
simplifies to a simple multiplication:
Y (s) = H(s)U(s). (1.9)

For nonlinear dynamic systems this is less applicable. The input-output behaviour of a
nonlinear dynamic system can not simply be classified by an impulse response like in
linear systems as the principle of superposition does not apply [6].
There are various different solution methods available to solve the IVP for a linear sys-
tem. Besides finding an analytic solution, which is arguably easier for linear systems, any
linear or nonlinear vector field defined as in equation (1.4) or (1.5) can be numerically
integrated. The simplest algorithm to numerically integrate the solution of the IVP is the
Euler integration. More advanced algorithms like Runge-Kutta type methods apply an
iterative approach with varying step size in order to get more accurate results. Another
approach is to discretize the state variables in time and to apply a numerical differentia-
tion matrix in order to approximate the discretized derivatives of each state variable. For
linear systems this transforms the system of linear differential equations into a system of
linear equations, which can be solved directly as a least squares problem with equality
constraints, which enforce the initial conditions [7]. It is therefore called the Global Least
Squares solution method. The power of this solution method lies in the fact that it can also
be used for the solution of the inverse problem [8]. As was recently shown, the Global
Least Squares solution method in conjunction with the variable projection method can be
used in the time domain system identification of linear state-space models [9].
The Global Least Squares solution method can also be applied to the solution of the IVP
for a systems of nonlinear differential equations, but this results in a system of nonlinear
equations and the least squares fit of which needs to be optimized in an iterative manner,
relying on good initial estimates of the solution [10].
For homogeneous i.e. unforced systems of linear differential equations it is also possible
to deploy the exponential matrix, which is based on the analytic solution of a linear sys-
tem, to compute the state of the system at any point in time [11].
When general nonlinear models are considered, they are often represented as a kind of
subclass called hybrid models, e.g. a Wiener model, which consists of a linear system
followed by a static nonlinearity, or a Hammerstein model, which consists of a nonlinear
actuator that feeds into a linear system [12]. This approach comes with the advantage
that techniques for the analysis of linear systems can be applied for the analysis of the
respective part of the hybrid model.
To summarize, while there is a whole array of methods available to deal with linear dy-
namic systems, the general approach to solve the simulation problem for a nonlinear
dynamic systems is an approximation of the solution via numerical integration. Further-
more, the fact that the input-output behaviour of a nonlinear dynamic system cannot be
classified by a single impulse response has consequences on the selection of identification
data for the system identification problem of nonlinear dynamic systems.
Chapter 2
System Identification Theory
In this chapter some of the core principles of the system identification theory are dis-
cussed. These include an overview of the system identification procedure, the selection
of a model structure whose parameters are to be identified, the experimental design that
is used in order to extract information about the object of interest, the goodness-of-fit cri-
terion that is based on the output error of the model, and the verification of the estimated
model parametrization.
2.1 System Identification as a Part of System Modeling

A dynamic system evolving over time is described by a mathematical model in the form
of input variables u(t), internal state variables x(t) and output variables y(t) as well as a
description of the relationships between the inputs, the internal state and the outputs [2].
The task of system modeling in terms of building a mathematical model is the determina-
tion of the input variables u(t), output variables y(t) and the necessary number of state
variables x(t) to describe the relations between the inputs and outputs.
In theoretical or physical modeling, so called first principles, i.e. physical laws, are used
to describe the behaviour of a system. For example the motion of an object of mass sub-
ject to forces and gravity can be described by Newton’s laws. Ohm’s law and Kirchhoff’s
laws can be used to describe the current in an electrical circuit. Larger systems can be
built as the combination of smaller subsystems whose behaviours are known. Sometimes
this is also called mechanistic modeling and can be performed supported by software like
Modelica or Simscape [13].
System identification is commonly used in conjunction with physical modeling or as a
complement thereof [2]. This means that physical modeling alone doesn’t lead to a work-
ing simulation model which actually represents a specific real world counterpart. In most
cases precise knowledge about a number of parameters ψ is required in order for the
simulation model to match the behaviour of the modeled system. Such a model can be
7
CHAPTER 2. SYSTEM IDENTIFICATION THEORY 8
described as
ẋ(t) = A(ψ)x(t) + B(ψ)u(t) (2.1)

y(t) = C(ψ)x(t) + D(ψ)u(t) (2.2)
for a linear time invariant (LTI) system, or as
ẋ(t) = f (x(t), u(t), ψ,t) (2.3)

y(t) = g(x(t), u(t), ψ,t) (2.4)
for a general, presumably nonlinear system in state space form. The determination of the
parameters ψ for any given model structure is the system identification problem.
2.2 The System Identification Procedure

System identification is a kind of modeling, which relies on observations from the system
in order to determine its properties. As such it is also called data driven modeling or
experimental modeling. To be more precise, not necessarily the properties of the system
but the properties of the system model are determined. Information about the system is
extracted in the form of input-output data sets [u(t), y(t)]. This is done by the means of
performing experiments on the object or process of interest. For a controlled dynamic
system a defined control sequence u(t) is applied to the system and the output y(t) is
measured.
This information is used to determine the parameters of a design model. Depending on
the amount of a priori information that is used in the determination of the structure of
the design model, so called black-box models or universal approximators can be distin-
guished from so called white-box or gray-box models [13].
Once input-output data is available and a design model is decided upon, the system iden-
tification problem is reduced to the task of fitting the output of the model for the given
input sequence to the output data of the system. For nonlinear dynamic systems this
is a nonlinear curve fitting problem, which can be formulated as the optimization of a
goodness-of-fit criterion.
2.3 Building a Design Model
2.3.1 Black-Box Models

One area of system identification involves the modeling of a processes or systems using
little if any a priori knowledge. The basis of black-box models are general mathematical
model structures which can be scaled in their differentiation order and therefore also in
their number of parameters. For example, this could be a general LTI system model in the
from of a state-space or transfer function representation, or a nonlinar model in the form
of a Wiener or Hammerstein model. In general, higher order models are more flexible in
the way that they can be more accurately fitted to any arbitrary data. This is similar to the
task of fitting a polynomial of varying degree to data. n data points can be approximated
in a vertical distance least squares sense by a polynomial of degree d < n − 1, but a poly-
nomial of degree d ≥ n − 1 can fit the n data points exactly.
This process of determining, based in input-output observations, to which class of systems
a specific one belongs is also called system classification [14]. With access to different
model structures of varying order, a process of trial and error can be deployed. Various
model structures of various differentiation order are fitted to the observation data in order
to make a selection on the best fitting one. Special attention must be paid to this differ-
entiation order. Ideally, the order of the model is set as high as necessary to accomplish
sufficient flexibility to be able to accurately fit the input-output data sets while keeping it
as low as possible to avoid a potential over fitting.
The advantage to this approach is clear. The underlying physical principles of the system
aren’t needed to be able to build a model that can replicate the input-output behaviour of
the system.
The main disadvantage of black-box models lies in the name and coincides with the prop-
erty of inverse problems. The solution of the system identification problem is not uniquely
defined. Black-box models can be used to accurately describe the relation of the input and
the output of a dynamic system, but the internal mechanisms of the model might be dif-
ferent to the ones in the real system. Therefore, even if a model is found that correctly
depicts the system behaviour, it might still be difficult to assign meaning to the identified
parameters.
Another disadvantage of black-box models is that in their role of universal approxima-
tors these general model structures are not necessarily designed to fit a specific system.
In order to fit different input-output behaviours, black-box models are typically over
parametrized, i.e. they have more flexibility than might be needed for any given task.
This does not mean that a black-box model is less able to form a good simulation model,
but that it might be more difficult to distinguish necessary and unnecessary parameters in
the process of system identification i.e. it might be more difficult to successfully execute
a numerical optimization of the larger range of parameters [15].
In summary it can be said that the task of system identification of black-box models re-
quires less information about the object of interest and less work leading up to the process
of system identification but also results in a system model that gives less insight into the
system that is to be identified.
2.3.2 White- and Grey-Box Models

So called white-box or gray-box models are the result of physical or theoretical modeling
[2]. They represent a system model as described by (2.3) with unknown parameters ψ that
are to be determined. According to [13] different shades of grey are used to denominate
the degree of disparity from a clear-white model, which would be a model based purely
on physical laws where all parameters are known. Darker shades reflect the presence of
unknown physical constants which are parameters that are to be determined by system
identification or a model that is not purely based on physical laws but that also might
include empirical models and simplified assumptions as an abstraction to the real system.
The advantage of using a so called white-box model is that as much a priori knowledge
as possible is used in the formulation of the system model. An engineer can investigate
the object of interest, map out all the relevant connections, and use first order principles
to quantify them. Therefore each state variable and each parameter can be attributed to
actual physical properties. This for one makes it easier to get good initial estimates for the
task of system identification. The rough range of at least some of the parameters can be
estimated and bounds can be placed on them based on physical limitations (e.g. a spring
stiffness or a damping coefficient will never be negative). In addition to that it can be easy
to distinguish between right and wrong results (local minima). Starting from a sensible
initial guess and having knowledge about the implications of the limitations of the system
make it easier to get correct results or handle a failure in the convergence to a good result
in the system identification procedure.
The main disadvantage of white-box models compared to black-box models is that every
relevant interaction needs to be correctly determined and incorporated into the model.
Otherwise the systematic error built into the model will be compensated by a wrong
parametrization which leads to the best possible fit.
In Summary it can be said that the task of system identification of white-box models re-
quires a lot of information and knowledge about the kind of system under investigation,
but also results in a system model that gives full insight into the internal behaviour of the
system and that comes with knowledge about the meaning of the identified parameters.
2.4 Experimental Design

In order to perform a system identification, information in the form of observations of the
behaviour of the system that is to be investigated needs to be at hand. In order to extract
information out of the system, experiments need to be performed. The design of those
experiments can have serious effects on the results of the system identification. Espe-
cially when dealing with nonlinear systems, which might behave differently depending
on the excitation level, an appropriate experimental design needs to be determined. Some
parameters that are to be identified might have little influence on the system output in cer-
tain operational regimes. This makes the identification difficult, especially when the noise
level in the measurement data is larger than the influence that a specific parameter has on
the system output, which might result in a large disturbance induced variation of the iden-
tified parameter value. This means that, for a single input system, an input sequence u(t),
t ∈ [t0 , t f ], needs to be selected which leads to a proper excitation of the system. In many
cases it might be advantageous to consider the combination of multiple different input-
output data sets as basis for the system identification. Different input sequences need to
be designed accordingly. In order to extract suitable identification data sets, at least the
whole operational regime of the system of interest should be considered.
Furthermore, the experimental design has an effect on the computability of the numerical
solution of the system identification problem. In cases where the solution of the simula-
tion problem already represents a challenge, the experiment could be designed in a way
which eases this. In cases where it is difficult to find a global minimizer for the non-
linear curve fitting problem, a different experimental setup might lead to improvements
in respect to less local minima or at least a clearer distinguishability of local and global
minima.
2.5 The Goodness-Of-Fit Measure
2.5.1 Vertical Distance Least Squares Fit of the Model Output

Once a design model is determined or decided upon, the task of system identification
is reduced to fitting the output of that model to the observed measurement data. The
execution of an experiment corresponds to the computation of a simulation of the system
model. For any parametrization ψ of the model, an initial value problem can be solved,
which corresponds to the execution of the mentioned experiment. The observations that
were made at defined points in time during the experiment correspond to discrete points
in time of the simulation. Therefore, for each measurement ξi at time ti a model output
y(ti ) can be computed. Identifying the optimal parameters ψ means finding the best fit
of the model output y to the measurements ξ. For measurements at fixed points in time
the sum of the squared vertical distances can be used as the goodness-of-fit Measure. This
reduces the problem of fitting a function to each point in the given data to the problem of
minimizing a scalar valued optimality criterion ε(ψ):
ψ = argmin ε(ψ). (2.5)

ψ
ε(ψ) can be defined in the following equivalent ways for n measurement points:
n
ε(ψ) = ∑ [ξ (ti ) − y(ti )]2 (2.6)
i=1
ε(ψ) = ξ − y(ψ)22 (2.7)
ε(ψ) = [ξ − y(ψ)]T [ξ − y(ψ)] (2.8)
2.5.2 Utilization of Multiple Data Sets

When multiple sets of measurements are available for the task of system identification, it
makes sense to utilize all of the available information in order to identify the best possible
model parametrization. For example two experiments might lead to the measurement
data sets ξA and ξB . If both of these correspond to the same experimental setup with the
same system input u(t) then the objective functions that described the sum of squares with
respect to each one of them are
εA (ψ) = ξA − y(ψ)22 and (2.9)

εB (ψ) = ξB − y(ψ)22 . (2.10)
These two objective functions can be minimized simultaneously by combining them into
2
ξ y(ψ)
A
εAB (ψ) = − . (2.11)
ξB y(ψ) 2
When the results of two different experiments are used, there ought to be two different
functions yA (ψ) and yB (ψ) for the model output.
2
ξ yA (ψ)
A
εAB (ψ) = − (2.12)
ξB yB (ψ) 2
For a system with multiple measured output variables a similar approach can be applied.
2.5.3 Weighting of the Model Output

In some problems it can be advantageous to put additional emphasis on specific portions
of the system output by adding a weight wi to the least squares difference at the point in
time ti .
n
ε(ψ) = ∑ wi [ξ (ti ) − y(ti )]2 (2.13)
i=1
In matrix form this can be represented by a weighting matrix W with diagonal entries wi :
ε(ψ) = W (ξ − y(ψ))22 . (2.14)
The weighting of a specific model output can be especially useful when performing the
system identification based on multiple data sets. Separate weights can be asserted on the
different data sets in order to balance out the effect that each individual data set has on the
overall objective function value.
2
W 0 ξA yA (ψ)
A
εAB (ψ) = − (2.15)
0 WB ξB yB (ψ) 2
2.5.4 Model Uncertainty

Disturbance Induced Variation
The objective function ε(ψ) is positive definite and would be zero if ξ = y(ψ). In general
this equality will never be satisfied because of noise perturbations δ in the measurement,
that can’t be reproduced by the model [16].
ξ = yexct + δ = y(ψ ) (2.16)
If the parameter set ψ corresponding to the global minimum of equation (2.6) is found,
one might expect the value of the objective function to be the norm of the noise δ22 .
This is true when speaking about the expected value in its statistical meaning. When
performing the system identification based on one set of noisy measurements ξi resulting
in the parameter set ψi , the estimated model output will not optimally interpolate the
exact system output yexct but rather try to approximate the measurement noise as good as
possible, as it is minimizing the sum of squares of [ξi − y(ψi )]. For an unbiased estimator
the expected value is the exact value of the parametrization of the underlying system that
produced the observations. However, because of the limited available measurement data,
the estimated results will deviate from the exact parametrization.
The important insight to take from this is that
ε = ε(ξ, ψ) = ε(yexct + δ, ψ) and therefore (2.17)

ψ = ψ (ξ) = ψ (yexct , δ) (2.18)
i.e. any identified parameter set ψ depends on the design model y(ψ) representing the
real system, the experimental design used to extract information about the system repre-
sented by yexct , and the accuracy of the measurement represented by δ.
This brings us to the term model uncertainty. As the disturbance in the measurement
induces a variation of the objective function which results in a variation of the identi-
fied parameters, so does this uncertainty of the parameters affect the output of the model.
Model uncertainty can usually be counteracted by increasing the measurement time, as
this will decrease the signal to noise ratio resulting in less disturbance induced varia-
tion [17]. For gaussian measurement noise the sum of squares optimality criterion is the
maximum likelihood estimator [18]. Under this condition and with infinite amounts of
identification data, the minimization of ε(ψ) would result in the exact parameter set. The
problem is that in practice there will only ever be a finite amount of identification data
available.
Influence of the Numerical Optimization
As ψ is the results of a numerical optimization of the curve fitting problem, the applied
numerical solution methods also has an effect on this solution. For a well posed curve
fitting problem that is optimized with a correctly set up numerical optimization method it
can be assumed that the influence of a numerical discretization error on the identified pa-
rameter set is at least small compared to a disturbance induced variation if not negligible,
as long as the iterative solution method is able to converge to a global minimizer of ε(ψ).
2.6 Verification
The task of verifying the validity of the model in its purpose of describing the behaviour
of the object of interest largely depends on the operational range in which the simulation
model is to be used. At a bare minimum the simulation model should be able to reproduce
the behaviour observed in the experiments used as a basis for the system identification.
This can be verified by comparing the estimated system output of the system model to the
identification data sets.
Of course, it would be advantageous if the simulation model would make it possible to
accurately predict the behaviour of the system under different operating conditions than
presented in the given identification data of the system. Therefore, it is useful to have a
verification data set at hand, which was measured in a separate experiment, that was not
included in the identification procedure. The simulated model output under the identified
parametrization for this experimental setup can then be compared to the corresponding
verification data set in order to evaluate the capability of the simulation model not only to
reproduce the observed behaviour presented by the identification data sets but also to be
able to predict the behaviour of the object of interest under different operating conditions.
In general, this procedure is called cross-validation, where only a subset of the available
data is used for the estimation and the rest is used for the validation.
Chapter 3
Numerical Solution Methods
In this chapter the two main numerical tools that can be used in the system identification
of nonlinear dynamic systems are looked at: The numerical approximation of the solution
of initial value problems and the numerical solution of nonlinear curve fitting problems. A
selected number of the available algorithms for each of these problems are discussed. At
the end of this chapter an overview of one implementation of an algorithm for the system
identification of nonlinear dynamic systems is presented.
3.1 Numerical Solution to the IVP

A dynamic system described in state space form is defined as a vector field
∂x
ẋ(t) = = f (x(t), u(t),t). (3.1)
∂t
The problem of finding a solution to this system of first order differential equations for a
control input u(t) and an initial state x(t = t0 ) = x0 is called the initial value problem.
For simple systems, this can be done analytically. However, for most nonlinear systems
there is no known analytic solution and the best approach to solving them is to develop a
numerical approximation which is good enough [19].
Given the inputs u(t) for t ∈ [t0 ,t f ] and the state x(t) of the system at time t0 the solu-
tion of the initial value problem for any vector field as defined in equation (3.1) can be
approximated by numerical integration.
3.1.1 Euler-Integration
Numerical integration is the iterative extrapolation of the state trajectory x(t) from an
initial point x(t0 ) based on gradient information given by ẋ(t) = f (x(t), u(t),t). For a
given u(t) this simplifies to ẋ(t) = f (x(t),t).
Euler integration is one of the simplest numerical integration methods. It is based on the
16
CHAPTER 3. NUMERICAL SOLUTION METHODS 17
approximation of the increment Δx = x(t + Δt) − x(t) with the help of the derivative
ẋ(t) = f (x(t),t). The equation for it can be derived by truncating the Taylor series
expansion of x(t) after the first derivative term:
x(t + Δt) ≈ x(t) + Δtẋ(t) (3.2)

x(t + Δt) ≈ x(t) + Δtf (x(t),t) (3.3)
This is the explicit Euler integration method as the approximation of the state at time
t + Δt explicitly depends on the state of the system at time t.
By expanding the Taylor series from the point t + Δt backwards a slightly different equa-
tion for the approximation of the increment can be derived:
x(t + Δt) ≈ x(t) + Δtẋ(t + Δt) (3.4)

x(t + Δt) ≈ x(t) + Δtf (x(t + Δt),t + Δt). (3.5)
This is an implicit equation in x(t + Δt) and is therefore called the implicit Euler inte-
gration formula. It is a more stable numerical integration method but comes with the
disadvantage of the computational effort to solve this implicit equation for x(t + Δt) in
every step.
On a side note, equation (3.2) corresponds to a forward finite difference formula for the
approximation of the derivative:
x(t + Δt) − x(t)

ẋ(t) ≈ (3.6)
Δt
and equation (3.4) corresponds to a backwards finite difference formula:
x(t + Δt) − x(t)

ẋ(t0 + Δt) ≈ . (3.7)
Δt
3.1.2 Higher Order Numerical Integration Methods

Higher order numerical integration methods make use of higher order finite difference
formulas and function evaluations at multiple intermediate steps in time. Runge–Kutta
methods are one-step multistage methods, which make use of multiple function evalua-
tions for each step Δt in order to achieve a better approximation of the state of the system
at time t + Δt [20].
n
x(t + Δt) ≈ x(t) + Δt ∑ wi ki (3.8)
i=1
For example, the fourth order Runge-Kutta method uses four evaluations of ẋ(t) = f (x(t),t)
for every step Δt [21]:
k1 = Δtf (x(t),t) (3.9)

1 1
k2 = Δtf (x(t) + k1 ,t + Δt) (3.10)
2 2
1 1
k3 = Δtf (x(t) + k2 ,t + Δt) (3.11)
2 2
k4 = Δtf (x(t) + k3 ,t + Δt) (3.12)
The approximation of x(t + Δt) is computed with w1 = w4 = 1

6 and w2 = w3 = 31 :
1 1 1 1
x(t + Δt) ≈ x(t) + k1 + k2 + k3 + k4 . (3.13)
6 3 3 6
The MATLAB® function ode45() is an implementation of a Runge Kutta Dormand

Prince algorithm [22]. It used two different formulas to estimate x(t + Δt). As the name
suggests, these are a fourth and a fifth order Runge-Kutta type formula. By design, it only
needs six evaluations of f (x(t),t) in order to compute both of these approximations. By
comparing the results of the fourth and fifth order algorithm, it is possible to estimate the
error due to the discrete step in time Δt, i.e. the local discretization error. This error esti-
mate is used to adapt the step size to smaller values where necessary in order to achieve
the demanded numerical accuracy as specified in the solver options or to increase the step
size where possible in order to improve the computation time.
3.2 Numerical Optimization

In any curve fitting problem a function y(x, ψ) is to be fitted to a number of N given data
points ξ. If the location of the data points respect to the independent variable x is assumed
to be exactly known, then only the vertical distance of the function y(x, ψ) evaluated at
these location [x1 , . . . , xN ] to those data points ξ needs to be optimized by adjusting the
parameters ψ. ⎡ ⎤ ⎡ ⎤
ξi y(xi , ψ)
⎢ . ⎥ ! ⎢ .. ⎥
ξ=⎢ .. ⎥ = ⎢
⎣ ⎦ ⎣ . ⎥ = y(x, ψ) = y(ψ)
⎦ (3.14)
ξN y(xN , ψ)
The fact that this equality cannot be met due to measurement noise leads to the definition
of a scalar valued optimality criterion ε(ψ) as described in section 2.5.
3.2.1 Topology of the Objective Function

Before presenting some of the algorithms that can be used to numerically optimize this
objective function, the topology of ε(ψ) for some simple problems is looked at in order
to discuss the difference between linear and nonlinear curve fitting problems.
In Linear Curve Fitting
In linear curve fitting, a design function y(x) that is to be fitted to the data is a linear
combination of n known basis functions fi (x).
n
y(x, ψ) = ψ1 f1 (x) + · · · + ψn fn (x) = ∑ ψi fi (x) (3.15)
i=1
The parameters that are determined in the solution of the linear curve fitting problem are
the n linear coefficients T
ψ = ψ 1 ψ2 . . . ψ n . (3.16)
For N given data points ξi at their respective locations xi with respect to the independent
variable x, one optimality criterion that can be used for the fit is the sum of the squared
vertical distances between the data points xi and the design function evaluated at those
locations y(xi ), i.e. the vertical distance least squares fit. This means that in order to
achieve the best fit under the assumption of gaussian measurement noise, the following
functional is to be minimized.
N
ε(ψ) = ∑ (ξi − y(xi , ψ))2 (3.17)
i=1
The same functional can also be formulated as
ε(ψ) = [ξ − y(ψ)]T [ξ − y(ψ)] (3.18)
with the discrete values ξi of the data that is to be fitted stacked on top of each other in
the vector T
ξ = ξ (x1 ) ξ (x2 ) . . . ξ (xN ) (3.19)
and the vectorized y(xi , ψ)

T
y(ψ) = y(x1 , ψ) y(x2 , ψ) . . . y(xN , ψ) . (3.20)
For a linear combination of known basis functions fi (x), y(ψ) can also be descibed as the
matrix product
y(ψ) = Vψ (3.21)
where V is a matrix similar to a Vandermonde matrix with n columns containing the

basis functions evaluated at the discrete values of the abscissae x. A Vandermonde matrix
typically only consists of evaluated basis functions which are monomials.
⎡ ⎤
f1 (x1 ) . . . fn (x1 )
⎢ . .. ⎥
V=⎢
⎣
.. ...
. ⎥ ⎦ (3.22)
f1 (xN ) . . . fn (xN )
This means that for the linear curve fitting problem the objective function can be described
as
ε(ψ) = [ξ − Vψ]T [ξ − Vψ]. (3.23)
The parameters ψ that lead to the best fit in the vertical distance least squares sense are
the ones that minimize this function.
ψ = argmin ε(ψ) (3.24)

ψ
By differentiating equation (3.23) with respect to ψ and setting the derivative to zero an
extrema can be found. The resulting criteria for this extrema is
ξ = Vψ (3.25)
which means that the the optimum is found as
ψ = V+ ξ (3.26)
where V+ is the Moore-Penrose pseudo inverse
V+ = (VT V)−1 VT . (3.27)
The important insight to take from this is that for the linear curve fitting problem an
explicit solution can be formulated that directly leads to the optimal solution. As will be
seen later, for nonlinear problems this is not the case. Iterative solution methods have to
be applied, which – starting from an initial guess – try to minimize the objective function
(3.18) step by step. The progress of the iterative solution method is determined by the
numerical optimization algorithm as well as by the topology of the objective function.
These iterative solvers can of course also be applied to linear problems. To discuss the
topology of the linear curve fitting problem, an example problem is looked at. The design
function that is to be fitted to some data is
y(x, ψ) = ψ1 x2 + ψ2 x. (3.28)
Figure 3.1: Fitting a polynomial to noisy data.
The data ξ is generated by evaluating y(x, ψ) for ψexct = [2, 12 ]T at 51 evenly spaced points
xi sampled from the interval x ∈ [−2, 3], i.e. with a step size of Δx = 0.1. Futhermore,
noise δ was added to the measurement data. This noise was sampled from the normal
distribution N (0, 1).
These data points as well as the fitted curve can be seen in Fig. 3.1. As equation (3.28)
has two parameters ψ1 and ψ2 , the value of the goodness-of-fit criterion ε(ψ) can be
visualized as a contour plot for a region of the two dimensional parameter space. This
can be seen in Fig. 3.2 for ψ1 ∈ [0, 5] and ψ2 ∈ [−2.5, 2.5]. As is typical for linear
curve fitting problems with linearly independent basis functions, the cost function forms
an ellipsoidal valley with one global minimum. This is due to the fact that the objective
function ε(ψ) is the result of squaring linear functions in ψ.
7000
6000
5000
4000
3000
2000
1000
Figure 3.2: Contour plot of the objective function for a linear curve fitting problem in two
parameters ψ1 and ψ2 .
In Nonlinear Curve Fitting
Any iterative optimization method needs to be initialized with an initial guess ψ0 . The
algorithm will in general compute steps in the parameter space that lead to a decrease in
the objective function value ε(ψ0 ). It is quite clear that for a linear curve fitting problem
with linearly independent basis functions taking downhill steps starting from any initial
point in the parameter space will eventually lead to a convergence towards the one global
minimum due to the convex topology of the objective function as visualized in Fig. 3.2.
For nonlinear curve fitting problems, this is not always the case. There might exist multi-
ple local minima which make convergence to the global minimum difficult. As an exam-
ple the function
y(x, ψ) = sin(ψx) (3.29)
is to be looked at. The data ξ that is to be fitted was generated by evaluating y(x, ψ) for
ψ = ψexct = 3 at 51 evenly spaced points x ∈ [0, 2π]. The objective function ε(ψ) for
this curve fitting problem that can be formulated as in (3.28) evaluated for ψ ∈ [−10, 10]
can be seen in Fig. 3.3. There is clearly one global minimum at ψ = ψexct . But there are
also other local minima. This means that any initial guess that is not near the exact value
might lead to a wrong result as the numerical optimization will converge into the wrong
valley.
Equation (3.29) can be augmented by adding a linear coefficient making it a function with
two parameters
y(x, ψ) = ψ1 sin(ψ2 x). (3.30)
Figure 3.3: The objective function for the nonlinear curve fitting problem in one parameter
for y(x) = sin(ψx).
Again, a curve fitting problem can be defined based on data points ξ generated by eval-
uating equation (3.30) for ψ = ψexct = [1, 3] at 51 evenly spaced points x ∈ [0, 2π]. The
objective function ε(ψ) for this example problem is visualized in Fig. 3.4 as a contour
plot for ψ1 ∈ [−4, 4] and ψ2 ∈ [−5, 5]. In this case there are actually two global minimum
at ψ = ψexct and at ψ = −ψexct as sine is an odd function. Making a slice through this
contour plot at ψ1 = 1 would result in the two dimensional plot of ε(ψ1 = 1, ψ2 ) over ψ2
as in Fig. 3.3.
As can be seen in Fig. 3.4 for one dimension and in Fig. 3.4 for two dimensions, the
topology of a nonlinear curve fitting problem might look uncanny on a global scale. But
near a minima it actually is shaped similar to a linear curve fitting problem. This is in fact
true for any neighbourhood around a specific local point in ψ. Any differentiable function
can be approximated by a Taylor series expansion around a local point. This means that
a linear curve fitting problem can be defined, using the linearised model derived from a
first order Taylor series expansion. The objective function of this linear curve fitting prob-
lem will of course look similar to the nonlinear curve fitting problem around the point of
linearisation.
600
500
400
300
200
100
Figure 3.4: The objective function for the nonlinear curve fitting problem in two parame-
ters for y(x) = ψ1 sin(ψ2 x).
A second example that is looked at here is the exponential function
y(x) = eψx . (3.31)
The data ξ that is to be fitted was generated by evaluating y(x, ψ) for ψ = ψexct = 0.5 at
51 evenly spaced points x ∈ [−2, 3]. The evaluation of the objective function ε(ψ) for
this curve fitting problem for ψ ∈ [−1, 1] can be seen in Fig. 3.5. There is clearly one
global minimum at ψ = ψexct . As can be seen in the figure, this curve fitting problem
results in a convex objective function similar to the linear curve fitting problem in section
3.2.1. Therefore the solution of the numerical optimization of ε(ψ) will hardly depend
on variations of the initial guess ψ0 . The exponential function defined in equation (3.31)
can be expanded by adding a second parameter in the form of a linear coefficient.
y(x) = ψ1 eψ2 x (3.32)
As before, a curve fitting problem can be defined based on data points ξ. These data points
are sampled from the solution of equation (3.32) at 51 evenly space points x ∈ [−2, 3] for
ψexct = [2, 0.5]T . The objective function ε(ψ) for this example problem is visualized in
Fig. 3.6 as a contour plot for ψ1 ∈ [−2, 3] and ψ2 ∈ [−1, 1]. As can be seen, ε(ψ) again
appears to be convex, as can be expected by adding a linear coefficient to the already
convex results of the exponential function (3.31), which depends on a single parameter.
As is shown by this example, just because a problem is labelled as nonlinear does not
Figure 3.5: The objective function for the nonlinear curve fitting problem in one parameter
for y(x) = eψx .
mean that it needs to be ill behaved. Nevertheless, one has to be mindful of potential local
minima and verify the results of a nonlinear curve fitting problem accordingly.
14000
12000
10000
8000
6000
4000
2000
Figure 3.6: The objective function for the nonlinear curve fitting problem in two parame-
ters for y(x) = ψ1 eψ2 x .
3.2.2 Gradient Based Numerical Optimization

As was already discussed, in a linear curve fitting problem a function y(x, ψ), which is a
weighted sum of basis functions, is fitted to some given data points ξ.
n
y(x, ψ) = ψ1 f1 (x) + · · · + ψn fn (x) = ∑ ψi fi (x) (3.33)
i=1
The parameters that are to be determined are the constant coefficients, the weights, of the
known basis functions fi (x). The function y(x, ψ) might be nonlinear in the independent
variable x as some of the basis functions are nonlinear in x, but y(x, ψ) is linear in the
parameters ψ.
∂ y(x, ψ)
= const. (3.34)
∂ ψi x=xk
In a nonlinear curve fitting problem y(x, ψ) is a nonlinear function in ψ.

∂ y(x, ψ)
= g(ψ) (3.35)
∂ ψi x=xk
This means that it is not possible to formulate an explicit solution for ψ as it is for the
linear curve fitting problem. Different numerical solution methods can be applied, in order
to minimize the objective function ε(ψ) in an iterative manner. Starting at an initial guess
ψ0 the gradient at that position in the parameter space can be evaluated. Different methods
use this gradient information in different ways in order to derive a step Δψ towards the
optimal parametrization ψ , which is a minimizer of ε(ψ).
Gradient Descent
If y(x, ψ) is differentiable with respect to ψ, then the gradient descent or steepest descent
method can be used for the numerical minimization of ε(ψ). Starting from an initial
guess ψ0 for the parameters, an updated parameter set can be found by making a step Δψ
into the negative direction of the gradient of ε(ψ).

∂ ε(ψ)
Δψ = −α (3.36)
∂ ψ ψ=ψ0
For the objective function
ε(ψ) = [ξ − y(ψ)]T [ξ − y(ψ)] (3.37)
this results in
∂ ε(ψ) ∂
= 2[ξ − y(ψ)]T [ξ − y(ψ)] (3.38)
∂ψ ∂ψ
∂ y(ψ)
= −2[ξ − y(ψ)]T (3.39)
∂ψ
= −2[ξ − y(ψ)]T J(ψ) (3.40)
Therefore a step from ψ0 in the direction of the negative gradient is given by
Δψ = JT (ψ0 )[ξ − y(ψ0 )] (3.41)
where J(ψ0 ) is the Jacobian of y(ψ) evaluated at ψ0

⎡ ⎤
∂ y(x,ψ) ∂ y(x,ψ) ∂ y(x,ψ)
∂ ψ1 x=x ,ψ=ψ ∂ ψ2 x=x ,ψ=ψ ... ∂ ψn x=x ,ψ=ψ ⎥
⎢ 1 1 1
⎢ ∂ y(x,ψ)
0
∂ y(x,ψ)
0
∂ y(x,ψ)
0
⎥
⎢ ... ⎥
⎢ ∂ ψ1 x=x ,ψ=ψ ∂ ψ2 x=x ,ψ=ψ
∂ ψn x=x ,ψ=ψ ⎥
J(ψ0 ) = ⎢ 2 0 2 0 2 0 ⎥ (3.42)
⎢ .. .. ... .. ⎥
⎢ . . . ⎥
⎣ ∂ y(x,ψ) ∂ y(x,ψ) ∂ y(x,ψ) ⎦
∂ ψ1 ∂ ψ2 ... ∂ ψn
x=xN ,ψ=ψ0 x=xN ,ψ=ψ0 x=xN ,ψ=ψ0
and the next point in the parameter space is
ψ1 = ψ0 + αΔψ0 . (3.43)
The gradient descent method is easy to implement but has the disadvantage that it of-
fers little control of the convergence performance beside scaling the step size with the
coefficient α [23]. Selecting a value that is too small might lead to poor convergence
due to making little progress in the parameter space. Selecting a values that is too high
might lead to poor convergence due to overshooting of the minimum or the trough of the
ellipsoidal valley.
Gauss-Newton
The Gauss-Newton method is a nonlinear curve fitting algorithm, which works by ap-
proximating the nonlinear design function y(x, ψ) by a first order Taylor series expansion
around the current point ψ0 in the parameter space. In each iteration a step Δψ is com-
puted, which – according to this linearised model – minimizes the sum of squares opti-
mality criterion.
For any inital guess ψ0 there will be a discrepancy between ξ and y(ψ0 ).
ξ = y(ψ0 ) + Δy (3.44)
The Gauss-Newton methods tries to find a step Δψ based on the first order Taylor series
expansion of y(ψ) around ψ0 , which minimizes this discrepancy.
y(ψ0 + Δψ) ≈ y(ψ0 ) + J(ψ0 )Δψ (3.45)

ξ = y(ψ0 ) + J(ψ0 )Δψ (3.46)
where J(ψ0 ) is the Jacobian of y(ψ) evaluated at ψ0 . Equation (3.46) is a system of N

linear equations in the n unknowns Δψ = [Δψ1 , . . . , Δψn ]. If the columns of the Jacobian
are linearly independent then the unique solution is
Δψ = J+ (ψ0 )(ξ − y(ψ0 )). (3.47)
Which is of course the minimum of the linear curve fitting problem, that arises when using
the linearised model (3.45):
ε(Δψ) = ξ − y(ψ0 ) − J(ψ0 )Δψ22 . (3.48)
The next point in the parameter space can be computed by
ψ1 = ψ0 + αΔψ. (3.49)
The relaxation factor α is typically set to be α < 1. This is due to the fact that the
linearisation of the nonlinear function y(x, ψ) might only be a good approximation in a
close proximity around ψ0 . Making too large steps αΔψ might lead to poor convergence
due to overshooting past the minimum of ε(ψ).
Levenberg-Marquardt
The Levenberg-Marquardt algorithm is based on the idea of K. Levenberg in 1944 and

later D. Marquardt in 1963 to use a damped Gauss Newton method for the numerical
solution of a nonlinear curve fitting problems [24, 25].
In the Gauss-Newton method the step Δψ is computed from the system of linear equations
JT JΔψ = JT (ξ − y(ψ0 )) (3.50)
which is equation (3.46) multiplied with JT from the left. In the Levenberg-Marquardt
algorithm this equation is modified by adding a damping term λ I to the matrix product on
the left hand side:
(JT J + λ I)Δψ = JT (ξ − y(ψ0 )) (3.51)
where λ is a scalar and I is a unity matrix of appropriate size. This equation can be derived
by adding a penalty term for the step Δψ in the parameter space to the cost function (3.48)
of the Gauss-Newton method:
ε(Δψ) = ξ − y(ψ0 ) − J(ψ0 )Δψ22 + λ Δψ22 . (3.52)
Setting the derivative of this cost function with respect to Δψ to zero results in equation
(3.51). Depending on the coefficient λ this leads to a different convergence behaviour.
For large values of λ the algorithm makes small steps in the direction of the steepest
descent
1
Δψ ≈ JT (ψ0 )[ξ − y(ψ0 )]. (3.53)
λ
For small values of λ the algorithm makes steps according to the Gauss-Newton method
Δψ ≈ J+ (ψ0 )[ξ − y(ψ0 )]. (3.54)
The damping coefficient λ does not have fixed values and an important aspect of the
Levenberg-Marquardt algorithm is the way in which it is manipulated during the iteration
procedure. Usually λ is initialized as a larger value to ensure a quick descent from the
initial guess ε(ψ0 ) along the direction of the gradient. Smaller values of λ ensure a
good convergence in the region near a minima, where the Gauss-Newton method shows
quadratic convergence if the objective function value at the minimum is approximately
zero: ε(ψ ) ≈ 0 [26].
Numerical Approximation of the Jacobian
In the curve fitting problem a function y(x, ψ) is to be fitted to the data points ξ given
at their locations x = [x1 , . . . , xN ]. Therefore the evaluation of y(x, ψ) at these location is
described as the vector valued function y(ψ). The Jacobian J, which is needed in some
algorithms for the numerical optimization of the sum of squares goodness-of-fit criterion
ε(ψ), contains the derivatives of y(ψ) with respect to each of the n parameters ψ.

∂ y(ψ) ∂ y(ψ) ∂ y(ψ)
J(ψ0 ) = ∂ ψ1 ∂ ψ2 . . . ∂ ψn (3.55)
ψ=ψ0 ψ=ψ0 ψ=ψ0
For the system identification problem, y(ψ) – the output of the system model – is not
known analytically. It can merely be evaluated for any selected point in the parameter
space ψ by numerical integration of the initial value problem. Therefore, in order to
apply a gradient based optimization method, a numerical approximation of the Jacobian
J needs to be found.
This is done by applying a finite difference formula and by varying one parameter at a
time for each column of the Jacobian J.

∂ y(ψ) y(ψ0 + δi ) − y(ψ0 )
≈ . . . forward (3.56)
∂ ψi ψ=ψ0 Δψi
y(ψ0 + δi ) − y(ψ0 − δi )
≈ . . . central (3.57)
2Δψi
y(ψ0 ) − y(ψ0 − δi )
≈ . . . backward (3.58)
Δψi
with
δi = Δψi ei (3.59)
where ei is a n dimensional unity vector pointing in the direction of ψi in the parameter

space.
⎧
T ⎨e = 1 if i = j
j
ei = e1 . . . en = (e j ) with ej = (3.60)
⎩e = 0 if i = j
j
The central finite difference formula needs 2n evaluations of y(ψ) for each approximation
of the Jacobian J, where the forward and backward formula suffice with (n + 1) evalua-
tions.
Regardless of whether a one sided or the central formula is used, the numerical approxi-
mation of the Jacobian using finite differences is computationally expensive [27]. As an
alternative the Broyden rank-1 update formula can be used to estimate the Jacobian for a
new parameter based on the known Jacobian for a nearby point in the parameter space.
J(ψ + Δψ) = F(J(ψ), Δψ) (3.61)
The disadvantage of this method is, that the successive updates of the Jacobian via Broy-
den’s formula lead to divergence from the real value. Therefore is it advisable to at least
after some steps perform a more precise approximation via a finite difference formula
[23].
3.2.3 Numerical Optimization Without Gradient Information

Finding the minimum of the scalar valued optimality criterion ε(ψ) can also be performed
without gradient information. In section 3.2.1 a region of the parameter space was inves-
tigated simply by plotting the objective function values for the purpose of visualization.
If this evaluated region of the parameter space includes a point ψ which leads to a ε(ψ)
which is smaller than for all surrounding points, then an approximate solution for one
minimum of the objective function has already been found. A subsequent evaluation of
ε(ψ) near that minima could be used to further improve on the accuracy of the solution.
Of course, a visual inspection of the objective function in order to find a minimum is
both computationally expensive and rather difficult to do on a two dimensional graphical
representation of ε(ψ), as soon as the dimension of the parameter space ψ is larger than
two.
One algorithm that improves on this idea of simply comparing the evaluated objective
function values for different points in the parameter space is the Nelder-Mead simplex
method.
The Nelder-Mead Simplex Method
Similar to gradient based numerical optimization methods, the Nelder-Mead simplex

method is also an iterative numerical optimization method that starts at an initial point
in the parameter space. Instead of utilizing gradient information in order to iteratively
find a way towards a minimum, the Nelder-Mead algorithm purely relies in function eval-
uations. For a n dimensional parameter space, n + 1 function evaluations around the initial
guess ψ0 lead to the vertices of a so called simplex [28]. The algorithm optimizes the ob-
jective function by trying out new points and comparing their function value to the ones
of the current simplex. By replacing the point with the highest value a new simplex is
formed and the process repeated.
The MATLAB® function fminsearch() is an implementation of the Simplex method.
The advantage of using this algorithm compared to gradient based methods lies in the
simple application as no gradient information has to be provided.
This MATLAB® function was tested on different problems during the work on this the-
sis, but in comparison to gradient based methods like the Gauss-Newton and Levenberg-
Marquardt algorithms it demonstrated worse performance, especially on higher dimen-
sional problems. This might be due to the fact that the simplex method can show poor
convergence in the neighbourhood of a minima when small convergence tolerances are
demanded as is mentioned in [28]. A discussion about the convergence properties of the
Nelder Mead simplex method in low dimensions can be found in [29].
3.2.4 Improving the Performance of the Numerical Optimization

Once a numerical solver for the IVP and the numerical optimization method has been
selected, the successful solution of the nonlinear curve fitting problem of determining
the parametrization of a given model structure based on some given identification data
mainly comes down to finding a close enough initial guess ψ0 for the set of parameters
that are to be identified. Other factors that influence the results are the selected settings
and convergence criteria for the numerical solution methods. However, the performance
of the numerical optimization can can be further improved by implementing the following
procedures, where applicable.
Variable Projection Method
The variable projection methods was introduced by [30] for the solution of nonlinear least
square problems whose variables separate. This means that the design function y(ψ, x)
of the curve fitting problem can be described as a linear combination of nonlinear basis
functions fi (β, x):
m
y(ψ, x) = y(α, β, x) = ∑ αi fi (β, x). (3.62)
i=1
In this case the sum of squares a can be denoted as

2
n m
ε(α, β) = ∑ ξ (x j ) − ∑ αi fi (β, x j ) (3.63)
j=1 i=1
The problem of minimizing this functional is also called a separable least squares prob-
lem. It can be solved in two steps by applying the variable projection method as shown in
[31]:
As the basis functions fi (β, x) only depend on the set of parameters β, for any choice
of β a matrix V(β) similar to a Vandermonde matrix containing the evaluated nonlinear
basis functions can be defined:
⎡ ⎤
f1 (β, x1 ) . . . fm (β, x1 )
⎢ .. .. ⎥
V(β) = ⎢⎣ .
...
. ⎥.
⎦ (3.64)
f1 (β, xn ) . . . fm (β, xn )
This means that the objective function ε(α, β) of the separable least squares problem can
be written as
ε(α, β) = ξ − V(β)α22 . (3.65)
For any β the linear coefficients α can be computed as the solution of a linear least squares
problem
α(β) = V+ (β)ξ. (3.66)
Substituting equation (3.66) into (3.65) results in

2
ε(β) = ξ − V(β)V+ (β)ξ 2 (3.67)
which is a function solely dependent on β. Therefore the two steps of solving the separa-
ble nonlinear least squares problem are:
1.

β = argmin ε(α, β)α=V+ (β)ξ (3.68)
β
, which is a nonlinear optimization problem for the determination of β . Subse-

quently, the optimal weighting α of the nonlinear basis functions fi (β, x) can be
evaluated as
2.
α = V+ (β )ξ. (3.69)
The advantage of the variable projection method is that number or parameters, that need
to be estimated in the nonlinear optimization, is reduced. This leads to better computa-
tional efficiency and a greater likelihood of finding a global minimum, rather than a local
one [30, 32].
The variable projection method has found application various disciplines, see for example
[31] for an overview. It was also applied in system identification problems. For example
in [33] it is applied in the identification of a Wiener model, which consists of a discrete
LTI system followed by a static non-linearity. In [15], a nonlinear system is identified as a
composite local linear state space model, a composition of discrete LTI systems, with the
help of the variable projection method. The application in [9] for the system identifica-
tion of a continuous LTI system utilizing the Global Least Squares solution method was
already mentioned in section 1.4.
Unfortunately, the application of the variable projection methods on the system identifica-
tion of nonlinear state space models appears to be severely limited by the fact that without
access to an analytic solution the state trajectory of a nonlinear dynamic system has to be
approximated by a numerical solver. This means that in most cases there is no sum of
basis functions that can be superimposed, no projection matrix that can be formulated
and no separation of variables into ones with linear and nonlinear influence on the basis
functions. It is of course possible to design a nonlinear dynamic system, whose output is
the superposition of two independent dynamic processes. In such a case it would be ad-
vantageous to apply the variable projection method on the system identification problem.
However, for the problems that are studied in chapter 4 this is not the case.
Normalization of the Parameter Space
In some problems the parameter space occupied by ψ is badly scaled or skewed in a

way that for example one parameter ψi lies on a completely different scale of magnitude
compared to another parameter ψ j .
ψi
= 10n , n >> 1 (3.70)
ψj
If the general range or order of magnitude of some parameters values is known then a
simple linear transformation can be applied in order to put them on a similar scale. For
example when ψ̂ i is the expected value of ψi , then a parameter transformation which
projects ψ̂ i → 1 but keeps zero at zero could be used:

0 ψ̂ i → 0 1 (3.71)
When ψmin are the minimum values of the parameters ψ and ψmax are the maximum
values, then each parameter can be projected onto the interval [0, 1] by
ψ − ψmin
φ(ψ) = . (3.72)
ψmax − ψmin
The above case where ψi = 0 remains at ψi = 0 can of course be achieved by setting ψmin
to be the zero vector.
A normalization or rescaling of the parameter space can help with the convergence of nu-
merical optimizations methods. One convergence criteria for such an algorithm is often
defined to be a lower limit of the acceptable step size Δψ. If the exact value of one of
the parameters is of smaller or equal order compared to the smallest accepted step size,
then that parameter might be identified less precisely. The extent of this numerical dis-
cretization error of course also depends on the influence that variations of this parameter
have on the system output and on the noise disturbance in the identification data.
One way of counteracting this is to set the mentioned convergence criteria to smaller
values. This can be imagined as using a scope to look at the topology of the objective
function in the neighbourhood of the minimum in more detail. But a rescaling of the
one problematic parameter takes on the problem in a slightly different way. Instead of
investing more computational resources in order to essentially resolve the whole param-
eter space on a smaller scale, the parameter space is stretched in that one problematic
dimension to make convergence easier.
3.3 Implementation of a System Identification Algorithm

in MATLAB ®
In the system identification problem the parameters of a system model are to be deter-
mined such that the behaviour of the model reflects the observed behaviour of the system.
This problem can be solved numerically by utilizing numerical integration and numerical
optimization methods.
The system model is given by the state-space and observer equations, e.g. for a single
input single output system:
ẋ(t) = f (x(t), u(t),t, p) (3.73)

y(t) = g(x(t), u(t),t, p) (3.74)
or in the case that it is given by higher order differential equations, it needs to be trans-
formed into a state-space representation.
The given n observational data points
T
ξ = ξ0 , ξ1 , . . . , ξn−1 , (3.75)
which correspond to a given input sequence u(t), were measured at the given points in
time T
t = t0 ,t2 , . . . ,tn−1 . (3.76)
This means that for any set of system parameters p and initial conditions x0 = x(t0 ) a
numerical solution method can be applied in order to compute the response of the system
model to the given input sequence u(t) at these points in time t:
T
y(ψ) = y(ψ,t0 ), y(ψ,t1 ), . . . , y(ψ,tn−1 ) , (3.77)
where ψ contains the model parameters p and the initial conditions x0 that are to be de-
termined in the system identification. If all the initial conditions are assumed to be known,
then ψ = p.
The goal here is to define a function in MATLAB® , which – provided with the input
of any value for the set of parameters ψ that are to be identified – returns an estimate
of the system output y(ψ) for this parametrization. Any ODE-solver can be applied
for the solution of the initial value problem for equation (3.73). For general problems
the MATLAB® function ode45() can be applied. For the solution of stiff systems
ode23s() might be more advantageous. The observer equation (3.74) can then be used
to extract the system output y(ψ) from the state trajectories [x1 (ψ), x2 (ψ) ...]. Once the
system output y(ψ) is defined as a function in the parameters ψ that are to be identified,
it is also possible to implement a function which approximates the jacobian J(ψ) of y(ψ)
by the use of finite difference formulae as described in section 3.2.2. This is necessary in
the case where a gradient based numerical optimization technique like the Gauss-Newton
or Levenberg-Marquardt algorithm is applied for the solution of the nonlinear curve fitting
problem. These algorithms usually have the sum of squares optimality criterion imple-
mented in their code and require the identification data ξ, the function that is to be fit to
that data y(ψ), the gradient information in the form of J(ψ) as well as an initial guess
for the parameter set ψ0 as input. The Nelder-Mead Simplex method implemented in the
MATLAB® function fminsearch() only requires the objective function ε(ψ) and an
initial guess ψ0 as an input. In this case the approximation of the gradient is not required
but the objective function needs to be implemented in MATLAB® utilizing the previously
defined function which returns y(ψ):
ε(ψ) = ξ − y(ψ)22 . (3.78)
There are different numerical solution methods for ODEs as well as different numerical
optimization methods for nonlinear curve fitting problems, some of which were men-
tioned previously. Any reader of this thesis should feel encouraged to combine any ODE
solver and any numerical optimization technique of their liking into an algorithm for the
system identification of nonlinear dynamic systems.
To summarize, here is a short overview of the implemented system identification proce-
dure:
1. Define the identification data [ξ, t] as well as the corresponding system input u(t).
2. Define the system model, whose parameters p are to be estimated:
ẋ(t) = f (x(t), u(t),t, p)

y(t) = g(x(t), u(t),t, p)
3. Determine whether the initial conditions of the experiment x0 = x(t0 ) are known
or to be estimated
(a) if x0 is known: ψ = p
(b) if x0 is to be estimated: ψ = [pT , xT0 ]T
4. Select or implement a suitable ODE solver.
5. Implement a function, e.g. estimateModelOutput(ψ), which utilizes the cho-

sen ODE solver and returns the estimated model output y(ψ) at the points in time
t for the given system input u(t) and any given parameter set ψ.
6. Select or implement a suitable numerical optimization algorithm.
(a) In the case of a gradient based numerical optimization method:

Implement a function, which approximates the jacobian J(ψ) of y(ψ) by uti-
lizing estimateModelOutput(ψ).
(b) In the case of a numerical optimization method that does not rely on gra-
dient information: Implement a function which evaluates ε(ψ) by utilizing
estimateModelOutput(ψ).
7. Select an initial guess ψ0 for the parameters that are to be identified.
8. Perform the numerical optimization by providing the selected algorithm with the
necessary inputs: ε(ψ) and ψ0 or ξ, y(ψ) and ψ0 .
This should result in the identified parameter set ψ .
9. Validate the results.

Chapter 4
Case Studies
In this chapter the presented system identification algorithm is applied to a number of

example problems. These include an object in free fall with drag due to air, a nonlinear
mass and spring system and a nonlinear dynamic friction model. In each of these case
studies the model representing the actual system is described and the experimental design
in order to extract information about the object of interest is investigated. The system
identification is performed on the basis of synthetic measurement data. This means that
the exact parametrization of the system model is decided upon and the measurement data
is generated as the simulation output of the exactly parametrized system. Different aspects
and problems that arise during the system identification of nonlinear dynamics systems
are investigated and illustrated with respect to the mentioned example problems. These
include the aforementioned experimental design, the selection of a fitting initial guess for
the set of parameters that are to identified and the utilization of multiple data sets in order
to achieve an accurate estimate of those parameters.
4.1 Free Fall with Drag due to Air
4.1.1 The Riccati Equation

The Riccati equation is a general nonlinear differential equation of first order. It holds a
special place in the theory of differential equations because of its relation to the general
linear differential equation of second order [34]. The general form of the Riccati equation
can be defined as
ẋ + q(t)x + r(t)x2 = p(t). (4.1)
A special case of equation (4.1) can be obtained by setting the coefficient functions q(t)
and r(t) as well as the disturbing function p(t) to constant values. This results in the
simplified description of a body in vertical free fall. The state variable was changed from
x to v, as it in this case describes the velocity of a body of mass m subject to constant
38
CHAPTER 4. CASE STUDIES 39
gravitational acceleration g and a friction force proportional to the square of the velocity
v2 .
c
v̇ − v2 = −g (4.2)
m
The general solution to this equation is known and can for example be found in [34]:

ert − ke−rt
v(t) = vlim (4.3)
ert + ke−rt
with the terminal velocity vlim as the velocity at the stationary state where the gravitational
and the frictional force cancel each other out:

mg
vlim = − and (4.4)
c

cg
r= . (4.5)
m
The special solution for any initial velocity v(t = 0) = v0 is
v0 + vlim tanh rt
v(t) = . (4.6)
1 + vvlim
0
tanh rt
Integrating this function will results in the analytic solution for the position s(t).
t
s(t) = s0 + v(t)dt (4.7)
0

vlim 2 gt v0 gt
s(t) = s0 − cosh + sinh (4.8)
g vlim vlim vlim
with s0 = s(t = 0) (4.9)
These nonlinear equations (4.6) and (4.8) as the solution to the problem of the body in
free fall show that even such a relatively simple nonlinear dynamic system might present
an interesting object of study for the task of system identification.
4.1.2 System Modeling

The same differential equation as (4.2) that resulted as a special case of the Riccati equa-
tion can also be obtained by applying Newton’s second law of motion. The body of mass
m is subject to a constant downward accelerating gravitational force Fg = mg. To ac-
count for a correct behaviour for all initial conditions the friction force has to be defined
in a slightly different way. The resistive friction force due to air drag that is defined as
Ff = cv2 as in (4.2) is changed to Ff = −cv2 sign(v) to account for a possible positive
initial velocity v0 . This results in the following equation of motion.
ms̈ = ∑ Fi = Ff − Fg (4.10)
ms̈ = −cṡ2 sign(ṡ) − mg (4.11)
As the position of the mass s(t) isn’t explicitly present in the equation, a change of vari-
ables can be performed. This results in an equation in the velocity, similar to equation
(4.2).
ṡ = v (4.12)
s̈ = v̇ (4.13)
mv̇ = −cv2 sign(v) − mg (4.14)
Whether the equation of motion (4.10) depending on the position s(t) or equation (4.12)
depending on the velocity v(t) is used for the purpose of system identification, depends
on what measurement data is available. The transformation of (4.10) into state-space
representation with x1 (t) as the position and x2 (t) as the velocity can be used in any case:
ẋ1 = x2 (4.15)
ẋ2 = −cx22 sign(x2 ) − mg. (4.16)
These two first order differential equations are combined into matrix form:

ẋ1 (t) x2 (t)
ẋ(t) = = . (4.17)
ẋ2 (t) −cx22 sign(x2 ) − mg
The observer equation is

c11 0 x1 (t)
y(t) = Cx(t) = (4.18)
0 c22 x2 (t)
with c11 = 1 and c22 = 0 for the output y(t) = x1 (t) or c11 = 0 and c22 = 1 for the output
y(t) = x2 (t).
4.1.3 Experimental Design

In general industrial applications the determination of drag coefficients is usually based
on measurements of the drag force Ff in wind tunnel experiments. Nevertheless, this
simple system of the body in free fall presents a good starting point for the application of
system identification techniques. For that purpose the experimental design will consist of
dropping an object of mass m from a specific height h and using the position data as the
basis for the parameter estimation. It is assumed that both the initial position s0 = h and
the initial velocity v0 as well as the mass m of the object and the gravitational acceleration
g are known. The only parameter that is to be determined is the drag coefficient c. It is
also assumed that the object is falling vertically straight down and doesn’t tumble during
the fall so that the drag coefficient remains constant over time.
One way of tackling this problem would be to find a model that can estimate the time it
takes for the mass to fall down the height h. This would mean that we are looking for the
solution to the boundary value problem
s(t0 ) = h t0 = 0 (4.19)
s(t f ) = 0 t f . . . extracted from the measurement (4.20)
The disadvantage of this is that we are only taking one measurement, the final time t f into
consideration.
Another approach would be to design the experiment in a way that the object reaches the
stationary state of terminal velocity vlim . If the terminal velocity can be estimated, then it
is trivial to estimate the drag coefficient from that using equation (4.4).
However, in the following section the system identification approach of minimizing the
output error of the model in relation to the observed position measurement as described
in chapter 2 will be applied. As such we want to minimize the functional
ε(ξ, ψ) = ξ − y(ψ)22 . (4.21)
The position measurements ξ (ti ) observed at the discrete points in time ti are contained
in the vector ξ. The parameter ψ to be estimated is the drag coefficient c. The model out-
put y(ψ,t) can be computed in two ways. For any experimental data ξ, where the initial
velocity v0 ≤ 0, the analytic solution (4.8) can be used. Alternatively the solution to the
differential equation in state-space representation (4.17) can be computed by numerical
integration of the initial value problem. For other problems involving nonlinear differen-
tial equations there might not be any analytic solution available. Therefore the path of
applying a numerical solution method will also be pursued in this case. The analytic solu-
tion might still come in handy in case it is necessary to confirm the accuracy of numerical
approximation of the solution of the initial value problem.
Generating Measurement Data
The specific experiment designed to extract information about the drag coefficient c =
ψ of an object of mass m = 10 kg is a drop from the height h = 30 m with an initial
velocity v0 = 0 and subject to the gravitational constant of acceleration g = 9.81 m/s2 .
The observed measurement is the position ξ (t) ≈ s(t). The exact position trajectory was
generated from the analytic solution (4.8) for the exact parameter ψexct = 0.5 kg/m. This
exact system output yexct (t) is sampled every ts = 0.1 s over the time interval [t0 ,t f ] = [0, 3]
resulting in the vector yexct = y(ψexct ). Gaussian measurement noise δ that is added
to this measurement is generated by sampling pseudorandom values from the normal
distribution N (μ, σ ). These disturbances are selected to be rather large with standard
deviation of σ = 1 and zero mean μ = 0. Any experiment i that is performed this way
is assumed to result in an observed system output based on the same exact solution yexct
disturbed by a different sequence of independent random variables δi sampled from the
same normal distribution.
ξi = yexct + δi (4.22)
4.1.4 System Identification Based on One Data Set

The first execution of the described experimental procedure is labeled as experiment A
and the corresponding set of observations as
ξA = yexct + δA . (4.23)
Based on equation (2.7) the cost function of the system identification problem based on
experiment A can be defined as
εA (ψ) = ε(ξA , ψ) (4.24)

εA (ψ) = ξA − y(ψ)22 . (4.25)
For problems like this, which involve only one parameter ψ that is to be estimated, it is
possible to visualize the objective function ε(ψ) over a selected region of the parameter
space. Figure 4.1 shows the cost function εA (ξA , ψ) based on experiment A as well as the
theoretical cost function ε(yexct , ψ) based on a measurement without any noise distur-
bances. Both are evaluated for ψ ∈ [0, 1]. As can be seen, for this problem ε(ψ) appears
to be a convex function with one global minimum. The location of this minima is found
by numerical optimization of the objective function.
ψA = argmin εA (ψ) = argmin ξA − y(ψ)22 (4.26)

ψ ψ
Table 4.1 shows the results of the numerical optimization of the above cost function using
the Simplex and the Gauss-Newton method. y(ψ) was computed from the state space
representation (4.17) by numerical integration using the Runge-Kutta method. For the
solution using the Gauss-Newton method the derivative of y(ψ) with respect to ψ was
computed as a numerical approximation using a finite difference formula. The initial star-
Figure 4.1: Local minima of the objective function for the free fall experiment A.
ing point was set at ψ0 = 0. Because of the mentioned convex form of the cost function,
convergence to the minimum from that initial point in the parameter space isn’t a prob-
lem. The deviation of these two results is not of special relevance here. It is due to the
different path in the parameter space that the applied numerical methods take during their
iterative solution procedure and due to the different convergence criteria that stop the it-
eration. The important result is that both methods lead to approximately the same result
which deviates from the exact parameter ψexct = 0.5 kg/m. This error in the identified
parameter is not due to wrong convergence or getting trapped in a wrong local minima.
The global minimum of the objective function ε(ξA , ψ) actually lies at the identified lo-
cation ψA . The error in the identified parameter with respect to the exact one is due to the
disturbance induced variation of the cost function. This is also visible in Fig. 4.1. The
minima of each curve, which are marked by a circle, are not the same. The large deviation
is to be expected for the large noise level that was purposefully selected to achieve this
clear distinction.
Simplex Gauss-Newton exact

ψA [kg/m] 0.450063 0.449959 0.5
Table 4.1: Results of the system identification based on the data from the free fall experi-
ment A.
Fig. 4.2 shows the exact system output yexct (t), the noise perturbed observations ξA
and the estimated output of the model y(ψA ,t). As the exact parametrization is known, it
is also possible to compare the respective velocity trajectories x2 (ψexct ,t) and x2 (ψA ,t),
Figure 4.2: Position data and estimate of the object in free fall with drag due to air resis-
tance.
which are shown in Fig. 4.3. The portrayed performance of the model seems to be the best
that is possible, based in the data that is given. Ways of improving the model uncertainty
would be to improve the measurement accuracy or to reduce the sample time ts or to
increase the measurement interval [t0 ,t f ] by increasing t f in the hope of reducing the
signal to noise ratio in order to reduce the disturbance induced variation of the identified
parameters [17]. This implies changing the experimental setup in a way to be able to make
better or more observations. Another way of improving the model uncertainty without
changing the experimental setup is explored in the next section.
Figure 4.3: Velocity estimate of the object in free fall with drag due to air resistance.
4.1.5 System Identification Based on Two Data Sets

A second experiment B with identical setup to experiment A is performed. This results in
the measurement data
ξA = yexct + δA (4.27)
ξB = yexct + δB (4.28)
to be available for the estimation of the drag coefficient ψ. The cost function
εB (ψ) = ξB − y(ψ)22 with (4.29)
ψB = argmin εB (ψ) (4.30)

ψ
can be defined for the measurement ξB like equation (4.25). In a similar way, a cost
function for the simultaneous vertical distance least squares fit to both data sets ξA and ξB
can be defined by stacking the measurement vectors on top of each other.
εA,B (ψ) = ε(ξA , ξB , ψ) (4.31)

2
ξ y(ψ)
A
εA,B (ψ) = − (4.32)
ξB y(ψ) 2
ψA,B = argmin εA,B (ψ) (4.33)
ψ
The question that is to be investigated is what kind of improvement can be achieved by

optimizing the fit to both the data sets of experiment A and B. Necessarily one of the
measurements will be better than the other as they won’t be identical. If we wouldn’t
have knowledge about the exact parameter ψexct there would be no way of telling which
one it is. For the one dimensional parameter space the only thing that can be said with
certainty is that either
ψexct < ψ1 < ψ2 (4.34)
or
ψ1 < ψ2 < ψexct (4.35)
or
ψ1 < ψexct < ψ2 (4.36)
where ψ1 is the smaller and ψ2 is the larger value of ψA and ψB . As we are necessarily
combining a better and a worse measurement, it can be expected that ψA,B is at least
a better parametrization than the parameter identified based on the worse measurement.
The question that arises from this is whether ψA,B is always in between ψA and ψB . If
true, this would mean that ψA,B is a kind of average result of experiment A and B and that
only in the case of (4.36) could ψA,B possibly be better than both ψA and ψB . However,
without knowledge about the exact parameter ψexct , which wouldn’t be available in a real
system identification problem, the only distinction that can be made between ψA and ψB
is that one of them is larger and the other one smaller. In this case ψA,B would still be the
best guess of a parametrization that depicts the behaviour of the system most accurately.
As ξA and ξB contain measurements samples drawn at the same point in time in reference
to the time interval [t0 ,t f ] of the experiment, another data set can be generated by taking
the mean of those two measurement sets and defining a fourth cost function accordingly.
ξA + ξB
ξAB = (4.37)
2
εAB (ψ) = ε(ξAB , ψ) (4.38)
ψAB = argmin εAB (ψ) (4.39)
ψ
As was presented in Fig. 4.1 for the first data set ξA , Fig. 4.4 shows the cost functions
εA , εB , εA,B and εAB for the region ψ ∈ [0, 1] of the parameter space. For visual clarity the
vertical axis is set to a logarithmic scale. For the system identification procedure, equation
(4.17) was again numerically integrated as described in section 4.1.4. The values of the
parameters that were identified using the Gauss-Newton method can be seen in table 4.2.
The resulting estimated state trajectories based on the identified parameters can be seen
in Fig. 4.5 and 4.6.
ψA ψB ψA,B ψAB ψexct

[kg/m] 0.449959 0.521219 0.484568 0.484568 0.5
Table 4.2: Results of the system identification based on the data from the free fall experi-
ment A and B.
As it turns out ψA,B is identical to ψAB with respect to not only the six digits displayed
here but up to the eleventh digit. To analyse this result, we will look at one step of the
Gauss-Newton minimization of the cost function εAB (ψ) that was defined in (4.32).

ξA − y(ψi ) J(ψi )
= Δψi (4.40)
ξB − y(ψi ) J(ψi )
Solving this system of equations is the same as simultaneously solving the two systems
of equations
ξA − y(ψi ) = J(ψi )Δψi (4.41)

ξB − y(ψi ) = J(ψi )Δψi (4.42)
which can be added to gain
ξA + ξB − 2y(ψi ) = 2J(ψi )Δψi (4.43)

ξA + ξB
− y(ψi ) = J(ψi )Δψi (4.44)
2
ξAB − y(ψi ) = J(ψi )Δψi (4.45)
which is a step in the Gauss-Newton minimization of the cost function εAB (ψ). Therefore,
each step in the minimization of εA,B (ψ) is the same as in the minimization of εAB . This
implies that both iterative solution procedures should lead to the same results, as has been
observed. Deviations from this identical result might occur due to different activation of
the convergence criteria. While the minima of εA,B (ψ) and εAB (ψ) lie at the same place
in the parameter space ψ, the value of εA,B (ψ) = εAB (ψ) as can be seen in Fig. 4.4.
Therefore, a limit set on the minimum allowed change of the objective function value
might trigger the stop of the iterative solution method for one but not the other.
To summarize, this means that fitting a model to multiple measurement data sets which
are based on the same experimental setup is the same as fitting the model to the mean of
those data sets.
Figure 4.4: Local minima of the objective functions for the free fall experiment A and B.
Figure 4.5: Estimated position trajectories for the free fall experiment based in the data
sets A and B.
Figure 4.6: Estimated velocity trajectories for the free fall experiment based in the data
sets A and B.
4.1.6 Statistical Analysis

To analyse the influence of the measurement noise on the identified parameter ψ, a Monte-
Carlo simulation is performed. For this purpose the system identification process of es-
timating the drag coefficient based on one data set corresponding to an experiment A as
in section 4.1.4 and based on two data sets corresponding to two experiments A and B as
in section 4.1.5 are performed n = 104 times. Different noise disturbances were gener-
ated for each run by sampling the normal distribution N (0, 1) as was done before. The
goal of the Monte-Carlo simulation is be able to analyse the resulting distribution of the
identified parameters. To check if the gaussian noise in the identification data results in
normally distributed parameters the Kolmogorov-Smirnov test can be used. In this test the
largest vertical deviation of the empirical cumulative distribution function of a data set
from the normal cumulative distribution function is used as a metric in order to evaluate
the probability that this data set stems from the normal distribution. In order to be able to
apply the Kolmogorov-Smirnov test, the results of the Monte-Carlo simulations have to
be normalized as the Kolmogorov-Smirnov test tests for the null hypothesis that a given
data sets stems from the standard normal distribution. This is performed via the following
formula, using the sample mean ψ̄ i and sample standard deviation sψi for each set of n
identified parameters ψA , ψB , ψA,B and ψAB .:
x−x
x̃ = . (4.46)
sx
Table 4.3 shows the resulting p-values pKS (ψi ) in addition to the sample mean ψ̄ i and
sample standard deviation sψi corresponding each set of identified parameters ψA , ψB ,
ψA,B and ψAB . In particular the results for ψA and ψA,B can be seen in Fig. 4.7, where
they are visualized in the form of empirical and normal cumulative distribution functions
(CDF). In addition to that Fig. 4.8 and 4.9 show the comparison of a histogram and the
normal distribution computed with the sample mean and sample standard deviation for
each of these two identified parameter sets. The important insight that is to be taken from
this is that gaussian measurement noise apparently results in a normal distribution of the
identified parameters due to disturbance induced variation. This is also confirmed by the
Kolmogorov-Smirnov tests for a significance level of α = 0.05 < pKS (ψi ).
As is exhaustively demonstrated – at least for this example problem – normally distributed
disturbances lead to normally distributed identified parameters. One thing to note is that
the results obtained here are based on the numerical optimization of the objective function.
This means that the error of the identified parameter ψ compared to the exact one ψexct is
not only due to the disturbance induced variation of the objective function ε(ψ) but also
due to the numerical accuracy of the solution method. However, as long as the disturbance
induced variation is large compared to the numerical accuracy, the identified parameters
should remain to be approximately normally distributed.
ψA ψB ψA,B ψAB
ψ̄ i 0.500277 0.500331 0.500041 0.500040
sψi 0.0258864 0.0262175 0.018495 0.0184945
pKS 0.256843 0.142756 0.424425 0.42518
Table 4.3: Results of the Monte-Carlo simulation of the system identification of the free
fall experiment based on one and two data sets.
In practice, where only limited identification data is available, a so called bootstrap-

ping method can be applied. Synthetic measurements of N data points can be generated
from a single set of measurements of N data points by random drawing with replacement.
These synthetic data sets can then be used as basis for a Monte-Carlo simulation in order
to get an insight into the statistical distribution of the to be estimated parameters [18]. An
issue that may arise is that not every data point has the same influence on the estimated
parameters as the excitation of a controlled dynamic system changes over time through
the applied input.
Figure 4.7: Cumulative distribution function of the results of the Monte-Carlo simulation
for the free fall experiment based on one and two data sets A and B.
Figure 4.8: Histogram of the results of the Monte-Carlo simulation for the free fall exper-
iment based on one data set A.
Figure 4.9: Histogram of the results of the Monte-Carlo simulation for the free fall exper-
iment based on two data sets A and B.
4.1.7 System Identification Based on More Than Two Data Sets

In the previous section the effect of combining the information of two data sets on the
disturbance induced variation of the identified parameter was investigated. This idea is
expanded to include more than two data sets in the identification procedure. This time
n = 1000 Monte-Carlo simulations are performed for the system identification utilizing
nexp ∈ {x | x ∈ N, x ≤ 6} different data sets, all of which are observations based on the
same experiment. The sample standard deviation sψ (nexp ) of the results of the Monte-
Carlo simulation corresponding to the system identification based on nexp data sets can be
seen in Fig. 4.10. A vertical distance least squares fit of the function
k
sψ (nexp ) = √ (4.47)
nexp
to the sample standard deviations sψ (nexp ) that resulted in the coefficient kLSQ = 0.0259 ≈
sψ (nexp = 1) can also be seen in the figure. This function seems to coincide with the
equation
sψ (nexp = 1)
sψ (nexp ) = √ (4.48)
nexp
which is to be expected under the hypothesis that the least squares fit of one function to
multiple data sets is equal to the fit of the function to the mean of those data sets. In this
case the disturbance induced variation is reduced by the same factor √n1exp , which follows
from the reduction of the variation of the mean of the noise disturbances based on nexp
Figure 4.10: Expected standard deviation due to disturbance induced variation of the
parameter ψ when combining multiple data sets. The results are based on a Monte-Carlo
simulation.
samples according to the central limit theorem.
4.1.8 System Identification of the Model Parameter and the Initial

Conditions of the Experiment
The model parameters of the system described by equation (4.17) include the mass of
the object m, the gravitational constant of acceleration g and the drag coefficient c. The
former two of these parameters are assumed to be known. The other two constants that
influence the result of the IVP are the initial conditions of the state variables: the initial
position x1 (t0 ) = s0 and the initial velocity x2 (t0 ) = v0 . When the exact values of these
initial values aren’t precisely known, they can be included as parameters that are to be
estimated by the system identification.
One can easily imagine an experiment that allows for the precise determination of all the
initial conditions. This would mean that these initial values do not need to be estimated.
But what would happen when the experiment and the measurement are not coordinated
correctly. A shift in time between the start of the experiment and the start of the measure-
ment would introduce an error into the data that cannot be compensated by fixed initial
conditions. This is not a problem in this example as the measurement identification data
is artificially generated at precise points in time. Nevertheless this section is about the
investigation of the effects of adding variable initial conditions to the task of system iden-
tification.
The set of all parameters that are to be identified are combined into one column vector ψ.
⎡ ⎤ ⎡ ⎤
ψ1 c
⎢ ⎥ ⎢ ⎥
ψ = ⎣ψ2 ⎦ = ⎣s0 ⎦ (4.49)
ψ3 v0
The new cost function defined for the system identification problem based on one data set
ξA is now dependent on these three parameters ψ.
εA (ψ) = ε(ξA , ψ) (4.50)

εA (ψ) = ξA − y(ψ)22 . (4.51)
This makes it more difficult to analyse the topology of the cost function in order get an
insight about potential local minima. If one of the parameters is set to a fixed value or
expressed as a combination of the other two, the objective function can be visualized as
a three dimensional surface plot over a two dimensional region of the parameter space.
This can be seen as a contour plot of ε(ψ1 , ψ2 ) for ψ1 ∈ [0, 1], ψ2 ∈ [25, 35] and ψ3 =
0 in Fig. 4.11 and for ψ1 = 0.5, ψ2 ∈ [25, 35] and ψ3 ∈ [−5, 5] in Fig. 4.12. These
two visualizations only represent two orthogonal slices through the three dimensional
parameter space. Nevertheless, the objective function appears to be monotonically falling
towards one global minimum. This means that neither larger deviations in the initial guess
for the parameter c nor for the initial conditions s0 or v0 should affect the results of the
identified parameter set.
To evaluate the effect that the addition of the initial conditions to the parameter set has on
the results of the drag coefficient, another Monte-Carlo simulation was performed based
on the same data sets as in section 4.1.4. As the initial condition were previously fixed at
the exact values and are now free to be estimated, it is to be expected that the disturbance
induced variation of the initial conditions will also have a negative effect on the estimated
drag coefficient c. The numerical optimization of εA (ψ) will use the freedom in the initial
conditions to get a better fit on the noisy ξA which results in a worse fit to the exact
system output yexct compared to the optimization of the cost function that was fixed to
the exact initial conditions as in section 4.1.4. The sample mean and standard deviation
of the identified parameters based on the results of N = 1000 identification procedures
are shown in table 4.4. Also listed are the p-values pKS of the Kolmogorov-Smirnov tests
performed on the normalized results.
As was expected, the standard deviation of the drag coefficient sψ1 is larger than in the
results of the Monte-Carlo simulation in section 4.1.4, where only c was identified. This
leads to the conclusion that it might be better to fix the initial conditions of the simulation
model to a best estimate based on the experimental setup instead of including them into
the identification parameter set, if this best estimate is more precise than the disturbance
2500
2000
1500
1000
500
Figure 4.11: Contour plot of the objective function ε(ψ) evaluated for a two dimensional
slice of the parameter space: ψ1 ∈ [0, 1], ψ2 ∈ [25, 35] and ψ3 = 0.
3500
3000
2500
2000
1500
1000
500
Figure 4.12: Contour plot of the objective function ε(ψ) evaluated for a two dimensional
slice of the parameter space: ψ1 = 0.5, ψ2 ∈ [25, 35] and ψ3 ∈ [−5, 5].
induced variation cause by the additional freedom of the objective function. When dealing
with more accurate measurement data, the disturbance induced variation is less impactful.
Therefore it might be less likely to find a better estimate for the initial conditions based on
the experimental setup compared to the system identification of those initial conditions.
ψ1 [kg/m] ψ2 [m] ψ3 [m/s]

ψ̄ i 0.50304 30.0256 -0.0611812
sψi 0.0571966 0.4999 0.688684
pKS 0.279727 0.924865 0.112976
Table 4.4: Results of the Monte-Carlo simulation of the system identification of the pa-
rameter and the initial conditions of free fall experiment.
4.2 The Duffing Equation and the Nonlinear Mass and

Spring System
The second case study looked at in this chapter is about the system identification of a
nonlinear mass and spring system. The equations of motion for the mass and spring
system can be derived by applying Newton’s second law by equating the sum of forces
acting on the mass m. For a mass m restricted to motions in one horizontal direction there
may be the force of the spring Fsp , a resistive friction force Ff and an external force F
acting on it. This results in the equation of motion
mẍ + Fsp + Ff = F. (4.52)
This seemingly simple system can be used to investigate a range of different nonlinear
systems. Both the force of the spring Fsp as well as the friction force Ff can be modeled as
linear or with varying degrees of nonlinearity [35]. For example the following equations
describe a linear, a hardening, and a softening spring respectively.
Fsp = kx (4.53)
Fsp = k[1 + (hx)2 ]x (4.54)
Fsp = k[1 − (sx)2 ]x (4.55)
This nonlinear behaviour is illustrated in Fig. 4.13 for a spring coefficient k = 64 N/m
and a hardening or softening coefficient of h = s = 2 m−1 respectively. Of course, any
other relation between the displacement of the mass x and the spring force Fsp in form of
a function could be used as well.
The general form of the so called Duffing equation can be formulated as
Figure 4.13: Spring characteristic for a linear, a hardening and a softening spring.
ẍ + αẋ + β x + γx3 = F. (4.56)
Assuming that the friction force is linear viscous Ff = bẋ(t) results in an equivalent equa-
tion. For example
mẍ + k[1 + (hx)2 ]x + bẋ = F. (4.57)
for a hardening spring.
4.2.1 Simulation Model

The system of the mass and hardening spring that is to be investigated is described by
equation (4.57). For F = u(t), x(t) = x1 (t) and ẋ(t) = x2 (t) the state space representation
of this system is

ẋ1 x2
ẋ = = 1 = f (x, u) (4.58)
ẋ2 m u − k[1 + (hx 1 )2 ]x − bx
1 2
with the observer equation

y = 1 0 x = x1 . (4.59)
For any set of model parameters

T
p= k h b (4.60)
and any set of initial conditions

T
x0 = x1,0 x2,0 = x(t = 0) (4.61)
and any input u(t) an approximate solution of the initial value problem can be computed
by numerical integration, e.g. by applying the MATLAB® function ode45().
4.2.2 System Identification Based on Two Data Sets A and B

For the purpose of identifying the parameters of the mass and hardening spring system
two experiments labeled A and B are defined. The sinusoidal input uA (t) and step input
uB (t) corresponding to those experiments are shown in Fig. 4.14. These input sequences
result in a system excitation on two different levels. The goal of this section is to analyse
the effect that these different experimental setups have on the identified parameters.
The simulation output of each experiment is labelled as

yA (ψ) = [y(ψ, uA (t))t=t , y(ψ, uA (t))t=t +tstep , . . . , y(ψ, uA (t))t=t ]T (4.62)
0 0 f

yB (ψ) = [y(ψ, uB (t))t=t , y(ψ, uA (t))t=t +t , . . . , y(ψ, uB (t))t=t ]T (4.63)
0 0 step f
The system output is sampled every tstep = 0.02 s. Each experiment lasts ten seconds:
[t0 , t f ] = [0 s, 10 s].
The parameters that are to be identified are the model parameters p as well as the initial
conditions of the respective experiment x0,A and x0,B . This can be summarized as an
identification parameter vector
T
ψ = pT xT0,A xT0,B . (4.64)
The synthetic measurement data sets ξA and ξB are generated by adding Gaussian mea-
surement noise, sampled from the normal distribution N (μ = 0, σ = 0.01), to the re-
sponse of the system to uA (t) and uB (t) under the exact parametrization
T T
pexct = kexct hexct bexct = 64 N/m 2 m−1 0.1 kg/s (4.65)
with the exact initial conditions

T
x0,A,exct = x0,B,exct = 0 m 0 m/s . (4.66)
The objective functions that are to be minimized for the purpose of system identification
Figure 4.14: System inputs uA (t) and uB (t) for the experiments A and B simulated on the
mass and hardening spring system.
based on the identification data sets ξA , ξB or both of them simultaneously are
εA (ψ) = ξA − yA (ψ)22 , (4.67)

εB (ψ) = ξB − yB (ψ)22 and (4.68)
2
ξ yA (ψ)
A
εAB (ψ) = − . (4.69)
ξB yB (ψ) 2
The initial guess for the parameters that are to be identified is

T
ψ0 = 50 N/m 0.1 m−1 0.01 kg/s ξA (1) 0 m/s ξB (1) 0 m/s . (4.70)
The results of the numerical optimization are denoted as
ψA = argmin εA (ψ) , (4.71)

ψ
ψB = argmin εB (ψ) and (4.72)

ψ
ψAB = argmin εAB (ψ). (4.73)

ψ
Fig. 4.15 shows the identification data set ξA as well as the exact system output yA (ψexct )
on which that data is based on and a comparison of the estimated system outputs yA (ψA ),
yA (ψB ) and yA (ψAB ) based on the results of the system identification procedure. Fig.
4.16 shows the same for experiment B. As can be seen, the estimated system output of
Figure 4.15: Comparison of the estimated system outputs of experiment A based on the
results ψA , ψA and ψAB of the system identification.
yA (ψA ) and yB (ψB ) fit rather nicely to the respective output of the exactly parametrized
system. However, the estimate of yA based on ψB as well as the estimate of yB based on
ψA results in rather poor performance. The estimates based on ψAB for each experiment
are similar to both yA (ψA ) and yB (ψB ).
Figure 4.16: Comparison of the estimated system outputs of experiment B based on the
results ψA , ψA and ψAB of the system identification.
4.2.3 Statistical Analysis – Monte-Carlo Simulation

As was presented in the last section, the combination of multiple data sets can be advan-
tageous in the identification of dynamic systems. Especially when dealing with nonlinear
dynamic systems it is important to take this into consideration when designing the exper-
iments that are used to extract information about a system.
To gain more insight into the effect that the combination of the information of two data
sets ξA and ξB has on the system identification of the nonlinear mass and spring system,
a Monte-Carlo simulation is performed. The same procedure of optimizing εA , εB and
εAB as in section 4.2.2 is performed n = 1000 times. New noise disturbances at the same
noise level were generated for each run. A comparison of the resulting system outputs
yA (ψ) and yB (ψ) for all parameter sets ψA , ψA and ψAB identified in the Monte-Carlo
simulation can be seen in Fig. 4.17 and 4.18. The same discussion about the accuracy
and inaccuracy of ψA , ψA and ψAB in the estimation of yA and yB as at the end of section
4.2.2 can be made again here. However, some major outlier trajectories can be made out
under the results of the Monte Carlo simulation. All n = 1000 successfully converged in
parameter sets ψA , ψA and ψAB . But some of those apparently lie at wrong local minima.
As was already mentioned in section 3.2.1 and section 4.1, a visual analysis of the topol-
ogy of an objective function ε(ψ) in a more than two dimensional parameter spaces ψ
is difficult. However, conclusions can be drawn from the results of the numerical op-
timization, especially when combining a larger number of results, as for example after
Figure 4.17: Comparison of the simulation output of experiment A corresponding to all

data sets ψA , ψA and ψAB identified in the Monte-Carlo simulation.
Figure 4.18: Comparison of the simulation output of experiment B corresponding to all

data sets ψA , ψA and ψAB identified in the Monte-Carlo simulation.
Figure 4.19: Comparison of the results of the Monte-Carlo simulation in the form of
histograms for the identified parameter k based on the identification data generated from
experiment A and B as well as both of them combined.
performing a Monte-Carlo simulation.

One thing of note is that as the objective function ε(ψ) depends on the identification data
ξ, which is slightly different in each run of the Monte-Carlo simulation that was per-
formed here. Nevertheless, unless when dealing with ill-posed problems, the topology
of the objective function should be dominated by the system model and the experimental
design, i.e. the input u(t), rather than the measurement noise. This means that a single
outlier trajectory might be caused by a disturbance induced local minima of the respective
objective function. But if there is an accumulation of identified parameter sets in a region
of the parameter space that is not the global minimum, then there is probably a local min-
ima regardless of the noise disturbance.
This can be visually investigated by looking at the histogram of a selected parameters.
As the spring stiffness k has a dominant effect in the behaviour of the system, that pa-
rameter was selected to be looked at more closely. Fig. 4.19 shows a comparison of the
histograms of the identified values of k based on the identification data of experiment A,
B and the utilization of both data sets. As can be seen, both kA and kB show a wider
approximately bell shaped distribution around the exact value of kexct = 64 N/m as well
as a larger amount of outliers. The reason for this might be that local minima of εA (ψ)
or εB (ψ) that might exist in certain regions of the parameter space are either not existent
for εAB (ψ) or that this objective function is shaped in a way that the path in the parameter
space that is taken in the iterative numerical optimization just does not fall into those lo-
cal minima but successfully converges to the global minimum for most cases. In fact, the
Figure 4.20: Histogram of the identified parameter kAB after the removal of outliers.
histogram of kAB shows hardly any outliers. Based on the histogram of kAB shown in Fig.
4.19 a range of 62 N/m < kAB,s < 66 N/m can be selected, which does not seem to contain
any vast outliers. ns = 995 of the n = 1000 results of the Monte-Carlo simulation fall into
that range. Fig. 4.20 shows a detailed histogram of these selected parameters kAB,s as well
as the normal distribution based on the sample mean k̄AB,s and sample standard deviation
skAB,s .
When trying to differentiate between outliers due to wrong convergence and parame-
ter induced variation a wider range has to be selected for the results of kA and kB , e.g.
60 N/m < kA,s , kB,s < 68 N/m. Table 4.5 shows the results of a Kolmogorov-Smirnov test
performed on the normalized data set of the selected parameter sets kA,s , kB,s and kAB,s in
the form of p-values pKS . The data sets were normalized according to the sample mean
k̄ and sample standard deviation sk , which are also shown in the table below. Performing
the test on the whole parameter sets including all outliers failed for kA , kB as well as for
kAB (pKS << 1%).
kA,s [N/m] kB,s [N/m] kAB,s [N/m]

ns 990 949 995
k̄ 64.0158 63.9046 63.9921
sk 0.5184 1.2272 0.1387
pKS 0.3490 0.0845 0.8819
Table 4.5: Results of the Monte-Carlo simulation for the nonlinear mass and spring system
after the removal of outliers.
Figure 4.21: Comparison of the system response to the verification test case with input
uV (t). The system is parameterized according to the selected results ψA,s , ψB,s and ψAB,s
as well as ψexct .
Fig. 4.21 shows a qualitative visualisation of the fact that the identification based on
two data sets ξA and ξB leads to better results than the identification based on a single data
set. Here the system response to a verification test case is presented. The input uv (t) is a
multi-step function ⎧
⎪
⎪ 10 if t < 3f
t
⎪
⎨
uv (t) = −10 if t3f ≤ t < 2t3f (4.74)
⎪
⎪
⎪
⎩ 20 if t ≥ 2t f
3
for t ∈ [t0 , t f ]. The presented system responses correspond to all the parameter sets ψA , ψB
and ψAB , which where identified in the Monte-Carlo simulation excluding those outliers
that were detected in the previous look at the histogram of the identified spring stiffness
k. As can be seen, the estimation of the system output yv (ψAB ) results in the best approx-
imation of the behaviour of the exactly parametrized system, even though the excitation
level of this verification test case lies further above experiment A and B.
4.2.4 Multiple Starting Point Procedure

As was seen in the results of section 4.2.3, the objective functions εA (ψ), εB (ψ) and
εAB (ψ) appear to not be of a perfect convex shape. Therefore, depending on the selected
initial guess ψ0 and the selected numerical solver, the numerical optimization of any
of these scalar-valued optimality criteria might result in a wrong local minima. When
encountering such a situation, the simplest way of getting around the issue is to have
another try at the parameter estimation after selecting a new, potentially better initial
guess. For system identification problems, where it is difficult to come up with a well-
founded initial guess ψ0 , a multiple starting point procedure can be applied. This means
that a set of initial guesses is selected and the results of each of those can be used in order
to come to a consensus on local and global minima or to at least achieve a single correct
convergence to the global minimum.
In this section the application of a multiple staring point procedure for the identification of
the nonlinear mass and spring system based on two experiments A and B is investigated.
For this purpose the same data sets ξA and ξB as in section 4.2.2 are used. The parameters
that are to be determined are
T
ψ = k h b x0,A ẋ0,A x0,B ẋ0,B . (4.75)
The set of initial guesses for each of those parameters are:
k ∈ {1, 10, 25, 50, 75, 100, 150} , (4.76)

h ∈ {0, 0.1, 0.5, 1, 2, 5, 10} , (4.77)
b ∈ {0, 0.01, 0.1, 0.5} , (4.78)
x0,A ∈ {0, ξA (1)} , (4.79)
ẋ0,A = 0 , (4.80)
x0,B ∈ {0, ξB (1)} and (4.81)
ẋ0,B = 0. (4.82)
This means that there are 784 different combinations for the initial guess ψ0 in the nu-
merical minimization of εAB (ψ), as defined in equation 4.69.
In this example problem every initial guess ψ0,i lead to convergence to an identified pa-
rameter set ψAB,i . This does not need to be the case in every problem. For example the
ODE-solver might fail to compute a solution for certain parameter sets. This needs to be
handled in the algorithm of the multiple starting point procedure.
In order to evaluate the goodness-of-fit of each results, the objective function value
εAB (ψAB ) is looked at. Fig. 4.22 presents εAB (ψAB,i ) on a logarithmically scaled ver-
tical axis over the the index i which corresponds to a unique initial guess ψ0,i that lead
to the result ψAB,i . As can be seen, there are accumulations of results on different lev-
els of the objective function value. These probably correspond to different local minima.
A majority of the results seems to approximately lie on the same lowest level which is
clearly separated from the other results. Therefore, a simple way of trying to distinguish
between parameter sets that correspond to the global minimum of εAB (ψ) and ones that
belong to different local minima would be to put a threshold on the goodness-of-fit. In this
case a value of εmax = 0.07 is selected. This results in 682 out of the 784 parameter sets
Figure 4.22: Comparison of the objective function values ε(ψAB ) for each identified pa-
rameter set ψAB based on a different initial guess that lead to a convergence of the numer-
ical optimization.
which correspond to ε(ψ) < εthresh . Fig. 4.23 shows the estimated system output based
on these selected parameters sets compared to the identification data and the response of
the exactly parametrized system. As can be seen, all identified parameter sets with objec-
tive function value below the selected threshold results in a practically identical system
behaviour.
Figure 4.23: Comparison of the simulation output of experiment A and B for all 682
identified parameter sets corresponding to ε(ψ) < εthresh compared to the identification
data.
4.3 Dynamic Friction Models – Stribeck Effect

In this section the system identification of a nonlinear dynamic friction model, which
embodies the Stribeck effect, is investigated.
4.3.1 Modeling of Friction

The modeling of friction is a whole field of science on its own. In general the resistive
force Ff is dependent on the state of the mass and the contact with the environment.
The phenomenon that friction decreases with increasing velocity is called the Stribeck
Effect [36]. In simple models of friction between two bodies that are in contact, Ff can
have components due to static and dynamic Coulomb friction. The exact modeling of the
transition from static to dynamic friction presents one of the biggest challenges of friction
modeling, both in the correct representation of reality and in the numerical solution of the
resulting mathematical models because of the ensuing discontinuity.
In the motionless state, static friction is described by a static friction coefficient μ0 and
the normal contact force FN . It is also called the stiction force or break away force. Ff
opposes the sum of the forces acting on the body and its orientation must therefore be
determined accordingly.

Ff ≤ μ0 FN if ẋ = 0. (4.83)
Only once the body is in motion, a simpler description is given by the dynamic friction
coefficient μ < μ0 :
Ff = μFN sign(ẋ) if ẋ = 0. (4.84)
If a moving body is in contact with a viscous medium like gases of fluids, a nonlinear
viscous friction term can be added:
Ff = b|ẋδẋ | sign ẋ (4.85)
which simplifies to
Ff = bẋ (4.86)
for linear viscous friction (δẋ = 1). The combination of these effects is defined in the
Stribeck friction model:
Ff = μFn sign(ẋ) + bẋ + Fs (ẋ) (4.87)
where Fs (ẋ) is a function describing the Stribeck effect [37]. In some simulation environ-
ments the characteristic (v, Ff ) curve resulting from the Stribeck effect is implemented as
a lookup table [38]. The accurate representation in the form of mathematical models has
been attempted in different ways. One of which is the LuGre friction model described in
the next section.
4.3.2 The LuGre Friction Model

The LuGre friction model is a dynamic friction model first described in [39], which in-
troduces a new state variable z(t) in order to represent real world friction characteristics
is an accurate way. For a body of mass m sliding on a surface and subject to friction, the
interface at the point of contact is envisioned as a little bristle. This bristle acts both as a
spring and a damper. The new state variable z(t) describes the deflection of that bristle.
For a stiffness σ0 and a damping coefficient σ1 of the bristle the force acting on the body
due to the deflection of the bristle can be described as
F = σ0 z(t) + σ1 ż(t). (4.88)
By further adding a linear viscous friction term proportional to the velocity of the body
v(t) as in equation (4.86) the dynamic friction force acting on the body is described as
Ff = σ0 z(t) + σ1 ż(t) + bv(t). (4.89)
The dynamics of the bristle are described by
∂z |v|
ż(t) = = v(t) − z(t) (4.90)
∂t g(v)
with g(v) describing the transition from static to dynamic Coulomb friction as an expo-
nentially decaying function with respect to v(t)2 :
v 2
σ0 g(v) = FC + (FS − FC )e−( vs ) (4.91)
where FS = μ0 FN is the static friction force and FC = μFN is the dynamic Coulomb friction
force. The Stribeck velocity vs determines the rate at which g(v) transitions from σFS0 to FσC0
with increasing velocity v(t).
4.3.3 The Simulation Model

For the purpose of investigating the system identification of the LuGre friction model,
a system of a mass m lying on a horizontal surface is looked at. The body of mass is
driven by a unidirectional force u(t). The system is subject to friction. This results in the
equation of motion
mẍ(t) = u(t) − Ff (t) (4.92)
where x(t) is the position of the mass m. The friction force Ff is modelled according
to the LuGre model including the linear viscous term. The exact system parameters are
assumed to be
m = 1 kg . . . mass
N
σ0 = 1 × 105 . . . stiffness of the bristle
m
m
v0 = 0.01 . . . Stribeck velocity
s
kg
b = 0.4 . . . linear viscous friction coefficient
s
μ0 = 0.15 . . . static Coulomb friction coefficient
μ = 0.10 . . . dynamic Coulomb friction coefficient
The damping coefficient σ1 of the bristle is set to critical damping as suggested by [39]:

σ0
σ1 = 2 (4.93)
m
The normal force FN is assumed to be constant and due to gravity: FN = mg with g =

9.81 m/s2 .
For the numerical approximation of the system response the MATLAB® functions ode45()
and ode23s() were tested. As both of them lead to practically the same results ode45()
was selected as the method of choice to compute the results shown in the following sec-
tions.
Figure 4.24: Triangular signal u(t) that is applied to the LuGre friction model in order to
analyse its properties.
The Influence of Varying System Excitation Levels
The LuGre friction model is able to produce system behaviour which is similar to experi-
mental observations of real world friction characteristics [37, 39, 40]. The LuGre model
accounts for micro or pre-sliding displacements corresponding to an applied excitation
force u << μFN as well as friction hysteresis of varying form in cyclic processes which
depends not only on the parametrization but also on the rate of change due to the nature
of the dynamic friction model .
For the purpose of identifying the system parameters of the LuGre friction model, it might
therefore be important to design experiments that result in measurement data which gives
insight into these different behaviours at smaller and higher excitation levels of the sys-
tem.
A downside of the LuGre model is, that it can pose difficulties in the numeric integration
due to the stiff parametrization of the bristle [37, 38]. Therefore it would be advantageous
if the experimental design results in a setup that is as easy a possible to solve numerically.
To investigate the behaviour of the model, a triangular signal u(t) as shown in Fig. 4.24
with varying amplitude û is applied to the system. A single cycle of u(t) is looked at with
a period of five seconds.
Fig. 4.25 shows the system response to the described input signal for a selection of small
amplitudes û < μ0 mg. These results show a small pre-sliding displacements x(t) which
the LuGre model allows for due to the fact that it is not perfectly rigid before the static
friction force is overcome. Fig. 4.26 shows the system response to larger input signals
û ≥ μ0 mg. As can be seen, once the static friction force is overcome, the system exhibits
macro level displacements x(t). A small displacement goes together with a velocity v > 0,
which lowers the friction force due to the Stribeck effect, which results in faster accel-
eration. As can be seen by comparing the two figures, the system response for different
excitation levels may lie on a completely different magnitude.
The important insight to take away from this is that there exists a sort of boundary be-
tween the system responding with micro or macro level displacements, which is related
the applied excitation force u(t) but which of course also depends on the system parame-
ters.
For example, a smaller bristle stiffness σ0 would result in a larger pre-sliding displace-
ment and therefore a higher velocity v(t) as a response to the same input force u(t). A
smaller v0 would result in a more rapid lowering of the friction force with rising velocity
v(t). These parameter changes would result in the overcoming of the static friction force
at lower input levels. Of course a lower μ0 would result in the same effect.
During the system identification procedure, starting from an initial guess ψ0 , a numeri-
cal optimization algorithm iterates over the parameter space in order to find the optimal
parameter set, which results in the fitting model output y(ψ) to the measured system re-
sponse ξ. When the measurement data corresponds to macro level displacements, but the
initial guess results in micro level displacements or the other way around, correct conver-
gence might be poor or even impossible. Therefore one has to be careful when working
with identification data sets that lie close to that boundary, i.e. where small variations in
the parameters cause the overcoming of that boundary and consequently a drastic change
in the system behaviour.
To conclude the discussion of the system behaviour at varying excitation levels the follow-
ing figures present the (x, Ff ) and (v, Ff ) characteristic for the same triangular waveform
inputs u(t) as discussed so far. Ff was computed according to equation (4.89). Fig. 4.27
and Fig. 4.28 show the friction hysteresis (x, Ff ) at lower and higher excitation of the
system. Fig. 4.29 shows the (v, Ff ) hysteresis for small excitation levels as well as one
flaw of the LuGre model. As the friction is modelled as a dynamic system via the state
z(t), the friction foce Ff does not always point against the direction of motion sign(v(t)).
Fig. 4.30 shows the (v, Ff ) characteristic for higher excitation levels. This results in the
typical Stribeck curve where Ff starts at μ0 mg, sharply falls off to μmg as the motion
begins and rises with the viscous friction term bv(t).
Figure 4.25: System response x(t) to the triangular input signal u(t) at smaller excitation
levels.
Figure 4.26: System response x(t) to the triangular input signal u(t) at larger excitation
levels.
Figure 4.27: (x(t), Ff (t)) characteristic for smaller excitation levels u(t) < μ0 mg.
Figure 4.28: (x(t), Ff (t)) characteristic for higher excitation levels u(t) > μ0 mg.
Figure 4.29: (v(t), Ff (t)) characteristic for smaller excitation levels u(t) < μ0 mg.
Figure 4.30: (v(t), Ff (t)) characteristic for higher excitation levels u(t) > μ0 mg.
The Influence of Varying Parameters
To look at the effect that a small variation of the parameters has on the system response,
the numerical approximation of the Jacobian J(ψ) of the system response x(t) is now
looked at. The parameters under investigation are
T
ψ = b σ 0 vs μ 0 μ . (4.94)
The Jacobian is approximated by a central finite difference formula for small changes
around ψexct as specified in the beginning of this section 4.3.3. As some of the param-
eters lie on a different scale of magnitude compared to others, an individual step size of
Δψi = 1%ψi was selected for each parameter ψi .
For a small excitation with the triangular signal u(t) with amplitude û = 0.5μ0 mg the Ja-
cobian is presented in Fig. 4.31. For this input, the system shows a larger change in the
system response for a variation of σ0 and μ0 . The variation of the other parameters shows
less of an effect. For clearer visibility, these components of the Jacobian are shown in
detail in Fig. 4.32. Note that the variation of vs shows a similar but not the same change
in the response as a variation of μ.
Fig. 4.33 shows the numerically approximated Jacobian for the same input form but with
a higher amplitude û = 1.5μ0 mg. As can be seen, for this test case the variation of σ0
results in the smallest variation of the response.
Based on this analysis it can be assumed that the system identification based in pre-sliding
displacement data could result in better estimation of σ0 compared to macro level dis-
placement data due to the larger effect of small variations on the value of the objective
function.
Figure 4.31: Numerically approximated Jacobian for a triangular input signal u(t) with
û = 0.5μ0 mg.
Figure 4.32: Selected components of the numerically approximated Jacobian for a trian-
gular input signal u(t) with û = 0.5μ0 mg.
Figure 4.33: Numerically approximated Jacobian for a triangular input signal u(t) with
û = 1.5μ0 mg.
4.3.4 System Identification of the LuGre Friction Model

The system under investigation is the same as described on section 4.3.3. As σ1 is as-
sumed to be dependent on m and σ0 via the relation of critical damping (4.93) it is ex-
cluded from the set of parameters that are to be identified. The mass m is also assumed to
be known. Therefore the parameter set that is to be identified is
T
ψ = b σ0 vs μ0 μ . (4.95)
with ⎡ ⎤ ⎡ ⎤
bexct 0.4 kg/s
⎢ ⎥ ⎢ ⎥
⎢σ0,exct ⎥ ⎢ 1 × 105 N/m ⎥
⎢ ⎥ ⎢ ⎥
ψexct =⎢ ⎥ ⎢ −2
⎢ vs,exct ⎥ = ⎢1 × 10 m/s⎥
⎥ (4.96)
⎢ ⎥ ⎢ ⎥
⎣μ0,exct ⎦ ⎣ 0.15 ⎦
μexct 0.10
System Identification Based on a Single Data Set ξA : Pre-Sliding Displacement
In the previous section the cyclic behaviour of the nonlinear friction model was investi-
gated. Testing has shown that attempting the system identification based a corresponding
input-output data set results in problems in the convergence of the nonlinear curve fitting
problem. Therefore, in this section a slightly simplified excitation force u(t), that only
points in one direction is selected for the experimental design. This eliminates zero cross-
ings from the state trajectory of the position x(t).
Figure 4.34: Triangular input signal uA (t) with û = 0.5μ0 mg that is applied to the LuGre
friction model on order to generate identification data.
An identification data set ξA is generated based on a triangular input signal uA (t) con-
sisting of a single peak with amplitude ûA = 0.5μ0,exct mg and a period of five seconds
as shown in Fig. 4.34. This input force will not be high enough to overcome the static
friction. Therefore the movement x(t) is limited to pre-sliding displacement.
For the identification the system response is sampled every tstep = 0.2 s. The response of
an arbitrarily parametrized system to the described input uA (t) at these points in time is
denoted as yA (ψ). The identification data set ξA is generated by adding gaussian mea-
surement noise δA , which is sampled from the normal distribution N (0, 2.5 × 10−7 ), to
the response of the exactly parametrized system:
ξA = yA (ψexct ) + δA . (4.97)
The resulting data points can be seen in Fig. 4.34. The initial guess for the system
parameters is selected as
T
ψ0 = 0.3 kg/s 0.75 × 105 N/m 2 × 10−2 m/s 0.12 0.12 . (4.98)
As the system response of to the input uA (t) lies on a reactively small scale, the objective
function is defined as
εA (ψ) = WA (ξA − yA (ψ))22 , (4.99)
Figure 4.35: Identification data set generated by adding noise to the sampled response of
the exactly parameterized system to the input signal uA (t).
where WA is a diagonal weighting matrix
1
WA = IN (4.100)
max(ξA )
with IN as a unity matrix of size N, the number of data points. Additionally, in order to
further improve the convergence of the numerical optimization the parameter space was
normalized by a linear transformation with the following mapping:
[0, ψexct ] → [0, 1]. (4.101)
Applying a Levenberg-Marquardt algorithm results in the identified parameter set

⎡ ⎤
0.300 595 75 kg/s
⎢ ⎥
⎢ 99 889.258 N/m ⎥
⎢ ⎥
ψA = ⎢ ⎥
⎢0.010 017 185 m/s⎥ . (4.102)
⎢ ⎥
⎣ 0.150 331 27 ⎦
0.120 019 62
As can be seen, the bristle striffness σ0 , the Stribeck velocity vs and the static friction
coefficient μ0 are identified accurately. However, this can not be said for the identified
values of the viscous friction coefficient b and the dynamic friction coefficient μ.
Figure 4.36: Identification data set generated by adding noise to the sampled response of
the exactly parametrized system to the input signal uB (t).
System Identification Based on a Single Data Set ξB : Macro Level Displacement
A second identification data set ξB is generated based on a triangular input signal uB (t).
The input has the same form as uA (t), which was shown in Fig. 4.34, consisting of a
single peak but with larger amplitude ûB = 1.5μ0,exct mg, which will overcome the static
friction and lead to macro level displacements. The input is again applied over a period
of five seconds.
As previously, for the identification the system response is sampled every tstep = 0.2 s.
The response of an arbitrarily parametrized system to the described input uB (t) at these
points in time is denoted as yB (ψ). In order to achieve a certain comparability to the
results of the system identification based on the pre-sliding experiment, a similar signal
to noise ratio is selected. The identification data set ξB is generated by adding gaussian
measurement noise δB , which is sampled from the normal distribution N (0, 2.5 × 10−2 ),
to the response of the exactly parametrized system:
ξB = yB (ψexct ) + δB . (4.103)
The resulting data points can be seen in Fig. 4.36. The initial guess for the system
parameters is selected as ψ0 as before. The objective function that needs to be optimized
in order to solve this system identification problem is defined as
εB (ψ) = WB (ξB − yB (ψ))22 , (4.104)

where WB is a diagonal weighting matrix
1
WB = IN (4.105)
max(ξB )
with IN as a unity matrix of size N, the number of data points. The same rescaling of the
parameter space as before was applied again. Applying a Levenberg-Marquardt algorithm
to minimize εB (ψ) results in the identified parameter set
⎡ ⎤
0.456 503 27 kg/s
⎢ ⎥
⎢ 67 374.558 N/m ⎥
⎢ ⎥
ψB = ⎢ ⎥
⎢0.010 472 251 m/s⎥ . (4.106)
⎢ ⎥
⎣ 0.153 278 04 ⎦
0.095 141 50
For this setup, the viscous friction coefficient b was identified more accurately, as can be
expected for a data set corresponding to a movement at higher velocities. The dynamics of
the bristle, which have only minor effects on the macro level motion of the system, repre-
sented by the bristle stiffness σ0 are overshadowed by the measurement noise. Therefore
this parameter was not identified very accurately. The static as well as the dynamic fric-
tion coefficient μ0 and μ were both identified with reasonable precision.
System Identification Based on Multiple Response Data Sets: ξA and ξB
As has already been discussed in section 4.1.5 multiple identification data sets can be
used simultaneously in order to make a more accurate prediction of the exact system
parameters. In this section the observations of micro level displacements ξA and of macro
level displacements ξB are utilized and the feasibility of overcoming the shortcomings in
the identified parameter sets when using those data sets individually is investigated.
The objective function for the identification of the parameters ψ based on two data sets
can be defined as 2
ξ y (ψ)
A A
εAB (ψ) = − . (4.107)
ξB yB (ψ) 2
As the system responses to uA (t) and uB (t) lie on a completely different scale an individual
weighting for the corresponding data sets is added to the objective function.
2
W 0 ξA yA (ψ)
A
εAB (ψ) = − (4.108)
0 WB ξB yB (ψ) 2
with the same weighting matrices WA and WB , that were defined in the previous sections.
Again, a Levenberg-Marquardt Algorithm was used to minimize this objective function,
which resulted in the identified parameter set

⎡ ⎤
0.499 620 43 kg/s
⎢ ⎥
⎢ 104 811.425 N/m ⎥
⎢ ⎥
ψAB = ⎢
⎢ 0.083 936 576 m/s ⎥.
⎥ (4.109)
⎢ ⎥
⎣ 0.137 755 71 ⎦
0.091 219 28
Compared to the exact parameters, these results are relatively good estimates of σ0 , μ0
and μ, a less accurate identified vs and b.
The resulting estimated system output for ψA , ψB and ψAB compared to the identification
data and the system response with exact parametrization ψexct can be seen in Fig. 4.37
for uA (t) and Fig. 4.38 for uB (t). While the parameter set ψAB identified based on both
data sets ξA and ξB does not produce the best performance in the test case A, it is still a
major improvement compared to the evaluation of the test case A with the parameter set
ψB or ψA in the test case B.
To summarize, the case study of the LuGre friction model clearly demonstrates the pos-
sible difficulty that can be encountered when only limited identification data is available
for the system identification of nonlinear dynamic systems. The correct estimation of the
model parameters is not necessarily only a question of achieving a high enough excitation
in the experiments that are used to extract information about the object of interest. It is
rather dependent on getting enough data at the right levels of system excitation. All in
all it was shown that different identification data sets can be combined in a system iden-
tification procedure in order to be able to compute an improved estimate of all the model
parameters.
Figure 4.37: Comparison of the system response to the input signal uA (t) for the exact
and the identified parameter sets.
Figure 4.38: Comparison of the system response to the input signal uB (t) for the exact
and the identified parameter sets.
Chapter 5
Conclusion
System identification of nonlinear dynamic systems is a challenging task because dealing

with nonlinear systems makes the system identification process more demanding com-
pared to linear dynamic systems. Both the problem of solving the differential equations
as well as the curve fitting problem of matching the obtained solution to the measured
data become more difficult. This is due to the fact that nonlinear dynamic system may
show a waste range of different behaviours depending on the parametrization as well as
the applied excitation. One the one hand, small changes in some parameters might lead
in large deviations of the system output, which makes it detrimental to have a good guess
about the probable range of the parameters. In most cases it should be possible to use a
priori knowledge about the system to select a good initial guess for the parameters. On the
other hand, disturbance induced variations might dominate the identified value of other
parameters. While the general approach of using the solution of the simulation problem
as basis for a nonlinear curve fitting problem can be followed every time, any object of
interest needs to be studied on a case by case basis. Appropriate experiments to extract
observational information need to be defined and a fitting simulation model needs to be
determined.
In this thesis the system identification of three nonlinear dynamic systems as presented
in chapter 4 based on synthetic measurement data was performed successfully. For this
task MATLAB® functions for the numerical integration of ODEs, mainly ode45(), in
conjunction with numerical optimization methods like the Gauss-Newton, Levenberg-
Marquardt and the Simplex method, the later of which is implemented in the MATLAB®
function fminsearch(), were incorporated in an algorithm for the system identifica-
tion based on the output error of the simulation model. Various aspects of the system
identification problem with respect to those example problems were discussed and illus-
trated.
In the case study of the object in free fall subject to drag due to air the system identifi-
cation of a single parameter - the friction coefficient - as well as the identification of this
parameter and the initial conditions was performed. As has been shown, in both cases the
85
CHAPTER 5. CONCLUSION 86
nonlinear curve fitting problem is well posed as the objective function appears to be of
convex shape over a wide range of the parameter space around the global minimum. The
influence of gaussian measurement noise on the identified parameters was investigated.
This resulted in the conclusion that for this example problem, a gaussian measurement
noise results in gaussian disturbance induced variation of the identified parameters. How-
ever, it is important to keep in mind that this will not be the case for every nonlinear
system identification problem. Furthermore, the compensation of the disturbance induced
variation by means of utilizing multiple identification data sets was investigated. This
resulted in the conclusion that, based on the results on a Monte-Carlo simulation, the
standard deviation of the identified parameters can be reduced by a factor of the inverse
of the square root of the number of data sets incorporated into the system identification
procedure.
In the case study of the nonlinear mass and spring system the utilization of observational
data from two different experimental setups in the system identification of both the model
parameters as well as the initial conditions of each experiment was investigated. This
example problem shows that the inclusion of multiple different data sets not only leads
to better results in terms of the disturbance induced variation of the identified parameters,
but can also improve the convergence of the nonlinear curve fitting problem. In addition
to that, the application of a multiple starting point procedure in order to find a way around
local minima to a global minimizer was presented.
In the third case study, the system identification of the LuGre friction model demanded
special attention to the experimental design, as the system exhibits a vastly different be-
haviour depending on the excitation level. As was shown, by combining identification
data sets corresponding to micro and macro level excitations of the system, a proper esti-
mation of all the model parameters can be achieved. Even though the technical feasibility
of executing these proposed experiments on the same system might cause practical prob-
lems, the advantage of combining the information contained in multiple data sets in order
to accurately identify a set of model parameters was clearly demonstrated.
Bibliography
[1] M. Bunge. Emergence and Convergence: Qualitative Novelty and the Unity of
Knowledge. Toronto studies in philosophy. University of Toronto Press, 2003.
[2] Lennart Ljung and Torkel Glad. Modeling of Dynamic Systems. Prentice-Hall, Inc.,
USA, 1994.
[3] Gene F. Franklin, Michael L. Workman, and Dave Powell. Digital Control of Dy-
namic Systems. Addison-Wesley Longman Publishing Co., Inc., 3rd edition, 1997.
[4] Brian Douglas. The fundamentals of control theory. Available at https://www.

engineeringmediallc.com/ (2020/01/14), 07 2019.
[5] Donald E. Kirk. Optimal Control Theory : An Introduction. Dover Publications,

April 2004.
[6] Amar G. Bose. A Theory of Nonlinear Systems. PhD thesis, Massachusetts Institute
of Technology, Dept. of Electrical Engineering, June 1956.
[7] G. Rath and M. Harker. Global least squares solution for lti system response. In
2017 International Conference on Applied Electronics (AE), pages 1–4, Sep. 2017.
[8] P. O’Leary and M. Harker. A framework for the evaluation of inclinometer data in
the measurement of structures. IEEE Transactions on Instrumentation and Mea-
surement, 61(5):1237–1251, May 2012.
[9] M. Harker and G. Rath. Global least squares for time-domain system identifica-
tion of state-space models. In 2018 7th Mediterranean Conference on Embedded
Computing (MECO), pages 1–6, 2018.
[10] M. Harker. Fractional Differential Equations: Numerical Methods for Applications.

Studies in Systems, Decision and Control. Springer International Publishing, 2020.
[11] Bernard Friedland. Control System Design: An Introduction to State-Space Meth-

ods. Dover Publications, 2005.
87
BIBLIOGRAPHY 88
[12] Lennart Ljung. Some aspects on nonlinear system identification. IFAC Proceed-
ings Volumes, 39(1):553 – 564, 2006. 14th IFAC Symposium on Identification and
System Parameter Estimation.
[13] Johan Schoukens and Lennart Ljung. Nonlinear system identification: A user-
oriented roadmap. ArXiv, abs/1902.00683, 2019.
[14] L. A. Zadeh. From circuit theory to system theory. Proceedings of the IRE,
50(5):856–865, 1962.
[15] José Borges, Vincent Verdult, Michel Verhaegen, and Miguel Ayala Botto. Separable
least squares for projected gradient identification of composite local linear state-
space models. 07 2004.
[16] Klaus Schittkowski. Numerical Data Fitting in Dynamical Systems: A Practical

Introduction with Applications and Software. Kluwer Academic Publishers, USA,
2002.
[17] Rolf Isermann and Marco Münchhof. Identification of Dynamic Systems: An Intro-
duction with Applications. Springer Publishing Company, Incorporated, 2014.
[18] William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery.
Numerical Recipes in C (2nd Ed.): The Art of Scientific Computing. Cambridge
University Press, USA, 1992.
[19] W. Richard Kolk and Robert A. Lerman. Analytic Solutions to Nonlinear Differential
Equations, pages 23–60. Springer US, Boston, MA, 1992.
[20] R. Ashino, M. Nagase, and R. Vaillancourt. Behind and Beyond the Matlab ODE
Suite. Computers and Mathematics with Applications, 40(4):491 – 512, 2000.
[21] Eric W. Weisstein. Runge-kutta method. From MathWorld—A Wolfram Web

Resource. https://mathworld.wolfram.com/Runge-KuttaMethod.
html (2020/06/10).
[22] Lawrence F. Shampine and Mark W. Reichelt. The matlab ode suite. SIAM J. Sci.
Comput., 18(1):1–22, January 1997.
[23] Kaj Madsen, Hans Bruun Nielsen, and Ole Tingleff. Methods for
Non-Linear Least Squares Problems (2nd ed.). 2004. PDF avail-
able at http://www2.imm.dtu.dk/pubdb/views/edoc_download.
php/3215/pdf/imm3215.pdf (2020/02/11).
[24] Kenneth Levenberg. A method for the solution of certain non-linear problems in
least squares. Quarterly of Applied Mathematics, 2(2):164–168, jul 1944.
BIBLIOGRAPHY 89
[25] Donald W. Marquardt. An algorithm for least-squares estimation of nonlinear pa-

rameters. Journal of the Society for Industrial and Applied Mathematics, 11(2):431–
441, 1963.
[26] C.T. Kelley. Iterative Methods for Optimization. Frontiers in Applied Mathematics.
Society for Industrial and Applied Mathematics, 1999. PDF available at https://
archive.siam.org/books/textbooks/fr18_book.pdf (2020/01/17).
[27] Henri P. Gavin. The levenberg-marquardt method for nonlinear least squares curve-
fitting problems. Available at http://people.duke.edu/˜hpgavin/
ce281/lm.pdf (2019/12/06), 2013.
[28] John A. Nelder and Roger Mead. A simplex method for function minimization.
Computer Journal, 7:308–313, 1965.
[29] Jeffrey Lagarias, James Reeds, Margaret Wright, and Paul Wright. Convergence
properties of the nelder–mead simplex method in low dimensions. SIAM Journal on
Optimization, 9:112–147, 12 1998.
[30] G.H. Golub and Victor Pereyra. The differentiation of pseudo-inverses and nonlin-
ear least squares problems whose variables separate. SIAM Journal on Numerical
Analysis, 10:413–432, 04 1973.
[31] Gene Golub and Victor Pereyra. Separable nonlinear least squares: the variable
projection method and its applications. Inverse Problems, 19:R1–R26(1), 01 2003.
[32] Dianne P. O’Leary and Bert W. Rust. Variable projection for nonlinear least squares
problems. Computational Optimization and Applications, 54(3):579–593, Apr 2013.
[33] J. Bruls, C.T. Chou, B.R.J. Haverkamp, and M. Verhaegen. Linear and non-linear
system identification using separable least-squares. European Journal of Control,
5(1):116 – 128, 1999.
[34] Harold T. Davis. Introduction to Nonlinear Differential and Integral Equations.

Dover, New York, 1962.
[35] H.K. Khalil. Nonlinear Control, Global Edition. Pearson Education Limited, 2015.
[36] B. Armstrong-Hélouvry. Control of Machines with Friction. The Springer Interna-

tional Series in Engineering and Computer Science. Springer US, 1991.
[37] V. van Geven. A study of friction models and friction compensation. Available at
http://www.mate.tue.nl/mate/pdfs/11194.pdf (2019/12/18), 2009.
traineeship report, Technical University Eindhoven, Department Mechanical Engi-
neering, Dynamics and Control Technology Group.
BIBLIOGRAPHY 90
[38] Andreas Krämer and Joachim Kempkes. Modellierung und simulation von nicht-
linearen reibungseffekten bei der lageregelung von servomotoren. FHWS Science
Journal, 1(2):47 – 57, 2013.
[39] Carlos Canudas de Wit, Henrik Olsson, Karl Johan Åström, and Pablo Lischinsky.
A new model for control of systems with friction. IRE Transactions on Automatic
Control, 40(3), 1995.
[40] T. Piatkowski. Dahl and lugre dynamic friction models — the analysis of selected
properties. Mechanism and Machine Theory, 73:91 – 100, 2014.

Ac16078457 1

Uploaded by

Copyright:

Available Formats

Ac16078457 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ac16078457 1

Uploaded by

Copyright:

Available Formats

Chair of Automation

System Identification of Nonlinear

Stefan Oliver Fladischer, BSc

System identiﬁcation is the experimental modeling of a dynamic system whose parame-

Systemidentiﬁkation ist die experimentelle Modellierung eines dynamischen Systems,

2 System Identiﬁcation Theory 7

3 Numerical Solution Methods 16

3.1 Fitting a polynomial to noisy data. . . . . . . . . . . . . . . . . . . . . . 21

4.10 Expected standard deviation due to disturbance induced variation of the

1.1 Problem Formulation

1.2 Systems and Models

y(t) = f (u(t)). (1.1)

y(t) = g(x(t), u(t),t). (1.3)

1.3 System Identiﬁcation and its Place in Control Theory

1.4 Why are Linear Models Preferred over Nonlinear Mod-

ẋ(t) = A(t)x(t) + B(t)u(t) (1.4)

compared to a general nonlinear model

ẋ(t) = f (x(t), u(t),t) (1.5)

as algebraic equations in the complex s domain:

Y (s) = H(s)U(s). (1.9)

System Identiﬁcation Theory

2.1 System Identiﬁcation as a Part of System Modeling

ẋ(t) = A(ψ)x(t) + B(ψ)u(t) (2.1)

for a linear time invariant (LTI) system, or as

ẋ(t) = f (x(t), u(t), ψ,t) (2.3)

2.2 The System Identiﬁcation Procedure

2.3 Building a Design Model

2.3.1 Black-Box Models

2.3.2 White- and Grey-Box Models

2.4 Experimental Design

2.5 The Goodness-Of-Fit Measure

2.5.1 Vertical Distance Least Squares Fit of the Model Output

ψ = argmin ε(ψ). (2.5)

2.5.2 Utilization of Multiple Data Sets

εA (ψ) = ξA − y(ψ)22 and (2.9)

2.5.3 Weighting of the Model Output

ε(ψ) = W (ξ − y(ψ))22 . (2.14)

2.5.4 Model Uncertainty

ξ = yexct + δ = y(ψ ) (2.16)

The important insight to take from this is that

ε = ε(ξ, ψ) = ε(yexct + δ, ψ) and therefore (2.17)

Inﬂuence of the Numerical Optimization

Numerical Solution Methods

3.1 Numerical Solution to the IVP

x(t + Δt) ≈ x(t) + Δtẋ(t) (3.2)

x(t + Δt) ≈ x(t) + Δtẋ(t + Δt) (3.4)

x(t + Δt) − x(t)

and equation (3.4) corresponds to a backwards ﬁnite difference formula:

x(t + Δt) − x(t)

3.1.2 Higher Order Numerical Integration Methods

k1 = Δtf (x(t),t) (3.9)

The approximation of x(t + Δt) is computed with w1 = w4 = 1

The MATLAB® function ode45() is an implementation of a Runge Kutta Dormand

3.2 Numerical Optimization

3.2.1 Topology of the Objective Function

In Linear Curve Fitting

The same functional can also be formulated as

ε(ψ) = [ξ − y(ψ)]T [ξ − y(ψ)] (3.18)

and the vectorized y(xi , ψ)

εA (ψ) = ξA − y(ψ)22 and (2.9)

ε(ψ) = W (ξ − y(ψ))22 . (2.14)

ξ = yexct + δ = y(ψ ) (2.16)

ε(Δψ) = ξ − y(ψ0 ) − J(ψ0 )Δψ22 . (3.48)

ε(Δψ) = ξ − y(ψ0 ) − J(ψ0 )Δψ22 + λ Δψ22 . (3.52)

ε(ψ) = ξ − y(ψ)22 . (3.78)

ε(ξ, ψ) = ξ − y(ψ)22 . (4.21)

ψA = argmin εA (ψ) = argmin ξA − y(ψ)22 (4.26)

εB (ψ) = ξB − y(ψ)22 with (4.29)