Nonlinear Systems and Control Spring 2015

1 Optimal Control
We consider first simple unconstrained optimization problems, and then
demonstrate how this may be generalized to handle constrained optimiza-
tion problems. With the observation that an optimal control problem is a
form of constrained optimization problem, variational methods are used to
derive an optimal controller, which embodies Pontryagins Minimum Princi-
ple. Subsequently an alternative approach, based on Bellmanss Principle of
Optimality, and Dynamic programming is used to derive the Hamilton-Jacobi
equations.

1.1 Unconstrained Optimization

Consider a function

L:<<
We want to find
min L(u)
u

Let us assume that L is sufficiently smooth, and consider the Taylor expan-
sion:

dL dL2
L(u) = L(u0 ) + |u=u0 (u u0 ) + 2 |u=u0 (u u0 )2 + ..
du du
Then we have a necessary condition
dL
|u=u0 = 0
du
and a sufficient condition

dL2
|u=u0 > 0
du2
Note that these are only conditions for a local minimum. Additional condi-
tions are required to find the global minimum if the function is non-convex.
If we have a function with more than one variable, that is L : <n < we
have the following conditions
L L L
{ , , ... }|u=u0 = 0
u1 u2 un

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 1 of 18

Nonlinear Systems and Control Spring 2015

and
2L 2L

u21
... u1 un
2L
|u=u0 = ... ... ... |u=u0 > 0

u 2
2L 2L
un u1
... u2n

i.e. is positive definite.

Example 1 Find the minimum of the function

f (x1 , x2 ) = 6x21 + 2x22 + 3x1 x2 8x1 + 3x2 4.

Necessary condition  
f 12x1 + 3x2 8
= =0
u 4x2 + 3x1 + 3
41 20
Solving these equations we find x0 = ( 39 , 13 ) and when we insert x0 into
the Hessian Matrix we see that

2L
 
12 3
(x0 ) = .
u2 3 4
the resulting matrix is positive definite. We conclude that the point x0 is a
minimum.

The plot in Figure 1 shows the function for which we are finding the mini-
mum.

1.2 Constrained Optimization

Theorem 1 Consider the problem

minx L(x), f (x) = 0

with L : <n < and f : <n <m . Then this problem is equivalent to

min L(x) + f (x).

x,

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 2 of 18

Nonlinear Systems and Control Spring 2015

Figure 1: Convex Function

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 3 of 18

Nonlinear Systems and Control Spring 2015

The function
H(x, ) = L(x) + f (x)
is called the Hamiltonian of the optimization problem. The coefficients
<m are called Lagrange Multipliers of the system.
Proof
Without loss of generality we consider x <2 . The necessary conditions for
a minimum are
H
=0
x,
or
H L f
= +
x1 x1 x1
H L f
= +
x2 x2 x2
H
= f (x1 , x2 )

The third condition is equivalent to the boundary conditions of the original
problem being satisfied.
The first two conditions are equivalent to saying that the vectors
!
 L  f
x1 x1
L ; f
x2 x2

are parallel or colinear. If these vectors are parallel, then the matrix
!
L f
x1 x1
L f
x2 x2

has rank less than 2, which means that the linear system obtained by equating
to zero the derivative of the Hamiltonian has a non trivial solution on .
With the help of a diagram in Figure 2, it is easy to understand that where
we have a minimum or maximum the two gradients (with the red vector
representing the gradient of L and the black vector representing the gradient
of f ) have to be parallel, as otherwise one can increase or decrease the value
of L while satisfying the constraint f (x) = 0.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 4 of 18

Nonlinear Systems and Control Spring 2015

Figure 2: Gradients of L and f must be colinear at the extrema

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 5 of 18

Nonlinear Systems and Control Spring 2015

Example 2 Find the minimum of the function

L(x1 , x2 ) = x21 4x1 + x22
with the constraints
x1 + x2 = 0.
Indeed, the Hamiltonian is given by
H(x, ) = x21 4x1 + x22 + (x1 + x2 ).
Thus, the expressions for
H
=0
x,
are
H
0= = 2x1 4 +
x1
H
0= = 2x2 +
x2
H
0= = x1 + x2

Solving this system of linear equations we find the solution (x1 , x2 , ) =
(1, 1, 2). This solution is only a candidate for solution. One must now
test whether these coordinates represented the solution we seek.
Example 3 Consider the problem
minx1 ,x2 ,x3 L(x1 , x2 , x3 ) = x21 + x22 + x23 ,
such that
x1 x2 + x2 x3 + x3 x1 = 1.
As per recipe we write the Hamiltonian.
H(x, ) = x21 + x22 + x23 + (x1 x2 + x2 x3 + x3 x1 1).
Taking the derivatives we see
0 = 2x1 + x2 + x3
0 = 2x2 + x1 + x3
0 = 2x3 + x1 + x2
0 = x1 x2 + x2 x3 + x3 x1 1

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 6 of 18

Nonlinear Systems and Control Spring 2015

From the first three equations we see that

x = y = z.

This result plus the last equation tell us that

r
1
x=y=z= , = 1.
3
Again, we must make sure that this candidate is indeed a solution to our
original problem.

1.3 Pontryagins Minimum Principle

Consider the system
x = f (x, u), x(0) = x0 (1)
with associated performance index
Z T
J[x0 , u()] = (x(T )) + L(x(t), u(t))dt (2)
0

and final state constraint

(x(T )) = 0 (3)
The following terminology is customary:

1. J[x0 , u()] is called cost function.

2. (x(T )) is called end constraint penalty

3. L(x, u) is called running cost

4. H(x, u, ) = L(x, u) + f (x, u) is called Hamiltonian.

The Optimal Control Problem is: Find the control function

u : [0, T ] 7 <m

such that the performance index is minimized and the final state constraint
and the system equations are satisfied.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 7 of 18

Nonlinear Systems and Control Spring 2015

Theorem 2 Solutions of the Optimal Control Problem also solve the follow-
ing set of differential equations:

State Equation:
x = H = f (x), (4)
L f
Co-State Equation: = Hx = + (5)
x x
L f
Optimality Condition: 0 = Hu = + (6)
u u
State initial condition: x(0) = x0 (7)
Co-state final condition: (T ) = (x + x )|x(T ) (8)

where is the Lagrange multiplier corresponding to end condition given by

Equation 3.

Proof We use the Lagrange multipliers to eliminate the constraints. Since

the main constraints are now given by a dynamical system x = f (x, u) it is
clear that they must hold for all t and thus the vector function : [0, T ] <n
is a function of time.
Using the notation H(x, u, ) = L(x, u) + f (x, u), the unconstrained
minimization problem can be written as
Z T
J(x, u, ) = (x(T )) + (x(T )) + [L(x, u) + (f (x, u) x)]dt

0
Z T
= (x(T )) + (x(T )) + [H(x, u, ) x)]dt.

0

The differentials are written as

J = [x (x) + x (x)]x)|x=x(T ) +
Z T
[Hx x + Hu u + H x + x]dt+

0
(x)|x(T )

Note that by integrating by parts:

Z T Z T
x = x|t=T + x|t=0 +
xdt
0 0

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 8 of 18

Nonlinear Systems and Control Spring 2015

Furthermore, since x(0) = x(0) is constant, it holds that x|t=0 = 0. So we

can rewrite the previous expression as

J = [x (x) + x (x) ]x)|x=x(T ) +

Z T

[(Hx + )x + (H x)
+ Hu u]dt+
0
(x)|x(T )

Now for the function u : [0, T ] <m to minimize the cost function, J must
be zero for any value of the differentials. Thus, all the expressions before the
differentials have to be zero for every t [0, T ]. This observation gives the
equations as required.

Remark 1 Just as in the static optimization case, where the zeros of the
derivatives represent candidates to be tested for extremum, the solutions of
the system described in Theorem 2 are to be seen as candidates to be the
optimal solution and their optimality must be tested for each particular case.
In other words, the Pontriaguin Maximum Principle delivers necessary, but
not sufficient, conditions for optimality.

Remark 2 The system of equations in Theorem 2 is a so called Two Point

Boundary Value problem. generally speaking, these problems are only solv-
able by dedicated software implementing suitable numerical methods.

Remark 3 The expression regarding u() in Equation 6 is a special case of

a more general minimum condition. In general we must seek

u = arg min H(x, u, )

for given values of x(t) and u(t). In other words, the Pontryagins Minimum
Principle states that the Hamiltonian is minimized over all admissible u for
optimal values of the state and co-state.

Remark 4 Special attention is needed in the case where Hu = const for all
u, in which case the solution is found where the constraints are active. This
sort of solution is called Bang-Bang solution.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 9 of 18

Nonlinear Systems and Control Spring 2015

Example 4 Consider the minimization problem

Z T
J[x0 , u()] = 0.5 u2 (t)dt
0

subject to

x1 = x2 , x1 (0) = x10
x2 = u, x2 (0) = x20

and with final constraint

(x(T )) = x(T ) = 0.

Applying Theroem 2, we transform the system into

x1 = H1 = x2 ; x1 (0) = x10
x2 = H2 = u; x2 (0) = x20
1 = Hx1 = 0; 1 (T ) = 1
2 = Hx2 = 1 ; 2 (T ) = 2

where
H(x, u, ) = 0.5u2 + 1 x2 + 2 u
Now we see that
Hu = u + 2 = 0
Thus, we can solve the differential equations and see that

1 (t) = 1
2 (t) = 1 (t T ) + 2
u(t) = 1 (t T ) 2

Placing these linear expressions in the dynamic equations for x and using
the initial conditions x(0) = x0 , we obtain a linear system of equations with
respect to (1 , 2 ), which gives us the final parametrization of the control law
u. Figure 3 shows the result of applying this control law with T = 15.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 10 of 18

Nonlinear Systems and Control Spring 2015

Figure 3: Trajectories for Minimal Energy Case. Arrival time T = 15.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 11 of 18

Nonlinear Systems and Control Spring 2015

Example 5 Consider the same problem as in Example 4, but with a perfor-

mance reflecting minimal time, i.e.
Z T
J[x0 , u()] = 1 dt
0

and constrained input

1 u(t) 1 t 0.

Now
H(x, u, ) = 1 + 1 x2 + 2 u.
We notice that Hu = 2 , i.e. Hu does not depend on u and thus the extremum
is to be reached at the boundaries, i.e. u (t) = 1, t 0. Using

u = arg min H(u) = arg min 2 (t)u(t)

u u

we see that
u (t) = sign2 (t).
Figure 4 depicts the result of deploying this control. Note that the control law
is discontinuous. Further, we observe that the system now needs less time to
reach the origin than in the previous Minimal Energy example. Of course,
this success is obtained at the cost of deploying more actuator energy.

1.4 Dynamic Programming

An alternative approach to solving optimal control problems to the varia-
tional methods above is given by Dynamic programming , this is based
on the following observation.

Remark 5 Bellmans principle of optimality: An optimal policy (control

law) has the property that no matter what the previous decisions (control)
have been, the remaining decisions must constitute an optimal policy with
respect to the state resulting from these decisions (the current state). This
implies that the optimal policy is computed backwards, in that we can split
the optimal problem into 2 optimal control sub-problems, corresponding to 2
time intervals.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 12 of 18

Nonlinear Systems and Control Spring 2015

Figure 4: Trajectories for Minimal Time Case. Arrival time T 13.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 13 of 18

Nonlinear Systems and Control Spring 2015

Figure 5: Dynamic Programming Route

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 14 of 18

Nonlinear Systems and Control Spring 2015

Figure 6: Dynamic Programming Optimal Route

Example 6 (Shortest Path Problem) Consider the problem of travelling

from A to B with the minimum total cost. The graph to be traversed, and the
associated edge costs are given in the following diagram
The approach taken is to work backwards from B noting the minimum
cost from there to B:
This results in the solution depicted in Figure6.
For the solution of an optimal control problem, we split the overall time
interval into two shorter time intervals. The solution of the first interval is
dependent on the solution of the second interval, as follows:
Re-write the optimal control problem by splitting the integral at
Z t Z T
J(t0 ) = (x(T ), T ) + L(x(, u( ))d + L(x(, u( ))d
t0 t

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 15 of 18

Nonlinear Systems and Control Spring 2015

Figure 7: Dynamic Programming Route Segment

This gives the 2 sub-problems:

Z t
J1 (t0 ) = V (x(t), t) + L(x(, u( ))d
t0
Z T
J2 (t) = (x(T ), T ) + L(x(, u( ))d
t

where we define the function

V (x(t), t) = J2 (t)

Note that in the limit as t 7 T , we find that J1 = J and J2 = (x(T ), T ).

Alternatively, as t 7 t0 , J1 = V and J2 = J.

Remark 6 The principle of optimality may now be restated as follows: let

u? be the optimal control for the original control problem, for all t [t0 , T ]
u? gives the optimal control for the control problems defined by J1 and J2 .

Definition 1 The function V (x, t) is called the value function or the cost
to go. It represents the value of the solution of the optimal control problem
starting at x at the time t.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 16 of 18

Nonlinear Systems and Control Spring 2015

Consider the minimization of a value function V (x, t) during t, T ]. Let

?
u be the optimal control for T = tf . Then
Z tf
? ?
V (x (t), t) = (x (tf ), tf ) + L(x? ( ), u? ( ))d
t
Z t
= min{(x(tf ), tf ) L(x( ), u( ))}
u tf

Additionally, we have to fulfill

x = f (x(t), u(t))

and the boundary condition

V (x, T ) = (x(tf ), tf )

It follows that
dV
= min L(x(t), u(t))
dt u
V V dx
+ = min L(x(t), u(t))
t x dt u
V V
= min{ f (x(t), u(t)) L(x(t), u(t))}
t u x
V
= min H(x, , u)
u x
Definition 2 This equation is known as the Hamilton-Jacobi equation.
This is a partial differential equation in V , with boundary condition V (x, T ) =
(x(tf ), tf ). Solving this equation gives the solution of the optimal control
problem with the control being a state feedback law given by
V
u? = arg min H(x, , u)
x
Note that the co-state introduced in the previous section is then given
V
by x
.
When the Hamilton-Jacobi-Bellman equation is to be solved in practical
cases, numerical methods seem appropriate. However, the solution of the
Hamilton-Jacobi equation by numerical methods is only tractable for low
dimensional systems. For higher dimensional systems, the number of data

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 17 of 18

Nonlinear Systems and Control Spring 2015

points required increases with the power of the dimension of the system (space
and time discretization). For example, for a nonlinear system of dimension
n where the value function is to be determined on a hyper-cube [a, a] with
n
a spatial discretization , a total of 2a

points will have to be stored. For
even modest requirements the amount of data becomes very large and not
tractable.

Lecture 6: Optimal Control: Pontriaguin and Dynamic Programming 18 of 18