ARTICLE IN PRESS

Fuzzy Sets and Systems ( )

www.elsevier.com/locate/fss
A reinforcement learning adaptive fuzzy controller for robots
Chuan-Kai Lin

Department of Electrical Engineering, Chinese Navel Academy, 669 Chun Hsiao Road, Kaohsiung 813, Taiwan
Received 2 January 2001; received in revised form 27 November 2001; accepted 5 June 2002
Abstract
In this paper, a new reinforcement learning scheme is developed for a class of serial-link robot arms. Tradi-
tional reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error
interactions with a dynamic environment. In the proposed reinforcement learning scheme, an agent is employed
to collect signals from a xed gain controller, an adaptive critic element and a fuzzy action-generating element.
The action generating element is a fuzzy approximator with a set of tunable parameters, and the performance
measurement mechanism sends an error metric to the adaptive critic element for generating and transferring
a reinforcement learning signal to the agent. Moreover, a tuning algorithm of the proposed scheme that can
guarantee both tracking performance and stability is derived from the Lyapunov stability theory. Therefore, a
combination of adaptive fuzzy control and reinforcement learning scheme is also concerned with algorithms
for eliminating a sequence of decisions from experience. Simulations of the proposed reinforcement adaptive
fuzzy control scheme on the cart-pole balancing problem and a two-degree-of freedom (2DOF) manipulator,
SCARA robot arm verify the eectiveness of our approach.
c 2002 Elsevier Science B.V. All rights reserved.
Keywords: Reinforcement learning; Adaptive fuzzy controller
1. Introduction
The reinforcement learning systems has enjoyed considerable success in application to nonlinear
control 1,13,3. Most of the successful results, however, are obtained when the desired trajectory
is set-point. The performance of the reinforcement learning systems for tracking control is less
satisfactory. Recently, reinforcement learning systems used parameterized function approximators
such as neural networks 13 in order to generalize between similar situations and actions. In these
cases there are no strong theoretical results on the accuracy of convergence and stability. Although
the learning speed can be improved by temporal dierence learning 16 and Q-learning methods 14,
the tracking performance and the stability cannot be guaranteed.

Tel.: 886-7-5834700 ext. 1312; fax: 886-7-5829681.

E-mail address: cklin@mail.cna.edu.tw (Chuan-Kai Lin).
0165-0114/02/$ - see front matter c 2002 Elsevier Science B.V. All rights reserved.
PII: S0165- 0114(02)00299- 3
2 Chuan-Kai Lin / Fuzzy Sets and Systems ( )
ARTICLE IN PRESS
Function approximation, which can be implemented by neural networks and fuzzy systems, is es-
sential to reinforcement learning 15. In 5, the proposed fuzzy actor-critic learning (FACL) and fuzzy
Q-learning (FQL) are reinforcement learning methods with function approximation implemented by
fuzzy inference systems. However, Both FACL and FQL based on Dynamic Programming principles
cannot guarantee the system stability and performance. Furthermore, in spite of the optimal policy
15,9 the stability is a severe topic when the reinforcement learning controller is applied to the real
system. Therefore, it needs control theory to provide a mathematical framework in which the design
of reinforcement learning controller can be formulated. The linear quadratic regulation (LQR) can
be combined with neural Q-learning to control nonlinear systems 4; however, LQR is not suitable
for tracking problems.
For reinforcement learning systems, an agent must learn behavior through trial-and-error inter-
actions with a dynamic environment 6. Adaptive control and fuzzy self-organizing control 10 are
also concerned with algorithms for improving a sequence of decisions from experience. The pro-
posed reinforcement learning systems combined with adaptive control and fuzzy control will be
capable of avoiding too many trial-and-error interactions according to the stability and perfor-
mance of reinforcement learning systems can be resorted to the powerful mathematical analysis
of adaptive control algorithms. The architecture of the proposed reinforcement learning systems
is similar to the fuzzy self-organizing control, which has a critic element, performance measure-
ment, fuzzy controller and an agent. For guaranteeing the stability and performance, the feed-
back linearization control law will introduce additional elements and the adaptation law of the
parameters of adaptive critic element and fuzzy approximator will be derived from Lyapunov
theory.
This paper consists of ve sections. The background for development of the reinforcement learn-
ing systems is provided in Section 2, based largely on the system presented in 1. In Section 3, the
new scheme of the reinforcement learning systems and the corresponding adaptation laws are intro-
duced. Stability of the closed-loop system is analyzed and the tracking performance is guaranteed.
Section 4 illustrates the operation of the proposed reinforcement learning systems using SCARA
robot arm simulations for demonstrating the eectiveness. The last section, Section 5, gives the
conclusions.
2. Preliminaries
2.1. Mathematical notation
The trace of a square matrix S=[s
i)
]R
nn
, tr(S), is dened as the sum of diagonal elements
of S. By the denition, it is obvious that the trace of a matrix and the trace of the transpose of a
matrix are the same, that is, tr(S)=tr(S
T
). Given BR
mn
and CR
nm
, it can be easily proved
that
tr(BC)=tr(CB). (2.1)
The norm of a vector and the Frobenius norm used in the following sections are dened previously.
Let R denote the real numbers, R
n
denote the real n-vectors, and R
mn
be the real mn matrices.
ARTICLE IN PRESS
Chuan-Kai Lin / Fuzzy Sets and Systems ( ) 3
The norm of a vector is dened as
x =
_
n

i=1
x
2
i
_
1}2
(2.2)
and the Frobenius norm of a matrix A=[a
i)
]R
nn
is dened by
A
F
= [tr(A
T
A)]
1}2
=

i,)
a
2
i)
. (2.3)
The Frobenius norm is also compatible with the two-norm so that Ax6A
F
x.
2.2. Stability
Consider the nonlinear dynamical system given by x=f (x, u, t), y=g(x, t), where x(t)R
n
, u(t)
is the input vector and y(t) is the output. If there exists a compact set JR
n
such that for all
x(t
0
)=x
0
J, there exists an c0 and a number 1(c, x
0
) such that x(t)c for all tt
0
+1, then
the solution is uniformly ultimately bounded (UUB) 8.
2.3. Fuzzy systems
General fuzzy systems consist of fuzzication unit, fuzzy rule base, inference engine and defuzzi-
cation unit. The fuzzy system is a nonlinear system performing a real (nonfuzzy) mapping from
UR
n
to R
m
. Inside the fuzzy system, fuzzier maps nonfuzzy input space to fuzzy sets and the
defuzzier works in the opposite way to map from fuzzy sets to nonfuzzy output space. Inference
engine is the brain of the fuzzy system deciding how to use the fuzzy rule base. As to the rule base,
it determines the computation accuracy, memory storage and computational eciency. Therefore, the
main part of fuzzy systems is the fuzzy rule base, which may be expressed in the form of linguistic
IF-THEN rules from experts. Assume the fuzzy rule consists of N rules as follows:
R
)
()th rule): If x
1
is A
)1
and x
2
is A
)2
and . . . and x
n
is A
)n
then ,
1
is B
)1
and ,
2
is B
)2
and . . .
and ,
m
is B
)m
, where ) =1, 2, . . . , N, x
i
(i =1, 2, . . . , n) are the input variables to the fuzzy system,
,
k
(k =1, 2, . . . , m) are the output variable of the fuzzy system, and A
)i
and B
)k
are linguistic terms
characterized by their corresponding fuzzy membership functions j
A
)i
(x
i
) and j
B
)k
(,
k
), respectively.
Each rule R
)
can be viewed as a fuzzy implication.
The fuzzy logic system with center-average defuzzier, product inference, and singleton fuzzier
is of the following form:
,
k
=

N
)=1
c
)k
(

n
i=1
j
A
)i
(x
i
))

N
)=1
(

n
i=1
j
A
)i
(x
i
))
, k = 1, . . . , m, (2.4)
where j
B
)k
(c
)k
)=1. Eq. (2.4) can be written as sum of ring strength of rules:
,
k
=
N

)=1
c
)k
[
)
, k = 1, . . . , m, (2.5)
4 Chuan-Kai Lin / Fuzzy Sets and Systems ( )
ARTICLE IN PRESS
where the ring strength [
)
=

n
i=1
j
A
)i
(x
i
)}

N
)=1
(

n
i=1
j
A
)i
(x
i
)). The shape of the membership func-
tion can be chosen as triangle, bell (gaussian function). Since Wang 17 proved that the fuzzy system
can be viewed as a universal approximator to any arbitrary accuracy, fuzzy control law has been
transformed to be linear feedback control combined with fuzzy systems approximating nonlinear
part. Hence, by Lyapunov theory and other traditional control theories, the stability and system
performance can be analyzed and guaranteed, respectively.
2.4. Dynamics of robot manipulators
Consider a rigid robot manipulator with n serial links described by the equation
M(V)

V +!
m
(V,

V)

V +G(V) +F

V +t
d
= t (2.6)
with vector VR
n
being the joint position vector; M(V)R
nn
being a symmetric positive denite
inertia matrix; !
m
(V,

V)

V being a vector of Coriolis and centripetal torques; G(V)R
n
representing
the gravitational torques; F=K
c
+ !
[
R
nn
being a diagonal matrix consisting of the back emf
coecient matrix K
c
and the viscous friction coecients matrix !
[
; t
d
R
n1
being the unmodeled
disturbances vector; and tR
n1
being the vector of control input torques. The structural properties
of the robot manipulator are hold such as skew-symmetry of matrix

M 2!
m
and boundedness of
M(V), !
m
(V,

V) and t
d
.
3. Controller design
Based on the architecture of reinforcement learning control systems 1, we employ the fuzzy system
corresponding to the associate search element (ASE) and modify the adaptive critic element (ACE).
In the spirit of actor-critic reinforcement learning control, the weights of fuzzy approximator (ASE)
will be tuned by the signal from ACE. Fig. 1 shows the architecture of the proposed reinforcement
learning adaptive fuzzy controller (RLAFC) and the details of the overall system will be subsequently
derived.
As in 1, the action-generating element is a fuzzy logic system tuned by the ACE, however, the
fuzzy system is used to approximate the nonlinear part of the nonlinear system instead of generating
control input directly for guaranteeing stability and performance. The behavior of the RLAFC is
that the performance measurement unit measures the system states to generate an error metric signal
vector s providing for the ACE to tune the fuzzy system. The ACE collecting the error metric signal
generates a reinforcement signal vector r to tune the fuzzy system. In the following, the control law,
weight updating law and stability analysis will be discussed.
In practical robotic systems, the load may vary while performing dierent tasks, the friction
coecients may change in dierent congurations and some neglected nonlinearities as backlash
may appear as disturbances at control inputs, that is, the robot manipulator may receive unpredictable
interference from the environment where it resides. Therefore, the control objective is to design a
robust RLAFC so that the movement of robot arms follow the desired trajectory and all signals in
the closed-loop system are bounded even when exogenous and endogenous perturbations are present.
ARTICLE IN PRESS
Chuan-Kai Lin / Fuzzy Sets and Systems ( ) 5
Robot

N
Fuzzy
Rule
Base
C

1
ACE
W
x

+
Performance
measurement
K
Robust
term
d


d

C
C

x
1
x
n
x
Norm
||s||
W
s
r
W
N
o
r
m
a
l
i
z
a
t
i
o
n
^
^
^
^
^
^
^
Fig. 1. The diagram of closed-loop system.
Denote the tracking error vector e(t) and error metric s(t) as
e(t) = V
d
(t) V(t),
s(t) = e(t) +Ae(t), (3.1)
where V
d
(t), is the desired robot manipulator trajectory vector and A=A
T
0. It is typical to dene
an error metric s(t) to be a performance measure. When the error metric s(t) is smaller, the system
performance is better. Therefore, dierentiating s(t) and using (3.1), the dynamics of robot arms can
be expressed in terms of s(t) as
M s = !
m
s +f +t
d
t, (3.2)
where the unknown function f is given by
f = M(V)(

V
T
d
+A e) +!
m
(V,

V)(

V
T
d
+Ae) +G(V) +F(

V) (3.3)
and the disturbance t
d
(t) is assumed to be bounded by b
d
. It is also assumed that the complex
nonlinear function f can be represented by an ideal fuzzy approximator C
T
(x) as follows:
f (x) = C
T
(x) +(x), (3.4)
where c(x) is the approximation error, C=[c
)k
]R
Nm
is the ideal weight matrix, (x)=[[
1
[
2
. . .
[
N
]
T
, and x(t) is given by
x(t) = [

V
T
d

V
T
d

V
T
V
T
]
T
. (3.5)
6 Chuan-Kai Lin / Fuzzy Sets and Systems ( )
ARTICLE IN PRESS
As we know, the function can be estimated by a fuzzy approximator

f (x) =

C
T
(x). (3.6)
Dene the control law as
t = Ks +

f +

d, (3.7)
where the positive matrix K=K
T
0, the output vector of the fuzzy approximator

f estimates f
and

d is the robustifying term to attenuate disturbances. The architecture of the closed-loop system
is shown in Fig. 1. Using the control input (3.7), we get closed-loop dynamics as
M s = (K +!
m
)s +

f +t
d


d, (3.8)
where the approximation error

f is denoted as

f = f

f =

C
T
(x) +(x) (3.9)
with

C=C

C and the robustifying term is given by

d(t) = k
d
s(t)}s(t) (3.10)
with k
d
b
t
. Therefore, the rewritten closed-loop error dynamics (3.8) implies that the overall system
is driven by the functional estimation error

f . From Fig. 1, we can obtain the reinforcement signal
from ACE as
r(t) = s(t) +s

R
T
. (3.11)
The reinforcement signal is used to update the weight matrix

C. Before the stability analysis,
there are some assumptions that should be made.
Assumption 1. The norms of optimal weights, R and C, are bounded by known positive
real values, i.e., R6W
m
and C6C
m
with some known W
m
and C
m
(the norm is Frobenius
norm).
Assumption 2. The approximation errors and disturbances are bounded, i.e., specied b
c
and b
t
satisfying 6b
c
and t
d
6b
t
, respectively.
Theorem 1. Let the desired trajectory V
d
be bounded. Suppose that Assumption 1 and 2 are hold.
If the weight tuning laws provided for the fuzzy approximator and ACE are as follows:

R = K
W
s(

C
T
)
T
pK
W
s

R, (3.12)

C = K
C
r
T
pK
C
s

C, (3.13)
ARTICLE IN PRESS
Chuan-Kai Lin / Fuzzy Sets and Systems ( ) 7
where constant matrices K
W
and K
C
are positive and diagonal, p
1
and p
2
are positive constants,
and the reinforcement signal r(t) is provided by (3.11), then the control input t(t) provided by
(3.7) guarantees that the error metric s(t),

R and

C are uniformly ultimately bounded.
Proof. Dene the Lyapunov function candidate
J(t) =
1
2
s
T
Ms +
1
2
tr{

C
T
K
1
C

C} +
1
2
tr{

R
T
K
1
W

R} (3.14)
with

R=R

R. Evaluating the time derivative of J(t) along the trajectories of the weight tuning
laws (3.12) and (3.13) yield

J(t) =
1
2
s
T

Ms +s
T
M s tr{

C
T
K
1
C

C} tr{

R
T
K
1
W

R}. (3.15)
Substituting (3.8), and (3.10)(3.13) into (3.15) we get

J(t) 6s
T
Ks +s
T
+str{

C
T
(

R
T
)
T
+ p

C
T

C +

R
T
(

C
T
)
T
+ p

R
T

R} (3.16)
by applying the property of skew-symmetry of matrix

M 2!
m
and the assumption k
d
b
t
. After
some manipulation, we can obtain the following:

J(t) 6s
T
Ks +s
T
+str{

C
T
(R
T
)
T
+ p

C
T

C +

R
T
(C
T
)
T
+ p

R
T

R}. (3.17)
Applying the matrix inequalities

C
T

C6

CC
m


C
2
and

R
T

R6

RW
m


R
2
, (3.17)
becomes

J(t) 6s
T
Ks +sb
c
+s{(W
m

2
+ C
m
)

C + (C
m

2
+ W
m
)

R
p

C
2
p

R
2
}. (3.18)
Furthermore, since
2
6N, there results

J(t) 6K
min
s
2
ps
_
_

C
NW
m
+ C
m
2p
_
2
+
_

R
NC
m
+ W
m
2p
_
2

_
_
NW
m
+ C
m
2p
_
2
+
_
NC
m
+ W
m
2p
_
2
+
b
c
p
__
, (3.19)
where K
min
is the minimum singular value of K. Thus,

J is negative as long as
s p
_
_
NW
m
+ C
m
2p
_
2
+
_
NC
m
+ W
m
2p
_
2
+
b
c
p
__
K
min
= o
s
or (3.20)
8 Chuan-Kai Lin / Fuzzy Sets and Systems ( )
ARTICLE IN PRESS

C
NW
m
+ C
m
2p
+

_
NW
m
+ C
m
2p
_
2
+
_
NC
m
+ W
m
2p
_
2
+
b
c
p
= o
C
or (3.21)

R
NC
m
+ W
m
2p
+

_
NW
m
+ C
m
2p
_
2
+
_
NC
m
+ W
m
2p
_
2
+
b
c
p
= o
W
. (3.22)
Clearly, from inequalities (3.20)(3.22), we can dene compact set D: D={(s,

C,

R) |
06s6o
s
and

C6o
C
and

R6o
W
}. By Lyapunov theory, for s(t
0
),

C(t
0
) and

R(t
0
)D,
there exists numbers 1(o
s
, o
C
, o
W
, s(t
0
),

C(t
0
),

R(t
0
)) such that 06s6o
s
,

C6o
C
and

R6o
W
for all tt
0
+ 1. In other words,

J(s,

C,

R) is negative outside the compact set D and then s,

C
and

R are UUB.
Remark 1. The updating law for weights of ASE (fuzzy approximator) is mainly determined by a
reinforcement signal with magnitude modication (the rst term of (3.13)). As to the rst term of
(3.12), it consists of three signals: s, and

C
T
. It is reasonable according to ACE generating
reinforcement signal to consider the performance s, ring strength , and the aected output

C
T
.
This indicates how the outputs of fuzzy approximator inuence the changes in the adaptive critic
element. In adaptive control, the phenomenon of the possible unboundedness of weight estimates
will occur when the persistency of excitation (PE) condition fails to hold. Without the last terms
of (3.12) and (3.13), the s, r, and

C
T
should be persistently exciting signals. In other words,
positive numbers 1
i
, o
i
, c
i
(i =1, 2, 3) exist such that given tt
0
, there exists t
i
[t, t +1
i
] such that
[t
i
, t
i
+ o
i
][t, t + 1
i
] and
1
1
i
_
t
i
+o
i
t
i
g
i
(t)g
i
(t)
T
dt c
i
I t t
0
, (3.23)
where g
1
=s, g
2
=r and g
3
=

C
T
. The last terms of the weight tuning rules (3.12) and (3.13) for
RLAFC, which are similar to the e-modication of adaptive control theory, are employed to guarantee
the boundedness of weight estimates even though PE does not hold. Therefore, the proposed control
scheme with actor-critic reinforcement learning rules can guarantee the boundedness of all signals
generated in the closed-loop system without making any assumptions of PE conditions.
Remark 2. It can be found that an implicit parameter p in (3.20)(3.22) determines the magnitudes
of s,

C and

R. A larger p will result in smaller convergence regions of s,

C and

R,
and vice versa. Another factor in (3.20) is K
min
, which can be increased to reduce s.
4. Simulation results
There are two examples in this section: the rst is the cart-pole problem used for comparing
RLAFC with other reinforcement learning controllers, and the second is controlling the SCARA
robot for demonstrating the eectiveness of the proposed RLAFC.
ARTICLE IN PRESS
Chuan-Kai Lin / Fuzzy Sets and Systems ( ) 9
0.00 2.00 4.00 6.00
Time (sec)
0.00
40.00
80.00
(
d
e
g
)
(t)
Fig. 2. Angular position 0(t) versus time (response of cart-pole system controlled by RLAFC).
Example 1. Controlling the cart-pole system can be viewed as the benchmark for reinforcement
learning controllers. If the position of the cart on the track is not taken into consideration, then
balancing a rigid pole mounted on a cart is similar to controlling a one-link robot. The dynamics of
the cart-pole system can be described as follows:

0 =
q sin 0 + cos 0[(u m!

0
2
sin 0 + j
c
sgn( x))}(m
c
+ m)] j

0}m!
![(4}3) mcos
2
0}(m
c
+ m)]
,
x =
u + m![

0
2
sin 0

0 cos 0] j
c
sgn( x)
m
c
+ m
with q=9.8 m}s
2
the acceleration due to gravity, m=0.1 kg the mass of pole, m
c
=1 kg the mass
of cart, !=0.5 m the half-pole length on cart, j
c
=0.0005 the coecient of friction of cart on track,
x the position of the cart on the track, 0 the angle of the pole with the vertical and u the force
applied to cart 1. Therefore, structural property

M 2J
m
=0 is satised according to M(0)=(4}3)
(mcos
2
0)}(m
c
+m) and J
m
(0,

0)=m!

0 cos 0 sin 0}(m

c
+m). The inputs of fuzzy system are chosen as
x
1
=0, x
2
=

0, x
3
=x and x
4
= x. The format of )th rule () =27(!
4
1)+9(!
3
1)+3(!
2
1)+(!
1
1))
is as follows:
If x
1
is A
!
1
)1
and x
2
is A
!
2
)2
and x
3
is A
!
3
)3
and x
4
is A
!
4
)4
then ,
1
is B
)1
, where ) =1, . . . , 81, !
i
=1, 2, 3
for i =1, 2, 3, 4, ,
1
is the output of fuzzy approximator. The membership functions for inputs are
selected as
j
A
!
i
)i
(x
i
) = e
((x
i
x
!
i
i
)}o
i
)
,
where x
!i
i
=a
i
+b
i
(!
i
1) with a
1
=a
3
= 1, a
2
=a
4
= 10, b
1
=b
3
=1, b
2
=b
4
=10, o
1
=o
3
=0.5,
and o
2
=o
4
=8. The membership functions for outputs are singletons. As other experiments of re-
inforcement learning controllers for the cart-pole system, the simulation will only concern about
10 Chuan-Kai Lin / Fuzzy Sets and Systems ( )
ARTICLE IN PRESS
Table 1
Reinforcement learning methods comparison on the cart-pole problem
Learning method Number of trials
AHC 26
Lees 8
FACL 6
RLAFC 1
z

2
Fig. 3. Two-links SCARA robot.
balancing the pole on the cart 1,5,7. The initial weights for

C and

R are small random numbers
and the design parameters are chosen as K=50, p=1, K
W
=50 and K
C
=50. Suppose each trial
has 6 s. The initial conditions are 0=45

,

0=0, x=0 and x=0. Fig. 2 shows simulation results
for balancing angle of the pole. The comparison with AHC 1, Lees 7, FACL approaches 5 and
proposed RLAFC are listed in Table 1. The proposed RLAFC needs only 4s to complete the control
task, however, the other three approaches need at least 6 trials (each trial has 500 000 time steps).
This demonstrates the faster convergence rate of the RLAFC.
Example 2. In this example, computer simulations were conducted on the 2DOF SCARA robot ma-
nipulator to verify the availability and tracking performance of the proposed controller. The SCARA
robot manipulator, which consists of two parallel revolute axis and two rigid links, is depicted in
Fig. 3. The axes of the revolute joints are directed upwards in positive : direction. Therefore, the
angles of the rst and second links were considered to be 0
1
and 0
2
, respectively. The numerical
values of parameters of the robot model were specied as that in 11. For demonstrating the tracking
performance of our proposed controller, the desired trajectories for 0
1
and 0
2
were set as
0
d1
= 0.5 + 0.2(sin t + sin 2t) (rad) for 0
1
and
0
d1
= 13 0.1(sin t + sin 2t) (rad) for 0
2
, respectively.
The inputs of fuzzy system are chosen as x
1
=0
1
, x
2
=

0
1
, x
3
=0
2
and x
4
=

0. The format of )th rule
() =27(!
4
1) + 9(!
3
1) + 3(!
2
1) + (!
1
1)) is as follows:
ARTICLE IN PRESS
Chuan-Kai Lin / Fuzzy Sets and Systems ( ) 11
0.00 2.00 4.00 6.00
Time (sec)
(a)
0.00
40.00
80.00
(
d
e
g
)

1
(t)
0.00 2.00 4.00 6.00
Time (sec) (b)
-20.00
0.00
20.00
(
d
e
g
)

2
(t)
desired trajectory
1
(t)
RLAFC: K=diag(5,5)
RLAFC: K=diag(50,50)
RLAFC: K=diag(500,500)
desired trajectory
2
(t)
RLAFC: K=diag(5,5)
RLAFC: K=diag(50,50)
RLAFC: K=diag(500,500)
Fig. 4. Simulations for (a) 0
1
(t) and (b) 0
2
(t) using RLAFC controller with dierent K.
If x
1
is A
!
1
)1
and x
2
is A
!
2
)2
and x
3
is A
!
3
)3
and x
4
is A
!
4
)4
then

[
1
is B
)1
and

[
2
is B
)2
, where
) =1, . . . , 81, !
i
=1, 2, 3 for i =1, 2, 3, 4,

[
1
and

[
2
are the outputs of fuzzy approximator. The mem-
bership functions for inputs are chosen as
j
A
!
i
)i
(x
i
) = e
((x
i
x
!
i
i
)}o
i
)
,
where x
i
=a
i
+ b
i
(!
i
1) with a
1
=a
3
= 2, a
2
=a
4
= 10, b
1
=b
3
=2, b
2
=b
4
=10, o
1
=o
3
=1.5,
and o
2
=o
4
=8. The membership functions for inputs are singletons. There are 81 rules in the rule
base and each rule has four inputs and two outputs, i.e., there are 162 weights for ACE and 162
weights for fuzzy rule base should be tuned.
12 Chuan-Kai Lin / Fuzzy Sets and Systems ( )
ARTICLE IN PRESS
0.00 2.00 4.00 6.00
Time (sec) (a)
-0.01
0.00
0.01
Reinforcement Signal r
1
(t)
0.00 2.00 4.00 6.00
Time (sec) (b)
-0.001
0.000
0.001
Reinforcement Signal r
2
(t)
Fig. 5. Reinforcement signals (a) r
1
(t) and (b) r
2
(t).
The weights for adaptive critic element and fuzzy approximator are small random numbers. Fig.
4 shows that the tracking errors are decreased as the gain matrix K is increased as expected. Al-
though we can only guarantee s is UUB, the tracking errors can be kept very small. The reinforce-
ment learning signals for K=diaq{500, 500}, p=1, K
W
=diaq{500, 500} and K
C
=diaq{50, 50}
are shown in Fig. 5. It can be found that r(t) are bounded. Moreover, random noises presented at
joint 1 and joint 2 with bounds 10 and 1 Nm, respectively. Fig. 6 shows the RLAFC can suppress
the external disturbances with the same parameters K=diaq{500, 500}, p=1, K
W
=diaq{500, 500}
and K
C
=diaq{50, 50}. All these simulations were carried out using Delphi programs on a PC
with AMD-Duron 700 MHz CPU and 128 MB RAM, and the running time is about 20 s. For
ARTICLE IN PRESS
Chuan-Kai Lin / Fuzzy Sets and Systems ( ) 13
0.00 2.00 4.00 6.00
Time (sec) (a)
0.00
40.00
80.00
(
d
e
g
)

1
(t)
0.00 2.00 4.00 6.00
Time (sec) (b)
-20.00
0.00
20.00
(
d
e
g
)

2
(t)
desired trajectory
1
(t)
RLAFC
desired trajectory
2
(t)
RLAFC
Fig. 6. Simulations for (a) 0
1
(t) and (b) 0
2
(t) using RLAFC controller with noises.
traditional reinforcement learning control, the controller can only deal with set-point regulation and
it takes many trials to achieve an acceptable performance. Therefore, the proposed RLAFC is more
eective.
5. Conclusions
In this paper, the development of a reinforcement learning adaptive fuzzy control system has
been presented for tracking control of a robot with uncertainties. By the weight updating rules of
RLAFC developed from Lyapunov stability theorem, we can show that all signals in the closed-loop
system are bounded without any assumptions of PE conditions to make the controller robust even in
14 Chuan-Kai Lin / Fuzzy Sets and Systems ( )
ARTICLE IN PRESS
the presence of approximation errors and external disturbances. All simulation results show that the
proposed RLAFC converges fast to a very small error region that improves the convergence rate of
traditional reinforcement learning controller.
References
[1] A.G. Barto, R.S. Sutton, C.W. Anderson, Neuronlike elements that can solve dicult learning control problems,
IEEE Trans. Systems Man Cybernet. 13 (1983) 835846.
[2] J. Campos, F.L. Lewis, Adaptive critic neural network for feedforward compensation, Proc. American Control Conf.,
1999, pp. 28132818.
[3] R.H. Crites, A.G. Barto, Improving elevator performance using reinforcement learning, Adv. Neural Inform. Process.
Systems 8 (1996) 10171023.
[4] S.H.G. ten Hagen, Continuous state space Q-learning for control of nonlinear systems, Ph.D. Thesis, University of
Amsterdam, The Netherlands, 2001.
[5] L. Joue, Fuzzy inference system learning by reinforcement methods, IEEE Trans. System Man Cybernet. 28 (1998)
338355.
[6] L.P. Kaelbling, M.L. Littman, A.W. Moore, Reinforcement learning: a survey, J. Artif. Intell. Res. 4 (1996) 237285.
[7] C.C. Lee, A self learning rule-based controller employing approximate reasoning and neural net concept, Internat.
J. Intell. Systems 6 (1991) 7193.
[8] F.L. Lewis, A. Yesilidrek, K. Liu, Multilayer neural-net robot controller with guaranteed performance, IEEE Trans.
Neural Networks 7 (1996) 388399.
[9] P. Marbach, J.N. Tsitsiklis, Simulation-based optimization of Markov reward processes, IEEE Trans. Automat. Control
46 (2001) 191209.
[10] T.J. Procyk, E.H. Mamdani, A linguistic self-organizing process control, Automatica 15 (1979) 1530.
[11] M. Saad, L.A. Dessaint, P. Bigras, K. Al-Haddad, Adaptive versus neural adaptive control: application to robotics,
Internat. J. Adaptive Control Signal Process. 8 (1994) 223236.
[12] J.-J.E. Slotine, W. Li, Adaptive manipulator control: a case study, IEEE Trans. Automat. Control 33 (1988)
9951003.
[13] R.S. Sutton, Generalization in reinforcement learning: successful examples using sparse coarse coding, Adv. Neural
Inform. Process. Systems 8 (1996) 10381044.
[14] R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
[15] R.S. Sutton, D. McAllester, S. Singh, Y. Mansour, Policy gradient methods for reinforcement learning with function
approximation, Adv. Neural Inform. Process. Systems 12 (2000) 10571063.
[16] J.N. Tsitsiklis, B. Van Roy, An analysis of temporal-dierence learning with function approximation, IEEE Trans.
Automat. Control 42 (1997) 674690.
[17] L.-X. Wang, J.M. Mendel, Fuzzy basis functions, universal approximation, and orthogonal least-squares learning,
IEEE Trans. Neural Networks 3 (1992) 807814.
[18] X.-J. Zeng, M.G. Singh, Approximation theory of fuzzy systemsMIMO case, IEEE Trans. Fuzzy System 3 (1995)
219235.