Rational and Convergent Learning in Stochastic Games

Michael Bowling Manuela Veloso

mhb@cs.cmu.edu veloso@cs.cmu.edu

Computer Science Department

Carnegie Mellon University
Pittsburgh, PA 15213-3890

Abstract In Section 2 we provide a rudimentary review of the neces-

sary game theory concepts: stochastic games, best-responses,
This paper investigates the problem of policy learn- and Nash equilibria. In Section 3 we present two desirable
ing in multiagent environments using the stochastic properties, rationality and convergence, that help to elucidate
game framework, which we briefly overview. We the shortcomings of previous algorithms. In Section 4 we
introduce two properties as desirable for a learning contribute a new algorithm toward achieving these properties
agent when in the presence of other learning agents, called WoLF (“Win or Learn Fast”) policy hill-climbing, and
namely rationality and convergence. We examine prove that this algorithm is rational. Finally, in Section 5 we
existing reinforcement learning algorithms accord- present empirical results of the convergence of this algorithm
ing to these two properties and notice that they fail in a number and variety of domains.
to simultaneously meet both criteria. We then con-
tribute a new learning algorithm, WoLF policy hill- 2 Stochastic Games
climbing, that is based on a simple principle: “learn 
 
  
 
quickly while losing, slowly while winning.” The A


stochastic game is a tuple  
, where
algorithm is proven to be rational and we present is the number of agents, is a set of states, is the set
empirical results for a number of stochastic games of actions available to agent  with being the

 !"   
joint
 
action
 $#
showing the algorithm converges. space
% & ' ( )
, is a transition function   *#
+
, and is a reward function for the  th agent
. This is very similar to the MDP framework except we
1 Introduction have multiple agents selecting actions and the next state and
rewards depend on the joint action of the agents. Also notice
The multiagent learning problem consists of devising a learn- that each agent has its own separate reward function. The
ing algorithm for our single agent to learn a policy in the goal for each agent is to select actions in order to maximize
presence of other learning agents that are outside of our con- its discounted future reward with discount factor , .
trol. Since the other agents are also adapting, learning in the SGs are a very natural extension of MDPs to multiple
presence of multiple learners can be viewed as a problem of agents. They are also an extension of matrix games to mul-
a “moving target,” where the optimal policy may be chang- tiple states. Two common matrix games are in Figure 1. In
ing while we learn. Multiple approaches to multiagent learn- these games there are two players; one selects a row and the
ing have been pursued with different degrees of success (as other selects a column of the matrix. The entry of the matrix
surveyed in [Weiß and Sen, 1996] and [Stone and Veloso, they jointly select determines the payoffs. The games in Fig-
2000]). Previous learning algorithms either converge to a ure 1 are zero-sum games, where the row player receives the
policy that is not optimal with respect to the other player’s payoff in the matrix, and the column player receives the neg-
policies, or they may not converge at all. In this paper we ative of that payoff. In the general case (general-sum games)
contribute an algorithm to overcome these shortcomings. each player has a separate matrix that determines its payoff.
We examine the multiagent learning problem using the &3.' '
framework of stochastic games. Stochastic games (SGs) - '/.'
' &4.'
are a very natural multiagent extension of Markov deci- .' '10 2
.' ' &65
sion processes (MDPs), which have been studied extensively
as a model of single agent learning. Reinforcement learn- Matching Pennies Rock-Paper-Scissors
ing [Sutton and Barto, 1998] has been successful at find-
ing optimal control policies in the MDP framework, and has Figure 1: Two example matrix games.
also been examined as the basis for learning in stochastic
games [Claus and Boutilier, 1998; Hu and Wellman, 1998; Each state in a stochastic game can be viewed as a matrix
Littman, 1994]. Additionally, SGs have a rich background in game with the payoffs
  879;:<
for each joint action determined by the
game theory, being first introduced in 1953 [Shapley]. matrix entries . After playing the matrix game and
receiving their payoffs the players are transitioned to another techniques showing that they fail to simultaneously achieve
state (or matrix game) determined by their joint action. We these properties.
can see that SGs then contain both MDPs and matrix games
as subsets of the framework. 3.1 Properties
We contribute two desirable properties of multiagent learning
Mixed Policies. Unlike in single-agent settings, deterministic algorithms: rationality and convergence.
policies in multiagent settings can often be exploited by the
other agents. Consider the matching pennies matrix game as Property 1 (Rationality) If the other players’ policies con-
shown in Figure 1. If the column player were to play either verge to stationary policies then the learning algorithm will
action deterministically, the row player could win a payoff of converge to a policy that is a best-response to their policies.

or policies. A mixed policy, 


one every time. This requires us  to # consider mixed strategies
6   
, is a function
This is a fairly basic property requiring the player to be-
have optimally when the other players play stationary strate-
that maps states to mixed strategies, which are probability gies. This requires the player to learn a best-response pol-
distributions over the player’s actions. icy in this case where one indeed exists. Algorithms that are
Nash Equilibria. Even with the concept of mixed strategies not rational often opt to learn some policy independent of the
there are still no optimal strategies that are independent of other players’ policies, such as their part of some equilibrium
the other players’ strategies. We can, though, define a notion solution. This completely fails in games with multiple equi-
of best-response. A strategy is a best-response to the other libria where the agents cannot independently select and play
players’ strategies if it is optimal given their strategies. The an equilibrium.
major advancement that has driven much of the development Property 2 (Convergence) The learner will necessarily con-
of matrix games, game theory, and even stochastic games is verge to a stationary policy. This property will usually be
the notion of a best-response equilibrium, or Nash equilib- conditioned on the other agents using an algorithm from some
rium [Nash, Jr., 1950]. class of learning algorithms.
A Nash equilibrium is a collection of strategies for each of The second property requires that, against some class of
the players such that each player’s strategy is a best-response other players’ learning algorithms (ideally a class encompass-
to the other players’ strategies. So, no player can get a higher ing most “useful” algorithms), the learner’s policy will con-
payoff by changing strategies given that the other players also verge. For example, one might refer to convergence with re-
don’t change strategies. What makes the notion of equilib- spect to players with stationary policies, or convergence with
rium compelling is that all matrix games have such an equi- respect to rational players.
librium, possibly having multiple equilibria. In the zero-sum In this paper, we focus on convergence in the case of self-
examples in Figure 1, both games have an equilibrium con- play. That is, if all the players use the same learning algorithm
sisting of each player playing the mixed strategy where all the do the players’ policies converge? This is a crucial and dif-
actions have equal probability. ficult step towards convergence against more general classes
The concept of equilibria also extends to stochastic games. of players. In addition, ignoring the possibility of self-play
This is a non-trivial result, proven by Shapley [1953] for zero- makes the naive assumption that other players are inferior
sum stochastic games and by Fink [1964] for general-sum since they cannot be using an identical algorithm.
stochastic games. In combination, these two properties guarantee that the
learner will converge to a stationary strategy that is optimal
3 Motivation given the play of the other players. There is also a connec-
tion between these properties and Nash equilibria. When all
The multiagent learning problem is one of a “moving target.”
players are rational, if they converge, then they must have
The best-response policy changes as the other players, which
converged to a Nash equilibrium. Since all players converge
are outside of our control, change their policies. Equilibrium
to a stationary policy, each player, being rational, must con-
solutions do not solve this problem since the agent does not
verge to a best response to their policies. Since this is true of
know which equilibrium the other players will play, or even
each player, then their policies by definition must be an equi-
if they will tend to an equilibrium at all.
librium. In addition, if all players are rational and conver-
Devising a learning algorithm for our agent is also chal-
gent with respect to the other players’ algorithms, then con-
lenging because we don’t know which learning algorithms
vergence to a Nash equilibrium is guaranteed.
the other learning agents are using. Assuming a general case
where other players may be changing their policies in a com- 3.2 Other Reinforcement Learners
pletely arbitrary manner is neither useful nor practical. On
There are few RL techniques that directly address learn-
the other hand, making restrictive assumptions on the other
ing in a multiagent system. We examine three RL tech-
players’ specific methods of adaptation is not acceptable, as
niques: single-agent learners, joint-action learners (JALs),
the other learners are outside of our control and therefore we
and minimax-Q.
don’t know which restrictions to assume.
We address this multiagent learning problem by defining Single-Agent Learners. Although not truly a multiagent
two properties of a learner that make requirements on its be- learning algorithm, one of the most common approaches is
havior in concrete situations. After presenting these proper- to apply a single-agent learning algorithm (e.g. Q-learning,
ties we examine previous multiagent reinforcement learning 
TD( ), prioritized sweeping, etc.) to a multi-agent domain.
They, of course, ignore the existence of other agents, assum-
1. Let


and be learning rates. Initialize,
ing their rewards and the transitions are Markovian. They
essentially treat other agents as part of the environment.

      
This naive approach does satisfy one of the two properties.
If the other agents play, or converge to, stationary strategies 2. Repeat,
(a) From state select action with probability 
7 : 7  :<
then their Markovian assumption holds and they converge to
an optimal response. So, single agent learning is rational. On
7
with some exploration.
the other hand, it is not generally convergent in self-play. This (b) Observing reward  and next state ,
is obvious to see for algorithms that learn only deterministic
!  " $#&% 

(' % +) * -
' ,/.!021 56 678 
policies. Since they are rational, if they converge it must be
to a Nash equilibrium. In games where the only equilibria
354
(c) Update 
are mixed equilibria (e.g. Matching Pennies), they could not 7 ;: 

converge. There are single-agent learning algorithms capable and constrain it to a legal probability
of playing stochastic policies [Jaakkola et al., 1994; Baird
distribution,
9
 :
;'=< IG > H @?A02BDC.!0E1 3F4   6 
and Moore, 1999]. In general though just the ability to play
J K(L J NG M
if
otherwise

stochastic policies is not sufficient for convergence, as will be
shown in Section 4.
Joint Action Learners. JALs [Claus and Boutilier, 1998] Table 1: Policy hill-climbing algorithm (PHC) for player  .
observe the actions of the other agents. They assume the
other players are selecting actions based on a stationary pol- mixed policy. The policy is improved by increasing the prob-

PO the highest valued action
Raccording
Q ' the al-
icy, which they estimate. They then play optimally with re- ability that it selects to
& ' (
spect to this learned estimate. Like single-agent learners they a learning rate . Notice that when
are rational but not convergent, since they also cannot con- gorithm is equivalent to Q-learning, since with each step the
verge to mixed equilibria in self-play. policy moves to the greedy ' policy executing the highest val-
Minimax-Q. Minimax-Q [Littman, 1994] and Hu & Well- ued action with probability (modulo exploration).
man’s extension of it to general-sum SGs [1998] take a dif- This technique, like Q-learning, is rational and will con-
ferent approach. These algorithms observe both the actions verge to an optimal policy if the other players are playing
and rewards of the other players and try to learn a Nash equi- stationary strategies. The proof follows from the proof of Q-
librium explicitly. The algorithms learn and play the equi- learning, which guarantees the S values will converge to SUT
librium independent of the behavior of other players. These with a suitable exploration policy.1 Similarly,  will converge
algorithms are convergent, since they always converge to a to a policy that is greedy according to S , which is converging
stationary policy. However, these algorithms are not ratio- to SVT , the optimal response S -values. Despite the fact that it
nal. This is most obvious when considering a game of Rock- is rational and can play mixed policies, it still doesn’t show
Paper-Scissors against an opponent that almost always plays any promise of being convergent. We show examples of its
“Rock”. Minimax-Q will still converge to the equilibrium so- convergence failures in Section 5.
lution, which is not optimal given the opponent’s policy.
4.2 WoLF Policy Hill-Climbing
In this work we are looking for a learning technique that is We now introduce the main contribution of this paper. The
rational, and therefore plays a best-response in the obvious contribution is two-fold: using a variable learning rate, and
case where one exists. Yet, its policy should still converge. the WoLF principle. We demonstrate these ideas as a modifi-
We want the rational behavior of single-agent learners and cation to the naive policy hill-climbing algorithm.
JALs, and the convergent behavior of minimax-Q.
The basic idea is to vary the learning rate used by the al-
gorithm in such a way as to encourage convergence, with-
4 A New Algorithm out sacrificing rationality. We propose the WoLF principle as
In this section we contribute an algorithm towards the goal of an appropriate method. The principle has a simple intuition,
a rational and convergent learner. We first introduce an algo- learn quickly while losing and slowly while winning. The
rithm that is rational and capable of playing mixed policies, specific method for determining when the agent is winning is
but does not converge in experiments. We then introduce a by comparing the current policy’s expected payoff with that
modification to this algorithm that results in a rational learner of the average policy over time. This principle aids in con-
that does in experiments converge to mixed policies. vergence by giving more time for the other players to adapt
to changes in the player’s strategy that at first appear benefi-
4.1 Policy Hill Climbing cial, while allowing the player to adapt more quickly to other
players’ strategy changes when they are harmful.
A simple extension of Q-learning to play mixed strategies The required changes for WoLF policy hill-climbing are
is policy hill-climbing (PHC) as shown in Table 1. The al- shown in Table 2. Practically, the algorithm requires two
gorithm, in essence, performs hill-climbing in the space of
mixed policies. Q-values are maintained just as in normal 1
The issue of exploration is not critical to this work. See [Singh
Q-learning. In addition the algorithm maintains the current et al., 2000a] for suitable exploration policies for online learning.
 1. Let ,



 be learning rates. Initialize, as essentially involving a variable learning rate, nor has such

  9       @2 :  an approach been applied to learning in stochastic games.

5 Results
2. Repeat,
In this section we show results of applying policy hill-
(a,b) Same as PHC in Table 1 climbing and WoLF policy hill-climbing to a number of dif-
(c) Update estimate of average policy,   ,
@ 2: @E(' ferent games, from the multiagent reinforcement learning lit-
 6   
 6 : 

6  ' erature. The domains include two matrix games that help to
M   9 6  # 
 6  
show how the algorithms work and the effect of the WoLF
principle on convergence. The algorithms were also applied
(d) Update 
7 ;:  to two multi-state SGs. One is a general-sum grid world do-
and constrain it to a legal probability main used by Hu & Wellman [1998]. The other is a zero-sum
9
 :
'< IG > H if @?A02BDC.!0E1 3F4 
6  
distribution, soccer game introduced by Littman [1994].
J K DL J G M otherwise The experiments involve training the players using the
same learning algorithm. Since PHC and WoLF-PHC are ra-
tional, we know that if they converge against themselves, then
 3 


  3  
! I 
where,

> ? < >>   ifotherwise
 to
 a Q  equilibrium. For the ma-

they must have converged Nash


 Q  was used.
trix game experiments , but for the other results a

and were decreased proportionately Into all
more aggressive '
cases both the
"! 87  , although
Table 2: WoLF policy hill-climbing algorithm for player  . the exact proportion varied between domains.

learning learning rate parameters,


and , with 



 . 5.1 Matrix Games
The algorithms were applied to the two matrix games, from
The learning rate that is used to update the
 policy depends
on whether the agent is currently winning ( ) or losing ( ).

Figure 1. In both games, the Nash equilibrium is a mixed pol-
icy consisting of executing the actions with equal probability.
This is determined by comparing the expected value, using The large number of trials and small ratio of the learning rates
the current Q-value estimates, of following the current policy were used for the purpose of illustrating how the algorithm
 in the current state with that of following the average policy learns and converges.
  . If the expectation of the current policy is smaller

(i.e. the The results of applying both policy hill-climbing and
agent is “losing”) then the larger learning rate, is used. WoLF policy hill-climbing to the matching pennies game is
WoLF policy hill-climbing remains rational, since only the shown in Figure 2(a). WoLF-PHC quickly begins to oscil-
speed of learning is altered. In fact, any bounded variation late around the equilibrium, with ever decreasing amplitude.
of the learning rate would retain rationality. Its convergence On the other hand, PHC oscillates around the equilibrium but
properties, though, are quite different. In the next section without any appearance of converging. This is even more
we show empirical results that this technique converges to ra- obvious in the game of rock-paper-scissors. The results are
tional policies for a number and variety of stochastic games. shown in Figure 2(b), and show trajectories of the players’
The WoLF principle also has theoretical justification for a re- strategies in policy space through one million steps. Policy
stricted class of games. For two-player, two-action, iterated hill-climbing circles the equilibrium policy without any hint
matrix games, gradient ascent (which is known not to con- of converging, while WoLF policy hill-climbing very nicely
verge [Singh et al., 2000b]) when using a WoLF varied learn- spirals towards the equilibrium.
ing rate is guaranteed to converge to a Nash equilibrium in
self-play [Bowling and Veloso, 2001]. 5.2 Gridworld
Something similar to the WoLF principle has also been We also examined a gridworld domain introduced by Hu and
studied in some form in other areas, notably when consid- Wellman [1998] to demonstrate their extension of Minimax-
ering an adversary. In evolutionary game theory the adjusted Q to general-sum games. The game consists of a small grid
replicator dynamics [Weibull, 1995] scales the individuals’ shown in Figure 3(a). The agents start in two corners and
growth rate by the inverse of the overall success of the popula- are trying to reach the goal square on the opposite wall. The
tion. This will cause the population’s composition to change players have the four compass actions (i.e. N, S, E, and W),
more quickly when the population as a whole is performing which are in most cases deterministic. If the two players at-
poorly. A form of this also appears as a modification to the tempt to move to the same square, both moves fail. To make
randomized weighted majority algorithm [Blum and Burch, the game interesting and force the players to interact, while
1997]. In this algorithm, when an expert makes a mistake, in the initial starting position the North action is uncertain,
a portion of its weight loss is redistributed among the other and is only executed with probability 0.5. The optimal path
experts. If the algorithm is placing large weights on mistaken for each agent is to move laterally on the first move and then
experts (i.e. the algorithm is “losing”), then a larger portion of move North to the goal, but if both players move laterally
the weights are redistributed (i.e. the algorithm adapts more then the actions will fail. There are two Nash equilibria for
quickly.) Neither research lines recognized their modification this game. They involve one player taking the lateral move
 1 1
WoLF-PHC: Pr(Heads) PHC
PHC: Pr(Heads) WoLF
Pr(Heads) 0.8 0.8

Pr(Paper)
0.6 0.6

0.4 0.4

0.2 0.2

0 0
0 200000 400000 600000 800000 1e+06 0 0.2 0.4 0.6 0.8 1
Iterations Pr(Rock)
(a) Matching Pennies Game (b) Rock-Paper-Scissors Game

Figure 2: (a) Results for matching pennies: the policy for one of the players as a probability distribution while learning with
PHC and WoLF-PHC. The other player’s policy looks similar. (b) Results for rock-paper-scissors: trajectories of one player’s
policy. The bottom-left shows PHC in self-play, and the upper-right shows WoLF-PHC in self-play.

and the other trying to move North. Hence the game requires evaluated after every 50,000 steps. The worst performing pol-
that the players coordinate their behaviors. icy was then used for the value of that learning run.
WoLF policy hill-climbing successfully converges to one Figure 3(b) shows the percentage of games won by the
of these equilibria. Figure 3(a) shows an example trajectory different players when playing their challengers. “Minimax-
of the players’ strategies for the initial state while learning Q” represents Minimax-Q when learning against itself (the
over 100,000 steps. In this example the players converged results were taken from Littman’s original paper.) “WoLF”
to the equilibrium where player one moves East and player represents WoLF policy hill-climbing learning against itself.
two moves North from the initial state. This is evidence that
WoLF policy hill-climbing can learn an equilibrium even in a with

Q
and
“PHC(L)”
and

“PHC(W)”
Q
 , respectively.
represents policy hill-climbing
“WoLF(2x)” represents
general-sum game with multiple equilibria. WoLF policy hill-climbing learning with twice the training
(i.e. two million steps). The performance of the policies were
5.3 Soccer averaged over fifty training runs and the standard deviations
The final domain is a comparatively large zero-sum soccer are shown by the lines beside the bars. The relative ordering
game introduced by Littman [1994] to demonstrate Minimax- by performance is statistically significant.
Q. An example of an initial state in this game is shown in Fig- WoLF-PHC does extremely well, performing equivalently
ure 3(b), where player ’B’ has possession of the ball. The goal to Minimax-Q with the same amount of training2 and contin-
is for the players to carry the ball into the goal on the opposite ues to improve with more training. The exact effect of the
side of the field. The actions available are the four compass WoLF principle can be seen by its out-performance of PHC,
directions and the option to not move. The players select ac- using either the larger or smaller learning rate. This shows
tions simultaneously but they are executed in a random order, that the success of WoLF-PHC is not simply due to changing
which adds non-determinism to their actions. If a player at- learning rates, but rather to changing the learning rate at the
tempts to move to the square occupied by its opponent, the appropriate time to encourage convergence.
stationary player gets possession of the ball, and the move
fails. Unlike the grid world domain, the Nash equilibrium for 6 Conclusion
this game requires a mixed policy. In fact any deterministic
In this paper we present two properties, rationality and con-
policy (therefore anything learned by an single-agent learner
vergence, that are desirable for a multiagent learning algo-
or JAL) can always be defeated [Littman, 1994].
rithm. We present a new algorithm that uses a variable learn-
Our experimental setup resembles that used by Littman
ing rate based on the WoLF (“Win or Learn Fast”) princi-
in order to compare with his results for Minimax-Q. Each
ple. We then showed how this algorithm takes large steps
player was trained for one million steps. After training,
towards achieving these properties on a number and variety
its policy was fixed and a challenger using Q-learning was
of stochastic games. The algorithm is rational and is shown
trained against the player. This determines the learned pol-
empirically to converge in self-play to an equilibrium even in
icy’s worst-case performance, and gives an idea of how close
games with multiple or mixed policy equilibria, which previ-
the player was to the equilibrium policy, which would per-
ous multiagent reinforcement learners have not achieved.
form no worse than losing half its games to its challenger.
Unlike Minimax-Q, WoLF-PHC and PHC generally oscillate 2
The results are not directly comparable due to the use of a dif-
around the target solution. In order to account for this in the ferent decay of the learning rate. Minimax-Q uses an exponential
results, training was continued for another 250,000 steps and decay that decreases too quickly for use with WoLF-PHC.
 G
A

S2 S1

Pr(West)
50
1 0.8 0.6 0.4 0.2 0
1 0
40
0.8 0.2

% Games Won
30
Pr(North)

Pr(North)
0.6 0.4

0.4 0.6 20

0.2 0.8 10
Player 1
Player 2
0 1
0 0.2 0.4 0.6 0.8 1 0
Pr(East) Minimax-Q WoLF WoLF(x2) PHC(L) PHC(W)

(a) Gridworld Game (b) Soccer Game

Figure 3: (a) Gridworld game. The dashed walls represent the actions that are uncertain. The results show trajectories of two
players’ policies while learning with WoLF-PHC. (b) Soccer game. The results show the percentage of games won against a
specifically trained worst-case opponent after one million steps of training.

Acknowledgements. Thanks to Will Uther for ideas and [Jaakkola et al., 1994] T. Jaakkola, S. P. Singh, and M. I. Jor-
discussions. This research was sponsored by the United dan. Reinforcement learning algorithm for partially ob-
States Air Force under Grants Nos F30602-00-2-0549 and servable markov decision problems. In Advances in Neu-
F30602-98-2-0135. The content of this publication does not ral Information Processing Systems 6, 1994.
necessarily reflect the position or the policy of the sponsors [Littman, 1994] M. L. Littman. Markov games as a frame-
and no official endorsement should be inferred. work for multi-agent reinforcement learning. In Proceed-
ings of the Eleventh International Conference on Machine
References Learning, pages 157–163, 1994.


[Baird and Moore, 1999] L. C. Baird and A. W. Moore. Gra- [Nash, Jr., 1950] J. F. Nash, Jr. Equilibrium points in -
dient descent for general reinforcement learning. In Ad- person games. PNAS, 36:48–49, 1950.
vances in Neural Information Processing Systems 11. The [Shapley, 1953] L. S. Shapley. Stochastic games. PNAS,
MIT Press, 1999.
39:1095–1100, 1953.
[Blum and Burch, 1997] A. Blum and C. Burch. On-line
[Singh et al., 2000a] S. Singh, T. Jaakkola, M. L. Littman,
learning and the metrical task system problem. In Tenth
and C. Szepesvári. Convergence results for single-step
Annual Conference on Computational Learning Theory,
on-policy reinforcement-learning algorithms. Machine
1997.
Learning, 2000.
[Bowling and Veloso, 2001] M. Bowling and M. Veloso.
[Singh et al., 2000b] S. Singh, M. Kearns, and Y. Mansour.
Convergence of gradient dynamics with a variable learn-
Nash convergence of gradient dynamics in general-sum
ing rate. In Proceedings of the Eighteenth International
games. In Proceedings of the Sixteenth Conference on Un-
Conference on Machine Learning, 2001. To Appear.
certainty in Artificial Intelligence, pages 541–548, 2000.
[Claus and Boutilier, 1998] C. Claus and C. Boutilier. The [Stone and Veloso, 2000] P. Stone and M. Veloso. Multia-
dyanmics of reinforcement learning in cooperative multi- gent systems: A survey from a machine learning perspec-
agent systems. In Proceedings of the Fifteenth National
tive. Autonomous Robots, 8(3), 2000.
Conference on Artificial Intelligence. AAAI Press, 1998.
[Sutton and Barto, 1998] R. S. Sutton and A. G. Barto. Re-
[Fink, 1964] A. M. Fink. Equilibrium in a stochastic n-
inforcement Learning. The MIT Press, 1998.
person game. Journal of Science in Hiroshima University,
Series A-I, 28:89–93, 1964. [Weibull, 1995] J. W. Weibull. Evolutionary Game Theory.
The MIT Press, 1995.
[Hu and Wellman, 1998] J. Hu and M. P. Wellman. Multi-
agent reinforcement learning: Theoretical framework and [Weiß and Sen, 1996] G. Weiß and S. Sen, editors. Adapta-
an algorithm. In Proceedings of the Fifteenth International tion and Learning in Multiagent Systems. Springer, 1996.
Conference on Machine Learning, pages 242–250, 1998.