17th Mediterranean Conference on Control & Automation Makedonia Palace, Thessaloniki, Greece June 24 - 26, 2009 Evolutionary Dynamics of Collaborative Environments with Heterogeneous Agents (Invited Paper) Kiran K. Somasundaram and John S. Baras Institute for Systems Research and Department of Electrical and Computer Engineering University of Maryland College Park College Park, MD 20742 USA Email: kirans@umd.edu, baras@umd.edu Abstract—The seminal work of Maynard Prince and Price in 1973 laid the foundations in establishing the concept of evolutionary game stable strategies. It attempts to select strategies which are robust to evolutionary selection pressures. We observe that most of the literature as evolved is concentrated on single objective games. In this work we extend the notion of evolutionary stability to games with vector payoff functions. We illustrate that this notion of Multicriteria evolutionary stability, models a much larger class of interactions in social, economic and biological problems. Particular applications of interest are in autonomic wireless sensor networks and autonomic networked control systems. We present a multi-population replicator dynamics, which models the evolution of agent actions in a network with varying levels of selfishness. We observe that this model clearly demarks the regions of cooperation among these selfish agents. We provide analytical results on the dynamic stability of the replication. This clearly characterizes the aforementioned demarcations. I. I NTRODUCTION We live in an age where information technology has matured to understand and incorporate social behaviors of individuals adopting the technology in an environment. And the abundance of inexpensive communications technology ([8], [16], [21]) has enabled these individuals to establish ties and collaborate among themselves in an almost autonomous setting. This form of collaboration enables autonomous agents (possibly humans) to accomplish tasks which would be otherwise impossible individually: [7], [1], [4], [6]. However these protocols require that the users dedicate their resources towards this collaboration [15]. This could be user’s time, device’s battery, computational effort, memory storage, or other infrastructure usages. Thus there is a clear quantification of the benefits a user receives from a collaboration and the resources he expends for the collaboration. In an autonomic network the agents might opt for different trade-offs between the benefits and cost of collaboration. This introduces a heterogeneity in the environment. For instance, in a large sensor network configuration there might be nodes with high computational capabilities to carry out the number crunching, which should be willing to sacrifice their battery for this regard. There might be also selfish nodes, which only want to benefit from the collaboration. Thus in modelling protocols in autonomous networks, we need to pay heed to realistic non-altruism in the participating agents. In this paper we provide a mathematical 978-1-4244-4685-8/09/$25.00 ©2009 IEEE model to describe the action dynamics in such a heterogeneous environment. In this paper, we do not consider any specific network protocol. Instead we adopt an abstract model for the action space of the agents participating in the protocol. In an autonomic environment, there is no centralized control of the user’s decision and thus each user has the free will to choose his actions. In an abstract sense, we refer to as the act of adhering to the protocol as cooperation. It should be mentioned here that this abstract model can be used to model a large class of network interactions. Examples include wireless ad-hoc networks, peer-to-peer networks, social networks and e-services. We observe that for every action of each agent, there are at least two metrics which reflect the benefit she derives from and the cost she pays to the network for collaboration. However in many contexts, there are no obvious methods to combine these metrics. For instance, in a tactical Mobile AdHoc Network (MANET) or for an e-business, the transactions are closely monitored by peers or specialized sentinel nodes [17], [10], [12]. In this case, the agents expend their resources following the agreed protocol. In doing so, they boost their reputation in the network or market which is beneficial for their future transactions. As these examples illustrate, the cost (battery life or user effort) and the benefits (reputation) are not trivially comparable. In such situations it is common in optimization theory to consider decision strategies which are Pareto Optimal. This multi-agent multi-criteria decision problem lends itself to Pareto Nash equilibria strategies. In this paper we consider a modified form of the standard replicator dynamics called the Multi-population replicator dynamics and study its dynamic stability properties under various levels of selfishness that the agents exhibit. We establish strong relations of the stability of the stationary states of the dynamics with the Pareto Nash strategies. To the best of our knowledge, there is no prior literature on the evolutionary dynamics for games involving such vector payoffs. In this paper, we show that for our dynamics, the perceived tradeoff function serves as a bifurcation parameter. This bifurcation clearly demarcates different forms of evolution in our agent setting. The corresponding stability analysis suggests methods 121 of mechanism design to propel the agent environment to a grand coalition. This paper is organized as follows. In section II we delineate the role of trust in a large sensor network setting. In section III we present a population dynamics model for the action dynamics in a heterogeneous setting. Section IV introduces our multi-population dynamics model. In sections V and VI, we present our stability results for this multi-population dynamics. II. ROLE OF T RUST IN AUTONOMOUS S ENSOR N ETWORKS Advances in low power electronics has fuelled the development and deployment of large scale sensor networks [16]. There have been several protocols proposed for selforganization in such large scale autonomous networks [19]. Most of the initial protocols assumed altruistic behavior of the sensor nodes for proper functioning of the system. But several security flaws in these protocols stemmed from selfish node behavior [12]. However traditional cryptographic techniques were handicapped to solve this problem of node selfishness. Instead trust and reputation methods from e-business and eservices were adopted to propel the functioning of sensor networked systems [17]. In these protocols the services are usually offered to and taken from nodes which have a high trustworthiness in the system. In such a framework the sensor nodes in the network are more likely to cooperate with the ones prescribed by the protocol (to boost their reputation). Any node with a good reputation in the system is likely to be trusted and thereby gains from the coalition. In the process of forming the coalition, the sensor nodes expend their resources for communication and cryptographic primitives. To treat both trust and cost metrics in the spirit of maximization, we define the residual-resource gain as the resources left after paying for the cost of cooperation. Sensor nodes are typically limited in these resources and in an autonomous setting display varying levels of selfishness. This is because different nodes operate at different trade-off points in the residual gain-benefit objective space. In this paper we model this heterogeneous environment as multi-population environment, where the agents within a population, bias their strategies based on their fitness in the current environment. We emphasize again that the heterogeneity arises from the different trade-offs the agents choose. As far as capability and limitations are concerned, all the agents are assumed to be homogeneous. In order to understand the relation between these dynamics and Pareto Nash strategies we present a series of ideas inspired from evolutionary biology and multi-criteria optimization in the forthcoming sections. III. M ODELING ACTION DYNAMICS Let us consider a large population of sensor agents. To accomplish any task in the network, all the agents must act as per the given protocol. Each agent might have a specific task and we do not attempt to model it. Instead we adopt a point of view that the agent might or might not adhere to the protocol. This then models the action space of the agents in the system. Definition If the agent conforms with the protocol, then the agent is said to cooperate (C). Otherwise the agent is said to defect (D). We borrow this terminology from the famous prisoner’s dilemma problem in game theory. Definition A task is said be achieved if all the participating agents cooperate. Any heterogeneous system which can achieve the task is said to have a grand-coalition trade-off. To model the action dynamics of the system, we use population dynamics that have been studied rigorously in evolutionary biology. In using the population models, we assume that the number of agents participating in a given task is very large and thus the population shares can be modelled as a continuum. These continuum population models are common in evolutionary biology [18]. In working with population dynamics the payoffs for the actions are usually the ensemble average payoffs. In our case this would correspond to the average residual-gain and average benefit in the current population state. Thus the bi-criteria payoff-objective space is convex and can be characterized by weighted-sum scalarization [13]. For our problem, we have a heterogeneous population P. This population is divided into n homogeneous factions each with its own perceived trade-off. Each of these factions, i ∈ {1, 2, · · · , n} chooses a particular trade-off between the residual gain and benefit (λi , 1 − λi ), 0 ≤ λi ≤ 1 (tradeoffs from weighted sum scalarization). We denote these subpopulations as {Pi , 1 ≤ i ≤ n}. Since we assume that the capabilities and limitations of the nodes are identical, associated with every agent are two identical (i.e the same for all agents) payoff functions which reflect the cost and benefit of collaboration. We denote the action set for every player as (C, D) for cooperate and defect respectively. Within each sub-population Pi , every player is assumed to play one of the pure strategies (C, D). Let the size of the subpopulation Pi be pi . Within Pi , let pC i agents play cooperate and pD agents play defect. The population shares within Pi i pD pC D C i are then represented by xi = pi and xi = pii respectively.  C  xi This forms the population share state in Pi , xi = . xD i Then the overall population share state is given by x = [x1 , x2 , · · · , xn ]. Let x−i denote the population share states of all sub-populations other than Pi . Let n-linear functions ur and ub represent the residual gain and benefit function for all the players. Standard population dynamics assumes that n players are chosen randomly, one from each sub-population and are made to compete. In our setting, one player is drawn for each sub-population Pi and they are assumed to carry out the task. Based on their perceived gains, these players change their action strategy. In evolutionary biology this perceived gain is defined as fitness. It represents the number of off-springs the action strategy A ∈ {C, D} produces in the environment. ur (C, x−i ) represents the residual gain left in player i after she cooperates. Similarly ub (C, x−i ) represents the benefit she receives out of the cooperation. If this player lives in the subpopulation Pi , then the average perceived fitness is 122 given by λi ur (C, x−i ) + (1 − λi )ub (C, x−i ). Correspondingly λi ur (C, x−i )+(1−λi )ub (C, x−i ) represents the fitness when she choses to defect. We refer the reader to Chapters 3 and 5 of [11] for the rationality arguments behind evolutionary dynamics. In the next section we consider a particular class of evolutionary dynamics driven by the fitness function. C D C D Let us assume there is some inherent birth and death rates βi and δi within each of these sub-populations. If we use the fitness function introduced in Section III then the corresponding population dynamics would turn to be = = (βi + λi (ur (C, x−i )) (βi + λi (ur (D, x−i )) ẋC 1 The corresponding population share action dynamics would then be = r r (λi (u (C, x−i ) − u (x)) +(1 − λi )(ub (C, x−i ) − ur (x)))xC i ẋD i = (λi (ur (D, x−i ) − ur (x)) +(1 − λi )(ub (D, x−i ) − ur (x)))xD i · · · (M P M CR) We refer to these equations as the Multi-Population MultiCriteria Replicator dynamics (MPMCR). We refer the reader to [11] for a detailed derivation of these dynamics. The MPMCR are simple dynamics which capture the action dynamics of our heterogeneous autonomous agent system. In the next section we study the properties of this dynamics in a bi-heterogeneous system before we go to the general nheterogeneous system. V. B I -H ETEROGENEITY In this case the population is divided into only two factions, with perceived trade-offs (λ1 , 1 − λ1 ) and (λ2 , 1 − λ2 ). The game can be modelled as a symmetric matrix game (homogeneity in capabilities assumption). Let player I and II be the row and column players respectively. For the scenario to conform to the rationality of cost and benefit, the following inequalities should be satisfied. rdc > rcc , rdd > rcd bdc < bcc , bdd < bcd ẋC 2 = = (λ1 ([1 0]U r x2 − x1 .U r x2 ) (λ2 ([1 0]U r T x1 − x2 .U r T x1 ) T T +(1 − λ2 )([1 0]U b x1 − x1 .U b x1 ))xC 2 D C D (a1 xC 2 − a2 x2 )x1 x1 (1) = (b1 xC 1 (2) − C D b2 x D 1 )x2 x2 Here a1 = −7λ1 + 15(1 − λ1 ), a2 = 8λ1 − 10(1 − λ1 ), b1 = −7λ2 + 15(1 − λ2 ) and b2 = 8λ2 − 10(1 − λ2 ). Different values for the bifurcation parameters create different dynamics. The normalized vector fields for the bifurcation pair (λ1 = 1 1 3 , λ2 = 4 ), for which the grand coalition is achievable, are shown in Figure 1. However for the non-cooperative setting with bifurcation parameters (λ1 = 23 , λ2 = 34 ), where the emphasis is on the residual gain, the grand coalition is not an asymptotically stable equilibrium. This is depicted by the normalized vector fields shown in Figure 2 The vector fields shown in the two figures suggest that none of the interior strategies are asymptotically stable. We prove this result is insensitive to the perceived trade-offs. Theorem 5.1: The MPMCR dynamics in a bi-heterogeneous environment always converges to population states, where all the individuals in a given subpopulation P1 or P2 either defect or cooperate. Proof: We use a standard proof method suggested in [11]. C D D Dividing both equations (1) and (2), by xC 1 x2 x1 x2 in the interior of the cartesian product of the simplex, we obtain +(1 − λ1 )([1 0]U b x2 − x1 .U b x2 ))xC 1 ẋC 2 = Here, a1 = λ1 (rcc −rdc )+ (1−λ1 )(bcc −bdc ), a2 = λ1 (rdd − rcd ) + (1 − λ1 )(bdd − bcd ), b1 = λ2 (rcc − rdc ) + (1 − λ2 )(bcc − bdc ) and a2 = λ2 (rdd − rcd ) + (1 − λ2 )(bdd − bcd ). Thus the pair λ1 , λ2 forms the bifurcation parameters for the dynamics. Let us consider an example system with payoffs     3 2 15 15 r b U = ,U = 10 10 0 5 For these payoff matrices, the MPMCR can be written as ẋC 1 D bcd bdd TABLE II U b MATRIX +(1 − λi )(ub (D, x−i )) − δi )pD i ẋC i C bcc bdc C D C And xD 1 = 1 − x1 , x2 = 1 − x2 for the dynamics on the cartesian product of the simplices. On simplifying the above expression we obtain, +(1 − λi )(ub (C, x−i )) − δi )pC i ṗD i D rcd rdd TABLE I U r MATRIX IV. M ULTI - CRITERIA M ULTI - POPULATION R EPLICATOR DYNAMICS ṗC i C rcc rdc ẋC 1 = ẋC 2 = D (a1 xC 2 − a2 x2 ) C D x2 x2 C (b1 x1 − b2 xD 1 ) D xC x 1 1 To analyze stability of the MCMPR of equations (1) and (2), it suffices to study the stability of the above dynamics, because 123 libria of an auxiliary bi-objective game. The ideas are inspired from the relations between standard replicator dynamics and the evolutionary stable Nash strategies in a single objective setting. We present a brief summary of the methods from vector payoff games before we present the linking results. We refer the reader to [20] for a detailed description of the methods and analysis that we have developed for these vector payoff games. These vector payoff games were originally introduced by Zeleny [14]. Subsequently several modifications of these games appeared in [9], [3] and [2]. An existential algorithm for sieving the equilibria of these games was proposed by [5]. However we observe that there is no constructive algorithm to obtain the set of equilibria. Interestingly, as with single objective games and the standard replicator dynamics, we show that the MPMCR dynamics have the property of obtaining certain Pareto Nash equilibria. Let us considera n-player bi-objective symmetric game G  ur . Each player’s pure and mixed action with payoffs u = ub strategies xi , ∀i ∈ I = {1, 2, · · · , n} live on the standard D D simplex ∆i = [0, 1] (xC xC i + xi = 1, i , xi ≥ 0). 1 0.9 0.8 0.7 xC 2 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 Fig. 1. 0.2 0.3 0.4 0.5 xC 1 0.6 0.7 0.8 0.9 1 Normalized Vector Field For Grand Coalition 1 0.9 0.8 0.7 A. Pareto Nash Equilibria xC 2 0.6 In comparing vectors in R2 we use the component-wise order where a > b, a, b ∈ R2 implies a1 ≥ b1 , a2 ≥ b2 and a 6= b. 0.5 0.4 0.3 0.2 Definition The Pareto Reply of player i ∈ I for the strategy profile x−i of the rest of the players is defined as that strategy xi ∈ ∆i such that the strategy profile (xi , x−i ) is Pareto optimal with respect to the vector payoff function u(., x−i ). 0.1 0 0 0.1 0.2 0.3 0.4 0.5 xC 1 0.6 0.7 0.8 0.9 1 This Pareto Reply correspondence for player i is given by Fig. 2. Normalized Vector Field For Non-Cooperation βiP (x−i ) = { constant positive scaling of the vector field does not change the orbits. The above vector field has zero divergence. By Liouville’s theorem (chapter 6 of [11]) none of the interior states (in the cartesian product of the simplexes) is asymptotically stable. Thus to obtain the grand coalition (C, C) as an asymptotically stable strategy we need bcc − bdc bcd − bdd λ1 , λ2 < min( , ) rdc − rcc + bcc − bdc rdd − rcd + bcd − bdd The agents with different perceived trade-offs cooperate to achieve a task if the above condition is met. Those perceived trade-offs, which satisfy the inequality, constitute the grand coalition tradeoffs for a bi-heterogeneous environment evolving under multi-population replicator dynamics. In the next section we extend the bi-heterogeneity stability results to a general n-heterogeneity stability. VI. DYNAMIC S TABLITY OF MPMCR IN n-Heterogeneity In this section we establish the relation between the stable dynamic equilibria of the MPMCR and the Pareto Nash Equi- xi ∈ ∆i :6 ∃z ∈ ∆i such that u(z, x−i ) > u(xi , x−i } . The combined best reply correspondence is given by β P (x). Definition A strategy profile xP is called a Pareto Nash Equilibrium for G if xP ∈ β P (xP ). The set of all Pareto Nash Equilibria is denoted by ΘP N E . Lemma 6.1: If xP ∈ ΘP N E , then every component strategy x∗i , ∀i ∈ I is a supported strategy. i.e. x∗i = arg max λi ur (z, x−i ) + (1 − λi )ub (z, x−i ) z∈∆i The proof for lemma 6.1 is given in [20]. Essentially, the lemma establishes that at Pareto Nash Equilibria, every component strategy xi∗ by a hyperplane, whose i is supported  λi normal is given . This gives an interesting dual (1 − λi ) interpretation to the Pareto Nash strategies. Let us denote the set of Pareto Nash strategy profiles supported by the collection of hyperplanes (whose normals are Λ = {(λ1 , 1 − NE λ1 )T , (λ2 , 1 − λ2 )T , · · · , (λn , 1 − λn )T }) as ΘP . Λ 124 Definition xS ∈ ΘP N E is a strict Pareto Nash equilibrium if β P (xs ) = {xs }. B. Linking the Equilibria In this final subsection, we present the linking relations between the dynamic stability of MPMCR dynamics and the Pareto Nash Equilibria. Theorem 6.2: If x is Lyapunov stable in MPMCR then x ∈ NE ΘP . Λ NE Proof: Let us suppose x 6∈ ΘP . Λ ⇒ ∃ a player i such that there is better Pareto reply z ∈ βiP (x−i ) supported by Λ. ⇒ ∃ an action A ∈ {C, D} λi ur (A,x−i )+(1−λi )ub (A,x−i )>λi ur (x)+(1−λi )ub (x) ⇒λi (ur (A,x−i )−ur (x))+(1−λi )(ub (A,x−i )−ub (x))>0 By continuity of ur and ub , there exists a neighborhood Nδ containing x, such that for y ∈ Nδ λi (ur (A,y−i )−ur (y))+(1−λi )(ub (A,y−i )−ub (y))≥δ>0 δt Then by Gronwall’s lemma, xA i ≥ e , for x0 ∈ Nδ . Thus x is not Lyapunov stable. Theorem 6.3: If xS ∈ ΘP N E is a strict equilibrium, then S x is asymptotically stable in MPMCR. Proof: If xS is a strict Pareto Nash equilibrium, then each component Pareto Nash best reply is a unique corner point/pure strategy. (Section 5 of [20]). ⇒ ∀i ∈ I, Ai = βiP (x−i ) ∈ {C, D}. i.e. xS is a collection of pure strategies. ⇒λi (ur (A,xS−i )−ur (xS ))+(1−λi )(ub (A,xS−i )−ub (xS ))<0 where A 6= Ai . Again by the continuity of ur and ub , we have a neighbourhood Nδ around xS such that for y ∈ Nδ ⇒λi (ur (A,y−i )−ur (y))+(1−λi )(ub (A,y−i )−ub (y))≤δ<0 δt Thus for x0 ∈ Nδ , xA i ≤ e , which decays exponentially to zero. Thus the vector field around a strict Pareto Nash equilibrium has an inward velocity, which makes a strict Pareto Nash equilibrium asymptotically stable. We trust that these methods of stability analysis provide a good insight to the mechanism design of trust and reputation games to achieve a grand coalition in a environment with heterogeneous agents. Our analysis can be extended to other regular, payoff monotonic and payoff positive [11] evolutionary dynamics to establish stability results. VIII. ACKNOWLEDGMENTS This research was partially supported by MURI award W 911 NF 0710287 from the US Army Research Office (ARO). R EFERENCES [1] Samad T. Bay J. Multiagent sensing and control: Surveillance and reconnaissance for urban operations. In Proceedings of 17th Internation Symposium on Mathematical Theory of Networks and Systems, pages 891–900, 2006. [2] D. Blackwell. An analog of the minimax theorem for vector payoffs. Pacific Journal of Mathematics, 6(1):1–8, 1956. [3] Rousseau J.J. Charnes A., Huang Z.M. and Wei Q.L. Corn extremal solutions of multi-payoff games with cross-constrained strategy set. Optimization, 21(1):51–69, 1990. [4] Karataz T. Cortes J., Martinez S. and Bullo F. Coverage control for mobile sensing networks. IEEE Transactions on Robotics and Automation, 20:243–255, April 2004. [5] Zelikovsky A. Dmitrii L., Solomon D. Multiobjective games and determining pareto-nash equilibria. BULETINUL ACADEMIEI DE STIINTE, A REPUBLICII MOLDOVA MATEMATICA, (3):115–122, 2005. [6] Kumar V. Grocholsky B., Keller J. and Pappas G. Cooperative air and ground surveillance. IEEE Robotics and Automation Magazine, 13:16– 25, September 2006. [7] Bay J. Demonstrating tactical information services from coordinated uav operations. In Defense Transformation and Network-Centric Systems, SPIE, volume 6249, June 2006. [8] Elson J. and Estrin D. Wireless Sensor Networks, chapter Sensor Networks: A bridge to the phyical world. Springer, 2004. [9] Zhao J. The equilibria of a multiple objective game. International Journal of Game Theory, 20:171–182, 1991. [10] Baras J.S. Security and trust for wireless autonomic networks: System and control methods. European Journal of Control:Special Issue, 13(23):105–133, 2007. [11] Weibull J.W. Evolutionary Game Theory. MIT Press, August 1997. [12] Buttyan L. and Hubaux J.P. Security and Cooperation in Wireless Networks. Cambridge University, 2008. [13] Ehrgott M. Multicriteria Optimization. Springer, 2 edition, 2005. [14] Zeleny M. Games with multiple payoffs. International Journal of Game Theory, 4(4):179–191, December 1975. [15] Murray R. Mostofi Y., Chung T. and Burdick J. Communication and sensing tradeoffs in decentralized mobile sensor networks. In 4th Internation Conference on Information Processing in Sensor Networks (IPSN), 2005. [16] Znati T. Raghavendar C.S., Sivalingham C.S. Wireless Sensor Networks. Springer, 2004. [17] Sastry S. Roosta T., Meingast M. Distributed reputation system for tracking applications in sensor networks. In In Proceedings of Internation Workshop on Advances in Sensor Networks, 2006. [18] Price G. R. Smith M. The logic of animal conflict. Nature, 246:15–18, 1973. [19] Ailawadhi V. Sohrabi K., Gao J. and Pottie G.J. Protocols for selforganization of a wireless sensor network. IEEE Personal Communications, 7(5):16–27, Oct 2000. [20] Baras J.S. Somasundaram K. Pareto nash replies for multi-objective games. Technical report, Institute for Systems Research, 2008. [21] Guibas L. Zhao F. Wireless Sensor Networks: An Information Processing Approach. Morgan Kaufmann, 2004. VII. C ONCLUSIONS We present a simple model of replication using biological fitness to understand collaboration in large sensor networks. This model clearly delineates the regions of operation in a network with agents which have varying levels of selfishness. We trust that this method provides simple control laws to provide incentives for nodes to change their actions to achieve a grand coalition. 125 View publication stats