​​​​​Information Compression in Dynamic Games​​​​​

We first present a brief literature survey on information compression in single-agent decision problems because it has inspired several of the key ideas presented in this paper.

Single-agent decision/control problems are problems where one agent chooses actions over time on top of an ever-changing system to maximize their total reward. These problems have been extensively studied in the control theory [23], operations research [39], computer science [41], and mathematics [8] literature. Models like Markov Decision Process (MDP) and Partially Observable Markov Decision Process (POMDP) have been analyzed and applied widely in real-world systems. It is well known that in an MDP, the agent can use a Markov strategy—making decisions based on the current state—without loss of optimality. A Markov strategy can be seen as a strategy based on compressed information: the full information—state and action history—is compressed into only the current state. Furthermore, in finite horizon problems, such optimal Markov strategies can be found through a sequential decomposition procedure. It is also well known that any POMDP can be transformed into an MDP with an appropriate belief acting as the underlying state [23, Chapter 6]. As a result, the agent can use a belief-based strategy without loss of optimality. A belief-based strategy compresses the full information into the conditional belief of the current state. Critically, this information compression is strategy-independent [2, 44, 45, 23]. For general single-agent control problems, sufficient conditions that guarantee optimality of compression-based strategies have been proposed under the names of sufficient statistic [43, 46, 58, 18, 47] and information state [23, 24, 48]. In these works, the authors transform single-agent control problems with partial observations into equivalent problems with complete observations with the sufficient statistic/information state acting as the underlying state.

Multi-agent dynamic decision problems are either teams where all agents have the same objective, or games where agents have different objectives and are strategic. Information compression in dynamic teams has been investigated in [55, 32, 34, 24, 54, 48, 19], and many other works (see [54] and [48] for a list of references). Dynamic games can be divided into two categories: those with a static underlying environment (e.g. repeated games), and those with an underlying dynamic system. Over the years, economics researchers have studied repeated games extensively (e.g. see [30, Chapter 7]). As our focus is on dynamic games with an underlying dynamic system, we will not discuss the literature on repeated games. Among models for dynamic games with an underlying dynamic system, the model of zero-sum games, as a particular class which possesses special properties, has been analyzed in [42, 28, 40] and many others (see [37] for a list of references). Non-zero-sum games with an underlying dynamic system and symmetric information have also been studied extensively [5, 9]. For such dynamic games with perfect information, the authors of [26] introduce the concept of Markov Perfect Equilibrium (MPE), where each player compresses their information into a Markov state. Dynamic games with asymmetric information have been analyzed in [29, 27, 31, 33, 13, 14, 35, 36, 53, 52, 56, 51, 37]. In [33], the authors introduce the concept of Common Information Based Markov Perfect Equilibrium (CIB-MPE), which is an extension of MPE in partially observable systems. In a CIB-MPE, all players choose their actions at each time based on the Common-Information-Based (CIB) belief (a compression of the common information) and private information instead of full information. The authors establish the existence of CIB-MPE under the assumption that the CIB belief is strategy-independent. Furthermore, the authors develop a sequential decomposition procedure to solve for such equilibria. In [36], the authors extend the result of [33] to a particular model where the CIB beliefs are strategy-dependent. They introduce the concept of Common Information Based Perfect Bayesian Equilibrium (CIB-PBE). In a CIB-PBE all players choose their actions based on the CIB belief and their private information. They show that such equilibria can be found through a sequential decomposition whenever the decomposition has a solution. The authors conjecture the existence of such equilibria. The authors of [51] extend the model of [36] to games among teams. They consider two compression maps and their associated equilibrium concepts. For the first compression map, which is strategy-independent, they establish preservation of equilibrium payoffs. For the second information compression map, which is strategy-dependent, they propose a sequential decomposition of the game. If the decomposition admits a solution, then there exists a CIB-BNE based on the compressed information. Furthermore, they provide an example where CIB-BNE based on this specific compressed information do not exist. The example also proves that the conjecture about the existence of CIB-PBEs, made in [36], is false.

In addition to the methods on information compression that appear in [26, 33, 36, 51], there are two lines of work on games where the players’ decisions are based on limited information. In the first line of work, players face exogenous hard constraints on the information that can be used to choose actions [38, 7, 12, 15, 3]. In the second line of work, players can utilize any finite automaton with any number of states to choose actions, however more complex automata are assumed to be more costly [1, 4]. In our work, we also deal with finite automaton based strategies. However, there is a critical difference between our work and both lines of literatureboth of the above-mentioned lines of work: Our primary interest is to study conditions under which a compression based strategy profile can form an equilibrium under standard equilibrium concepts when unbounded rationality and perfect recall are allowed. Under these equilibrium concepts, we do not restrict the strategy of any player, nor do we impose any penalty on complicated strategies. In other words, a compression based strategy needs to be a best response compared to all possible strategies with full recall in terms of the payoff alone. The methodology for information compression presented in this paper is similar in spirit to that of [26, 33, 36, 56, 37]. However, this paper is significantly different from those works as it deals with the discovery of information compression maps that lead not only to the existence (in general) of various types of compressed information based equilibria but also to the preservation of all equilibrium payoff profiles (a topic not investigated in [26, 33, 36, 56, 37]). This paper builds on [51]; it identifies embodiments of the two information compression maps studied in [51] for a much more general class of games than that of [51], and a broader set of equilibrium concepts.

Our main contributions are the following:

1. 1.

   We propose two notions of information states/compressed information for dynamic games with asymmetric information that result in from strategy-independent compression maps: Mutually Sufficient Information (MSI) and Universally Sufficient Information (USI) — Definitions 4 and 1, respectively. We present an example that highlights the differences between MSI and USI.

2. 2.

   We show that in finite dynamic games with asymmetric information, Bayes–Nash Equilibria (BNE) and Sequential Equilibria (SE) exist when all players use MSI-based strategies — Theorems 2 and 4, respectively.

3. 3.

   We prove that when all players employ USI-based strategies the resulting sets of BNE and SE payoff profiles are same as the sets of BNE and SE payoff profiles resulting when all players use full information based strategies — Theorems 3 and 5, respectively.

4. 4.

   We prove that when all players use USI-based strategies the resulting set of weak Perfect Bayesian Equilibrium (wPBE) payoff profiles can be a proper subset of the set of all wPBE payoff profiles — Proposition 6. A result similar to that of Proposition 6 is also true under Watson’s PBE [57].

   Figure 1 depicts the results stated in Contributions 3 and 4 above.

5. 5.

   We present several examples — Examples 5.3 through 5.6 — of finite dynamic games with asymmetric information where we identify MSI and USI.
   

Additional contributions of this work are:

1. 1.

   A set of alternative definitions of SE — Appendix B. These definitions are equivalent to the original definition of SE given in [21] and help simplify some of the proofs of the main results in this paper.

2. 2.

   A new methodology for establishing existence of equilibria. The methodology is based on a best response function defined through a dynamic program for a single-agent control problem.

3. 3.

   A counterexample showing that a well-known strategy-dependent compression map, resulting in sufficient private information along with common information-based beliefs, does not guarantee existence of equilibria based on the above-stated compressed information.

We follow the notational convention of stochastic control literature (i.e. using random variables to define the system, representing information as random variables, etc.) instead of the convention of game theory literature (i.e. game trees, nodes, information sets, etc.) unless otherwise specified. This allows us to apply techniques from stochastic control, which we rely heavily upon, in a more natural way. We use capital letters to represent random variables, bold capital letters to denote random vectors, and lower case letters to represent realizations. We use superscripts to indicate players, and subscripts to indicate time. We use $i$ to represent a typical player and $-i$ to represent all players other than $i$. We use $t_{1}:t_{2}$ to indicate the collection of timestamps $(t_{1},t_{1}+1,\cdots,t_{2})$. For example, $X_{1:4}^{i}$ stands for the random vector $(X_{1}^{1},X_{2}^{i},X_{3}^{i},X_{4}^{i})$. For random variables or random vectors represented by Latin letters, we use the corresponding script capital letters to denote the space of values these random vectors can take. For example, $\mathcal{H}_{t}^{i}$ denotes the space of values the random vector $H_{t}^{i}$ can take. The products of sets refers to Cartesian products. We use $\mathbb{P}(\cdot)$ and $\mathbb{E}[\cdot]$ to denote probabilities and expectations, respectively. We use $\Delta(\varOmega)$ to denote the set of probability distributions on a finite set $\varOmega$. For a distribution $\nu\in\Delta(\varOmega)$, we use $\mathrm{supp}(\nu)$ to denote the support of $\nu$. When writing probabilities, we will omit the random variables when the lower case letters that represent the realizations clearly indicate the random variable it represents. For example, we will use $\mathbb{P}(y_{t}^{i}|x_{t},u_{t})$ as a shorthand for $\mathbb{P}(Y_{t}^{i}=y_{t}^{i}|X_{t}=x_{t},U_{t}=u_{t})$. When $\lambda$ is a function from $\varOmega_{1}$ to $\Delta(\varOmega_{2})$, with some abuse of notation we write $\lambda(\omega_{2}|\omega_{1}):=(\lambda(\omega_{1}))(\omega_{2})$ as if $\lambda$ is a conditional distribution. We use $\bm{1}_{A}$ to denote the indicator random variable of an event $A$.

In general, probability distributions of random variables in a dynamic system are only well defined after a complete strategy profile is specified. We specify the strategy profile that defines the distribution in superscripts, e.g. $\mathbb{P}^{g}(x_{t}^{i}|h_{t}^{0})$. When the conditional probability is independent of a certain part of the strategy $(g_{t}^{i})_{(i,t)\in\varOmega}$, we may omit this part of the strategy in the notation, e.g. $\mathbb{P}^{g_{1:t-1}}(x_{t}|y_{1:t-1},u_{1:t-1})$, $\mathbb{P}^{g^{i}}(u_{t}^{i}|h_{t}^{i})$ or $\mathbb{P}(x_{t+1}|x_{t},u_{t})$. We say that a realization of some random vector (for example $h_{t}^{i}$) is admissible under a partially specified strategy profile (for example $g^{-i}$) if the realization has strictly positive probability under some completion of the partially specified strategy profile (In this example, that means $\mathbb{P}^{g^{i},g^{-i}}(h_{t}^{i})>0$ for some $g^{i}$). Whenever we write a conditional probability or conditional expectation, we implicitly assume that the condition has non-zero probability under the specified strategy profile. When only part of the strategy profile is specified in the superscript, we implicitly assume that the condition is admissible under the specified partial strategy profile. In this paper, we make heavy use of value functions and reward-to-go functions. Such functions will be clearly defined within their context with the following convention: $Q$ stands for state-action value functions; $V$ stands for state value functions; and $J$ stands for reward-to-go functions for a given strategy profile (as opposed to $Q$ or $V$, both of which are typically defined via a maximum over all strategies).

In this section we formulate a general model for a finite horizon dynamic game with finitely many players.

Denote the set of players by $\mathcal{I}$. Denote the set of timestamps by $\mathcal{T}=\{1,2,\cdots,T\}$. At time $t$, player $i\in\mathcal{I}$ takes action $U_{t}^{i}$, obtains instantaneous reward $R_{t}^{i}$, and then learns new information $Z_{t}^{i}$. Player $i$ may not necessarily observe the instantaneous rewards $R_{t}^{i}$ directly. The reward is observable only if it is part of $Z_{t}^{i}$. Define $Z_{t}=(Z_{t}^{i})_{i\in\mathcal{I}},U_{t}=(U_{t}^{i})_{i\in\mathcal{I}}$, and $R_{t}=(R_{t}^{i})_{i\in\mathcal{I}}$. We assume that there is an underlying state variable $X_{t}$ and

where $(f_{t})_{t\in\mathcal{T}}$ are fixed functions. The primitive random variable $X_{1}$ represents the initial move of nature. The primitive random vector $H_{1}=(H_{1}^{i})_{i\in\mathcal{I}}$ represents the initial information of the players. The initial state and information $X_{1}$ and $H_{1}$ are, in general, correlated. The random variables $(W_{t})_{t=1}^{T}$ are mutually independent primitive random variables representing nature’s move. The vector $(X_{1},H_{1})$ is assumed to be mutually independent with $W_{1},W_{2},\cdots,W_{T}$. The distributions of the primitive random variables are common knowledge to all players.

Define $\mathcal{X}_{t},\mathcal{U}_{t},\mathcal{Z}_{t},\mathcal{W}_{t},\mathcal{H}_{1}$ to be the sets of possible values of $X_{t},U_{t},Z_{t},W_{t},H_{1}$ respectively. The sets $\mathcal{X}_{t},\mathcal{U}_{t},\mathcal{Z}_{t},\mathcal{W}_{t},\mathcal{H}_{1}$ are assumed to be common knowledge among all players. In this work, in order to focus on conceptual difficulties instead of technical issues, we make the following assumption.

We assume perfect recall, i.e. the information player $i$ has at time $t$ is $H_{t}^{i}=(H_{1}^{i},Z_{1:t-1}^{i})$, and player $i$’s action $U_{t}^{i}$ is contained in the new information $Z_{t}^{i}$. A behavioral strategy $g^{i}=(g_{t}^{i})_{t\in\mathcal{T}}$ of player $i$ is a collection of functions $g_{t}^{i}\colon\mathcal{H}_{t}^{i}\mapsto\Delta(\mathcal{U}_{t}^{i})$, where $\mathcal{H}_{t}^{i}$ is the space where $H_{t}^{i}$ takes values. Under a behavioral strategy profile $g=(g^{i})_{i\in\mathcal{I}}$, the total reward/payoff of player $i$ in this game is given by

We initially consider two solution concepts for dynamic games with asymmetric information: Bayes–Nash Equilibrium (BNE) and Sequential Equilibrium (SE). We define BNE and SE below.

Whereas Definition 2 of SE is different from that of [21], we show in Appendix B that it is equivalent to the concept in [21]. We use Definition 2 as it is more suitable for the development of our results.

In this paper, we are interested in analyzing the performance of strategy profiles that are based on some form of compressed information. Let $K_{t}^{i}$ be a function of $H_{t}^{i}$ that can be sequentially updated, i.e. there exist functions $(\iota_{t}^{i})_{t\in\mathcal{T}}$ such that

Write $K^{i}=(K_{t}^{i})_{t\in\mathcal{T}}$ and $K=(K^{i})_{i\in\mathcal{I}}$. We will refer to $K^{i}$ as the compression of player $i$’s information under $\iota^{i}=(\iota_{t}^{i})_{t\in\mathcal{T}}$. A $K^{i}$-based (behavioral) strategy $\rho^{i}=(\rho_{t}^{i})_{t\in\mathcal{T}}$ is a collection of functions $\rho_{t}^{i}\colon\mathcal{K}_{t}^{i}\mapsto\Delta(\mathcal{U}_{t}^{i})$. A strategy profile where each player $i$ uses a $K^{i}$-based strategy is called a $K$-based strategy profile. If a $K$-based strategy profile forms an Bayes-Nash (resp. sequential) equilibrium, then it is called a $K$-based Bayes-Nash (resp. sequential) equilibrium. Note that unlike [38, 7, 12, 15, 3], we require the $K$-based BNE and $K$-based SE to contain no profitable deviation among all full-history-based strategies.

Our goal is to discover properties/features of the compressed information $K$ sufficient to guarantee that (i) there exists $K$-based BNE and SE; (ii) the set of $K$-based BNE (resp. SE) payoff profiles is equal to the set of (general strategy based) BNE (resp. SE) profiles under perfect recall.

To achieve the above-stated objectives we proceed as follows: First, we introduce two notions of information state, namely MSI and USI (Section 3). Then, we investigate the existence of MSI-based and USI-based BNE and SE, as well as the preservation of the set of all BNE and SE payoff profiles when USI-based strategies are employed by all players (Section 4).

In words, MSI represents mutually consistent compression of information in a dynamic game: Player $i$ could compress their information to $K^{i}$ without loss of performance when other players are compressing their information to $K^{-i}$. Note that MSI imposes interdependent conditions on the compression maps of all players: It requires the compression maps of all players to be consistent with each other.

The following lemma provides a sufficient condition for a compression maps to yield mutually sufficient information.

In words, the condition of Lemma 1 means that $K_{t}^{i}$ has the same predictive power as $H_{t}^{i}$ in terms of forming a belief on the current state and other players’ compressed information whenever other players are using compression-based strategies. This belief is sufficient for player $i$ to predict other player’s actions and future state evolution. Since other players are using compression-based strategies, player $i$ does not have to form a belief on other player’s full information in order to predict other players’ actions.

The definition of USI can be separated into two parts: The first part states that the conditional distribution of $H_{t}^{i}$, player $i$’s full information, given $K_{t}^{i}$, the compressed information, does not depend on other players’ strategies. This is similar to the idea of sufficient statistics in the statistics literature [20]: If player $i$ would like to use their “data” $H_{t}^{i}$ to estimate the “parameter” $g^{-i}$, then $K_{t}^{i}$ is a sufficient statistic for this parameter estimation problem. The second part states that $K_{t}^{i}$ has the same predictive power as $H_{t}^{i}$ in terms of forming a belief on the current state and other players’ full information. In contrast to the definition of mutually sufficient information, if $K^{i}$ is unilaterally sufficient information, then $K^{i}$ is sufficient for player $i$’s decision making regardless of whether other players are using any information compression map.

Using Lemma 1 it can be shown that if $K^{i}$ is USI for each $i\in\mathcal{I}$, then $K=(K^{i})_{i\in\mathcal{I}}$ is MSI. The converse is not true. The following example illustrates the difference between MSI and USI.

The main idea for the proof of Theorem 2 is the definition of a best-response correspondence through the dynamic program for an underlying single-agent control problem.

The intuition behind Theorem 3 is that one can think of player $i$’s information that is not included in the unilaterally sufficient information $K_{t}^{i}$ as a private randomization device for player $i$: When player $i$ is using a strategy that depends on their information outside of $K_{t}^{i}$, it is as if they are using a randomized $K^{i}$-based strategy. The main idea for the proof of Theorem 3 is to show that for every BNE strategy profile $g$, player $i$ can switch to an “equivalent” randomized $K^{i}$-based strategy $\rho^{i}$ while maintaining the equilibrium and payoffs.³³3Besides the connection of USI to sufficient statistics, the idea behind the construction of the equivalent $K^{i}$-based strategy is also closely related to the idea of the Rao–Blackwell estimator [20], where a new estimator is obtained by taking the conditional expectation of the old estimator given the sufficient statistics. The theorem then follows from iteratively switching the strategy of each player.

Example 3.2 can also be used to illustrate that when $K$ is an MSI but not an USI, $K$-based BNE exist but $K$-based strategies do not attain all equilibrium payoffs.