GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations

Tic-Tac-Toe⁴⁴4https://en.wikipedia.org/wiki/Tic-tac-toe is a paper-and-pencil game for two players who take turns marking the spaces in a three-by-three grid with X or O. The player who succeeds in placing three of their marks in a horizontal, vertical, or diagonal row is the winner. It is a solved game, with a forced draw assuming optimal play from both players.

• •

  Observation (input): Our observation contains “opponent moves” and “self moves”. “Opponent moves” contains all the current opponent agent’s historical actions. “Self moves” contains all the current agent’s history actions.

• •

  Actions: We define our action in the following format: $<$C$x$R$y$$>$, in which C and R mean columns and rows respectively, while $x$ and $y$ mean the index of column and row. Each player may make their own action in turn.

Prisoner’s Dilemma⁵⁵5https://en.wikipedia.org/wiki/Prisoner%27s_dilemma is a game theory thought experiment that involves two rational agents, each of whom can cooperate for mutual benefit or betray their partner ("defect") for individual reward.

• •

  Observation (input): Our observation contains “opponent moves” and “self moves”. “Opponent moves” contains all the current opponent agent’s historical actions. “Self moves” contains all the current agent’s history actions.

• •

  Actions: We define our action in the following format: $<$Silent$>$ or $<$Testify$>$. All players must take their action simultaneously.

Breakthrough⁶⁶6https://en.wikipedia.org/wiki/Breakthrough_(board_game) Breakthrough is an abstract strategy board game invented by Dan Troyka in 2000 and made available as a Zillions of Games file (ZRF). It won the 2001 8x8 Game Design Competition. The first player to reach the opponent’s home row — the one farthest from the player — is the winner. In our work, we scale the size of the board to 3*8 while maintaining its competitiveness.

• •

  Observation (input): Our observation contains “opponent moves”, “self moves”, and “board preview”. “Opponent moves” contains all the current opponent agent’s historical actions. “Self moves” contains all the current agent’s history actions. The “board preview” feature maintains the status of each grid on the board through a list of strings, denoting whether it contains a black piece, a white piece, or is empty.

• •

  Actions: We define our action in the following format: A$x$-$>$B$y$, in which A and B mean the current column index and destination column index respectively, while $x$ and $y$ mean the index of current row and destination row. Each player may make their own action in turn.

Connect Four⁷⁷7https://en.wikipedia.org/wiki/Connect_Four is a game in which the players choose a color and then take turns dropping colored tokens into a six-row, seven-column vertically suspended grid. The pieces fall straight down, occupying the lowest available space within the column. The objective of the game is to be the first to form a horizontal, vertical, or diagonal line of four of one’s own tokens.

• •

  Observation (input): Our observation contains “opponent moves” and “self moves”. “Opponent moves” contains all the current opponent agent’s historical actions. “Self moves” contains all the current agent’s history actions.

• •

  Actions: We define our action in the following format: $<$C$x$$>$ in which C means column, while $x$ means the index of column. Each player may make their action in turn.

Blind Auction⁸⁸8https://en.wikipedia.org/wiki/First-price_sealed-bid_auction is a common type of auction. In this type of auction, all bidders simultaneously submit sealed bids so that no bidder knows the bid of any other participant. The highest bidder pays the price that was submitted. All players must take their action simultaneously.

• •

  Observation (input): Our observation contains “valuation”.“Valuation” contains each of the values of all the items for the current player.

• •

  Actions: We define our action in the following format: $<$$x$$>$, in which $x$ represents the amount that a certain player would like to bid for.

Kuhn Poker⁹⁹9https://en.wikipedia.org/wiki/Kuhn_poker is a simplified form of poker. Kuhn is a simple model zero-sum two-player imperfect-information game, amenable to a complete game-theoretic analysis. In Kuhn poker, the deck includes only three playing cards, for example, a King, Queen, and Jack. One card is dealt to each player, which may place bets similarly to a standard poker. If both players bet or both players pass, the player with the higher card wins, otherwise, the betting player wins.

• •

  Observation (input): Our observation contains “card” , and “moves”. Among these, “card” denotes the current player’s hand card in this match, while “moves” represents the history of all characters’ moves together with the index of the rounds.

• •

  Actions: We define our action in the following format: $<$Pass$>$ or $<$Bet$>$. Each player may make their own action in turn.

Liar’s Dice¹⁰¹⁰10https://en.wikipedia.org/wiki/Liar%27s_dice is a class of dice games for two or more players requiring the ability to deceive and detect an opponent’s deception.

• •

  Observation (input): Our observation contains: “Self dice face value” and “last move”. “Self dice face value” describes all the face values of dices the current player has, while “last move” represents the previous player’s action.

• •

  Actions: We define our action in the following format: $<$ $x$ dices, $y$ value$>$ or $<$Liar$>$. Among these, $x$ means the quantity of dice, and $y$ means the face values of the dice. The option “Liar” denotes the current player wants to stop and challenge the previous players. Each player may make their own action in turn.

Pig¹¹¹¹11https://en.wikipedia.org/wiki/Pig_(dice_game) is a simple dice game. Players take turns to roll a single dice as many times as they wish, adding all roll results to a running total, but losing their gained score for the turn if they roll a 1.

• •

  Observation (input): Our observation contains: “self current score”, “opponent current score”, and “turn total score”. “Self current score” and “opponent current score” represent the game culminated score of the current player and opponent player respectively. While “turn total score” denotes the sum of the score of the current turn.

• •

  Actions: We define our action in the following format: $<$stop$>$ or $<$roll$>$. Each player may make their own action in turn.

Nim¹²¹²12https://en.wikipedia.org/wiki/Nim is a mathematical game of strategy in which two players take turns removing objects from distinct heaps or piles. On each turn, a player must remove at least one object and may remove any number of objects provided they all come from the same heap or pile.

• •

  Observation (input): Our observation contains: “piles”. “Piles” denotes the number of matches different piles have.

• •

  Actions: We define our action in the following format: $<$pile:$x$, take:$y$$>$. Among these, $x$ represents the index of the pile that the current player takes, and $y$ represents the number of matches the current player takes. Each player may make their own action in turn.

Negotiation¹³¹³13https://arxiv.org/pdf/1706.05125.pdf

• •

  Observation (input): Our observation contains: “turn type”, “item pool”, “most recent proposal”, “most recent utterance”, and “self value vector”. “turn type” is an enum variable, it has two options: proposal and utterance. The “Proposal” is the turn that the current player could think about the desired quantities of the items, and the “Utterance” is the turn that the current player states the values to its opponent. “item pool” represents the quantities of all the items.“most recent proposal” and “most recent utterance” represent the opponent’s latest proposal and utterance. “self value vector” represents how much the value of the items to the current player.

• •

  Actions: We define our action in the following format: $<$Agree$>$ or $<x$, $y$, $z>$. Among these, $<$Agree$>$ represents the current player agreeing on the opponent’s utterance. $x$, $y$, and $z$ represent the quantities of different items that the current player wants to get.

Complete and Incomplete Information One fundamental dimension along which games are classified is the level of information available to players. In complete information games, players possess perfect knowledge regarding the game’s structure, including the available strategies, payoffs, and the actions taken by other players. Examples of complete information games include canonical examples like chess and Tic-Tac-Toe, where all relevant information is transparent to all players throughout the game. Conversely, incomplete information games involve situations where players must make decisions without having full knowledge of the game’s parameters or the actions of other players. Classic examples of incomplete information games include strategic interactions in economics, such as auctions or negotiations, where players have limited knowledge about the valuations or preferences of other participants.

Dynamic and Static Another crucial dimension for classifying games is the timing of players’ decisions. In static games, players make decisions simultaneously, without the opportunity to observe or react to other players’ moves. Examples of static games include simultaneous-move games like the Iterated Prisoner’s Dilemma. In contrast, dynamic games involve sequential decision-making, where players observe previous moves before choosing their actions. Dynamic games encompass a wide range of strategic environments, from turn-based board games like chess to dynamic settings like Kuhn Poker, where players strategically make their actions based on the unfolding dynamics of the game.

Probabilistic and Deterministic Games can also be differentiated based on the role of uncertainty in decision-making. In deterministic games, the outcomes of players’ actions are fully determined by the game’s rules and the strategies chosen by players. Deterministic games include classic examples like chess or Tic-Tac-Toe, where each move leads to a predictable outcome based on the game’s rules and the players’ strategies. Conversely, probabilistic games involve randomness or uncertainty in determining outcomes. This uncertainty can stem from elements such as dice rolls, card draws. Examples of probabilistic games include games of chance like Kuhn Poker, Liar’s Dice, or Pig, where players must contend with the inherent uncertainty of probabilistic outcomes.

MCTS Chaslot et al. (2008) is a heuristic search algorithm that has gained prominence in recent years, particularly in the domain of board games and decision-making under uncertainty. It is characterized by its ability to efficiently explore large search spaces by sampling potential future outcomes through Monte Carlo simulations. The algorithm iteratively builds a search tree by simulating random sequences of moves from the current game state and evaluating their outcomes through repeated simulations. By focusing computational resources on promising branches of the search tree, MCTS aims to guide the search towards regions of the game space that are more likely to lead to favorable outcomes. MCTS has demonstrated remarkable success in various domains, including games like Go, where traditional search algorithms struggle due to the game’s immense complexity and branching factor.

Tit-for-Tat Axelrod (1981) is a simple but powerful strategy in the realm of repeated games and social dilemmas. The strategy is based on the principle of reciprocity, where an agent initially cooperates and then mimics the opponent’s previous action in subsequent rounds. Specifically, Tit-for-Tat starts by cooperating in the first round and then replicates the opponent’s last move in each subsequent round. Despite its simplicity, Tit-for-Tat has been shown to be remarkably effective in promoting cooperation and achieving favorable outcomes in various scenarios, including Iterated Prisoner’s Dilemma and evolutionary simulations. Its success stems from its ability to balance cooperation and retaliation, fostering reciprocal behavior and encouraging cooperation among interacting agents.

When prompting LLMs to generate the next action during the course of a game, the prompt is composed of four individual components, to make sure all the participants access the same observations and information from environments:

System Prompt provides general guidance on how the LLMs should perform.

Head Prompt provides the general background and rules of the game.

Observation Prompt is formatted by a fixed game-wise template, providing sufficient observations from the environment regarding the current gaming state, to make LLMs capable of making decisions. The following provides the template used in the Blind Auction environment:

Here $<$valuation$>$ and $<$legal_moves$>$ are variables and are obtained from a unified $<$observation$>$ object. In this way, all the participants are guaranteed to assess the same information.

Reasoning Prompt guides the LLM’s generation process, e.g., “Let’s think step by step” for the CoT Agent.

The system prompt is the initial text or context provided by the user to prompt the model to generate a response. This prompt serves as the starting point for the model to understand the user’s query or input and generate a relevant and coherent response based on the provided context. We use the same system prompt for different tasks in our work. Our system prompt is presented as follows: {mdframed} System Prompt: You are a powerful gaming agent who can make proper decisions to beat the user in gaming tasks. You are a helpful assistant that strictly follows the user’s instructions. You must answer your questions by choosing one of the legal moves given by the user!

The head prompt is the text that explains the rules of the games to the LLMs. In our work, we designed different prompts for each of the games.

• •

  Tic-Tac-Toe {mdframed} Head Prompt: Tic Tac Toe is a two-player game played on a grid. Players take turns marking a space with their respective symbols. The goal is to get 3 of one’s own symbols in a row, either horizontally, vertically, or diagonally, before the opponent does. If all nine squares are filled and no player has three in a row, the game is a draw. The Tic Tac Toe game is played on a 3 by 3 grid, with the winning length as 3. Each move is represented by a string consisting of two parts: the column (C) and the row (R), in that order. For instance, C1R2 means the movement at the position of the first column and the second row of the grid. You are playing this game with the user (opponent).

• •

  Iterated Prisoner’s Dilemma {mdframed} Head Prompt: You and your partner are in the Prisoner’s Dilemma situation. Specifically, if you $<$Testify$>$ against your partner and your partner remains $<$Silent$>,$ you will go free while your partner will get 3 years in prison on the main charge. If you remain $<$Silent$>$ but your partner $<$Testify$>$ against you, you will serve 3 years in prison and your partner will be set free. If you and your partner $<$Testify$>$ against each other, you and your partner will each serve 2 years. If both you and your partner remain $<$Silent$>$, you and your partner will each serve 1 year.

• •

  Breakthrough {mdframed} Head Prompt: Breakthrough is a two-player game played on a rectangular board. Players take turns moving their pieces, which can move one space straight or diagonally forward if the target square is empty. A piece can also move diagonally forward to capture an opponent’s piece. Capturing is optional, and a player can only capture one piece per turn. The goal is to be the first to reach the opponent’s home row, the farthest row from the player. If all of a player’s pieces are captured, they lose. The game does not allow draws, as pieces can only move forward or be captured. The Breakthrough board is identified by columns labeled starting from A (from left to right) and rows numbered 1 to 8 (from bottom to top). The intersection of a column and a row specifies a unique square on the board.

• •

  Connect Four {mdframed} Head Prompt: Connect 4 is a two-player connection board game, where the players choose a color and then take turns dropping colored discs into a vertically suspended grid. The pieces fall straight down, occupying the next available space within the column. The objective of the game is to be the first to form a horizontal, vertical, or diagonal line of four of one’s own discs. You are a gaming agent who aims to beat me in Connect 4 games. Each move is represented by a string consisting of two parts: the column (C) and the row (R), in that order. For instance, C1 means the first column.

• •

  First-price sealed-bid auction {mdframed} Head Prompt: A first-price sealed-bid auction (FPSBA) is a common type of auction. It is also known as the blind auction. In this type of auction, all bidders simultaneously submit sealed bids so that no bidder knows the bid of any other participant. The highest bidder pays the price that was submitted.

  Each action is represented by $<$$x$$>$ where $x$ refers to the bid.

• •

  Kuhn Poker {mdframed} Head Prompt: Kuhn poker is a simple model zero-sum two-player imperfect-information game, amenable to a complete game-theoretic analysis. In Kuhn poker, the deck includes only three playing cards: a King (K), a Queen (Q), and a Jack (J). One card is dealt to each player, and the third is put aside unseen. The players take turns either $<$Bet$>$ to match the bet raised by the opponent or $<$Pass$>$ to concede the game.

  If a player bets, the other player must either call the bet by matching it or fold by conceding the game. If both players pass, the game is over, and the player with the higher-ranking card wins. The card rankings are as follows: King (K) $>$ Queen (Q) $>$ Jack (J).

  You are playing Kuhn poker with the opponent. The actions are denoted by $<$Bet$>$ and $<$Pass$>$.

• •

  Liar’s Dice {mdframed} Head Prompt: Liar’s Dice is a game of bluffing and probability, played with two players and each player has 1 dice. During each turn, a player can either bid a higher quantity of any particular face value or the same quantity of a higher face value than the previous bid. Each player tries to outbid their opponent without being caught in a lie. The move in this game is denoted in $<$$x$ dices, $y$ value$>$, meaning there are at least $x$ dices with face values as $y$.

• •

  Pig {mdframed} Head Prompt: Pig is a fast-paced dice game where players risk accumulating points with each roll but risk losing them all if they roll a 1. Each player must decide when to stop rolling and bank their points, aiming to be the first to reach 100 points. You are playing Pig with the other.

• •

  Nim {mdframed} Head Prompt: In Nim, a strategic game with a set of four piles containing 1, 3, 5, and 7 matches respectively, players aim to avoid taking the last match. During each turn, a player may take any number of matches from a single pile, but must take at least one and cannot exceed the number remaining in that pile. The objective is to force the opponent to pick up the final match, thereby winning the game.

  The action is presented in $<$pile:$x$, take:$y$$>$, which means take $y$ match(es) from the $x$-th pile.

• •

  Negotiation {mdframed} Head Prompt: You are negotiating the division of Peppers, Strawberries, and Cherries with the opponent. Different values these items hold for both you and your opponent. The process is structured into two stages per round: the proposal stage and the utterance stage.

Our research team has developed a range of observation prompts tailored to different types of games. The list of these prompts is presented below.

• •

  Tic-Tac-Toe {mdframed} Observation Prompt: Your opponent has finished actions: $<$opponent_moves$>$. You have finished actions: $<$self_moves$>$.

• •

  Iterated Prisoner’s Dilemma {mdframed} Observation Prompt: You have been through this situation in the past and here are the decisions you and your partner made: (In the $idx+1$ th round, you decided to $<$move$>$ and your opponent decided to $<$opponent_move$>$) * $n$ round

• •

  Breakthrough {mdframed} Observation Prompt: The board now looks like : $<$board_preview$>$. Among which, the letter ‘b’ represents a black piece, while the letter ‘w’ represents a white piece. And the character “.” represents vacant space. The numbers in the board are the indexes of the rows. Your opponent has finished actions: $<$opponent_moves$>$.You have finished actions: $<$self_moves$>$.

• •

  Connect Four {mdframed} Observation Prompt: Your opponent has finished actions: $<$opponent_moves$>$. You have finished actions: $<$self_moves$>$.

• •

  First-price sealed-bid auction {mdframed} Observation Prompt: Now, you are in an auction with an opponent. You want to win the object and at the same time, your budget is $<$valuation$>$. Your bid must be strictly lower than or equal to $<$valuation$>$. You shall bid wisely against your opponent. Your opponent also has an expected valuation and you do not know it.

• •

  Kuhn Poker {mdframed} Observation Prompt: In this match, your card is $<$card$>$. Here are the past moves in this match: $<$self_moves$>$, $<$opponent_moves$>$.

• •

  Liar’s Dice {mdframed} Observation Prompt: Currently, the face value of your dice is $<$face_value$>$. Last time, the opponent called $<$opponent_last_action$>$. You are playing the Liar’s Dice with another opponent. Therefore, there are only two dice in total.

• •

  Pig {mdframed} Observation Prompt: Right now, your current score is $<$agent_current_score$>$ and your opponent’s current score is $<$opponent_current_score$>$. In this turn, you have earned $<$turn_total_score$>$ score.

• •

  Nim {mdframed} Observation Prompt: Currently, the 1st pile has $<$piles[0]$>$ match(es), the $2$nd pile has $<$piles[1]$>$ match(es), the $3$rd pile has $<$piles[2]$>$ match(es), $4$th pile has $<$piles[3]$>$ match(es).

• •

  Negotiation We proposed two different prompts for the “proposal” turn and “utterance” turn respectively.

  For the “proposal” turn, we have: {mdframed} Observation Prompt: Now, the opponent propose to take $<$opponent_proposal_take[0]$>$ peppers, $<$opponent_proposal_take[1]$>$ strawberries, and $<$opponent_proposal_take[2]$>$ cherries from the item pool. Last time, the utterance of the opponent was to take $<$opponent_utterance_take[0]$>$ peppers, $<$opponent_utterance_take[1]$>$ strawberries, and $<$opponent_utterance_take[2]$>$ cherries from the item pool.

  Now, it is your decision. If you find the proposal raised by the opponent is acceptable, you should output Agree. Otherwise, you should output your proposal in the format $<$Proposal: [$a$, $b$, $c$]$>$.

  For the “utterance” turn, we have:

  {mdframed}

  Observation Prompt: Last time, you propose to take $<$agent_proposal_take[0]$>$ peppers, $<$agent_proposal_take[1]$>$ strawberries, and $<$agent_proposal_take[2]$>$ cherries from the item pool. Last time, the utterance of the opponent was to take $<$opponent_utterance_take[0]$>$ peppers, $<$opponent_utterance_take[1]$>$ strawberries, and $<$opponent_utterance_take[2]$>$ cherries from the item pool.

  Now, it is your turn to provide your utterance regarding the division of items. The utterance is what you want to tell your opponent and does not mean your real intent. You should output your utterance in the format $<$Utterance: [$a$, $b$, $c$]$>$.

• •

  Prompt agent: Prompt agent does not necessitate the use of LLMs to apply any predetermined strategy prior to decision-making. Rather, it simply requests LLMs for inference and subsequently provides the outcome. {mdframed} You must choose a legal action to set up advantages. Your output must be in the following format:

  Action: Your action wrapped with $<>$, i.e., $<$format$>$

  Please return your answer without explanation!

• •

  CoT agent: CoT agent makes LLMs consider the given observation first, then give out the action according to its thinking.

  {mdframed}

  First think about your current situation, then you must choose one action from legal actions to set up advantages.

  Your output must be in the following format strictly:

  Thought: Your thought.

  Action: Your action wrapped by $<>$, i.e., $<$format$>$

  Remember, you can only choose one move from the legal actions.

• •

  SC-CoT agent: SC-CoT agent is an advanced version of the CoT agent. It obtains actions from multiple CoT trajectories. It employs the same prompt templates as in the CoT agent. {mdframed} First think about your current situation, then you must choose one action from legal actions to set up advantages.

  Your output must be in the following format strictly:

  Thought: Your thought.

  Action: Your action wrapped by $<>$, i.e., $<$format$>$

  Remember, you can only choose one move from the legal actions.

• •

  ToT agent: we follow the text generation task implementation in the official codebase of ToT ¹⁴¹⁴14https://github.com/princeton-nlp/tree-of-thought-llm/blob/master/src/tot/prompts/text.py. Specifically, the ToT is factorized into 1). candidate thought generation, 2). thought voting, 3). candidate action generation, 4). action voting:

  Here we provide the basic prompt template used in ToT. {mdframed} Step Prompt: First think about your current situation, then choose one move from legal positions to set up advantages.

  Your output should be of the following format:

  Thought:

  Your thought.

  Move:

  Your action wrapped with $<$$>$, e.g., $<$format$>$ After executing step prompts in a breath-first search manner, we utilize the original ToT vote prompt: {mdframed} Vote Prompt: Given an instruction and several choices, decide which choice is most promising. Analyze each choice in detail, then conclude in the last line "The best choice is s", where s the integer id of the choice.

To evaluate the effectiveness of our framework, we perform a sanity check by calculating the completion rates of each game. The completion rates are calculated as $\frac{50}{N}$ where $N$ is the number of matches that will take to achieve 50 valid matches. Here, a valid match means all the participants will always generate legal moves at each turn of the match. Results are summarized in Table A7. We show that all the LLM agents achieve $\geq 90\%$ completion rate, showing that the prompts are properly configured and LLMs are capable of following instructions to finish the game.

Assume that the last round of bidding was $b_{1}$ for the player and $b_{2}$ for the opponent. Assume the player’s valuation is $v$, then the regret value is calculated by

    if b_1 > b_2 + 1:
        regret = b_1 - (b_2 + 1)
    else:
        if (b_2 + 1) < v:
            regret = v - (b_2 + 1)
        else:
            regret = 0

The regret value Iterated Prisoner’s Dilemma is simply the accumulation of the regret value of per-turn Prisoner’s Dilemma:

    if player_move == ’Testify’ and opponent_move == ’Silent’:
        regret = 0
    elif player_move == ’Testify’ and opponent_move == ’Testify’:
        regret = 0
    elif player_move == ’Silent’ and opponent_move == ’Testify’:
        regret = 1
    else:
        regret = 2