Monte Carlo Tree Search

Interest in artificial intelligence is growing. Its application in games is hoped to produce new approaches that will be able to transfer to real-world applications. The present paper presents an investigation into AI-techniques in the context of the classic board game Dots and Boxes. A rule-based approach, Monte Carlo Tree Search, Q-learning, MiniMax, including alpha-beta pruning, were developed, and their performance compared. All new AI approaches struggled to compete with the traditional rule-based AI. Different variations of the above mentioned techniques were investigated, yet even with incorporation as these insights helped challenging the status of the rule-based AI. The implemented MCTS performed poorest.

Hex is a two-player board game that was invented by Piet Hein in 1942 and reinvented by Nobel-prize John Nash in 1948. Hex is the most famous connection game. Although one could definitely view it as a mathematical game, numerous theorems have indeed involved Hex, this game has succeeded on developing a strong playing community. The first book dedicated to Hex strategy was released a few years ago [2]. ... Fig. 1. Game of Hex won by Black ... Rules are simple, the board is made of hexagons and is diamond-shaped. Opposing borders ...

In this paper we present a new algorithm for negotiations in non-zero-sum games. Although games have been studied extensively, most game playing algorithms have been developed under the assumption that players do not communicate. Many real-world problems, however, can be modeled as non-zero-sum games in which players may mutually benefit if they coordinate their actions, which requires negotiation. The field of Automated Negotiations is another important topic in AI, but in this field one usually assumes that utility functions have explicit expressions and can therefore be calculated easily. Traditional approaches do not apply to domains in which the utility values are instead determined by the rules of a complex game. In this paper we aim to bridge the gap between General Game Playing and Automated Negotiations. Our algorithm is an adaptation of Monte Carlo Tree Search that allows players to negotiate. It is completely domain-independent in the sense that it is not tailored to any specific game. It can be applied to any non-zero-sum game, provided that its rules are described in Game Description Language.

Real-Time Strategy (RTS) games are well-known for their substantially large combinatorial decision and state spaces, responsible for creating significant challenges for search and machine learning techniques. Exploiting domain knowledge to assist in navigating the expansive decision and state spaces could facilitate the emergence of competitive RTS game-playing agents. Usually, domain knowledge can take the form of expert traces or expert-authored scripts. A script encodes a strategy conceived by a human expert and can be used to steer a search algorithm, such as Monte Carlo Tree Search (MCTS), towards high-value states. However, a script is coarse by nature, meaning that it could be subject to exploitation and poor low-level tactical performance. We propose to perceive scripts as a collection of heuristics that can be parameterized and combined to form a wide array of strategies. The parameterized heuristics mold and filter the decision space in favor of a strategy expressed in terms of parameters. The proposed agent, ParaMCTS, implements several common heuristics and uses NaïveMCTS to search the downsized decision space; however, it requires a preceding manual parameterization step. A genetic algorithm is proposed for use in an optimization phase that aims to replace manual tuning and find an optimal set of parameters for use by EvoPMCTS, the evolutionary counterpart of ParaMCTS. Experimentation results using the µRTS testbed show that EvoPMCTS outperforms several state-of-the-art agents across multiple maps of distinct layouts.

This paper is going to be focusing on the theoretical concepts of both Monte Carlo Tree Search and Q-Learning. There will be a technical overview which will discuss the history of these concepts, how they have evolved over the years and how these AI techniques work. Following this the paper will go on to discuss and evaluate how these AI techniques have been implemented within video games. Finally, the paper will conclude with an evaluation as to which of the two techniques would be the most suitable for producing an AI companion which can learn how to behave in a similar manner to the player during the games runtime.

The impressive performance of Monte Carlo Tree Search (MCTS) based game-playing agents in high branching-factor domains such as Go, motivated researchers to apply and adapt MCTS to even more challenging domains. Real-time strategy (RTS) games feature a large combinatorial branching factor and a real-time aspect that pose significant challenges to a broad spectrum of AI techniques, including MCTS. Various MCTS enhancements were proposed, such as the combinatorial multi-armed bandit (CMAB) based sampling, state/action abstractions, and machine learning. In this paper, we propose to employ move pruning as a way to improve the performance of MCTS-based agents in the context of RTS games. We describe a class of possibly detrimental player-actions and propose several pruning approaches targeting it. The experimentation results in µRTS indicate that this could be a promising direction.

Monte-Carlo Tree Search (MCTS) grows a partial game tree and uses a large number of random simulations to approximate the values of the nodes. It has proven effective in games with such as Go and Hex where the large search space and difficulty of evaluating positions cause difficulties for standard methods. The best MCTS players use carefully hand-crafted rules to bias the random simulations. Obtaining good hand-crafting rules is a very difficult process, as even rules promoting better simulation play can result in a weaker MCTS system [12]. Our Hivemind system uses evolution strategies to automatically learn effective rules for biasing the random simulations. We have built a MCTS player using Hivemind for the game Hex. The Hivemind learned rules result in a 90% win rate against a baseline MCTS system , and significant improvement against the computer Hex world champion, MoHex.

Affect Control Theory (ACT) is a mathematical model that makes accurate predictions about human behaviour across a wide range of settings. The predictions, which are derived from statistics about human actions and identities in real and laboratory environments, are shared prescriptive and affective behaviours that are believed to lead to solutions to everyday cooperative problems. A generalisation of ACT, called BayesAct, allows the principles of ACT to be used for human-interactive agents by combining a probabilistic version of the ACT dynamical model of affect with a utility function encoding external goals.
Planning in BayesAct, which we address in this paper, then allows one to go beyond the affective prescription, and leads to the emergence of more complex interactions between “cognitive” and “affective” reasoning, such as deception leading to manipulation and altercasting. We use a continuous variant of a successful Monte-Carlo tree search planner (POMCP) that dynamically discretises the action and observation spaces while planning. We give demonstrations on two classic two-person social dilemmas.

An autonomous robot team can be employed for continuous and strategic coverage of arbitrary environments for different missions. Stochastic multi-robot planning is a very powerful technique for several problem domains. Many planning problems, e.g. swarm robotics, coverage planning, and multi-robot patrolling, require high degree of coordination which yields scalability issues for traditional joint-space planners. The other main challenge for traditional joint-space planners is the exploration versus exploitation trade-off during policy search. Exploration versus exploitation dilemma is very well studied in the context of Multi-armed bandit problem. Stochastic sampling based planners employ the multi-armed bandit theory to address the aforementioned challenges. Particularly in this work, we have been investigating stochastic tree search approaches in policy space for the multi-robot patrolling
problems. We proposed a new variant of Monte Carlo Tree Search algorithm for life-long policies by exploiting periodic trajectories of the robot team.

The core challenge facing search techniques when used to play Real-Time Strategy (RTS) games is the extensive combinatorial decision space. Several approaches were proposed to alleviate this dimensionality burden, using scripts or action probability distributions, based on expert knowledge. We propose to replace expert-authored scripts with a collection of smaller parametric scripts we call heuristics and use them to pre-select actions for Monte Carlo Tree Search (MCTS). The advantages of this proposal consist of granular control of the decision space and the ability to adapt the agent's strategy in-game, all by altering the heuristics and their parameters. Experimentation results in µRTS using a proposed implementation have shown a significant performance gain over state-of-the-art agents.

An autonomous robot team can be employed for continuous and strategic coverage of arbitrary environments for different missions. In this work, we propose an anytime approach for creating multi-robot patrolling policies. Our approach involves a novel extension of Monte Carlo Tree Search (MCTS) to allow robots to have life-long, cyclic policies so as to provide continual coverage of an environment. Our proposed method can generate near-optimal policies for a team of robots for small environments in real-time (and in larger environments in under a minute). By incorporating additional planning heuristics we are able to plan coordinated patrolling paths for teams of several robots in large environments quickly on commodity hardware.

This paper presents two AIs that enable a fighting game to be played or live-streamed as an audience participation game. The proposed fighting game AIs imitates a social facilitation in human psychology by dynamically adjusting its strength based on audience responses during gameplay. The two AIs exploit Monte-Carlo Tree Search. They have three important mechanisms, for dynamic difficulty adjustment, humanlike behavior promotion, and social facilitation integration. We developed the two AIs with different evaluation functions. The common feature is an integrated social facilitation parameter. However, the difference is that one AI is developed by generalizing an existing AI by simply adding a parameter for setting the targeted HP difference, whereas the other was particularly designed for an social facilitation by a strict control of damage score difference. Our experiment result shows that the former one yields more human-like behavior, while the latter one yields slightly better strength adjustment.

Monte Carlo tree search has brought significant improvements to the level of computer players in games such as Go, but so far it has not been used very extensively in games of strongly imperfect information with a dynamic board and an emphasis on risk management and decision making under uncertainty. In this paper we explore its application to the game of Kriegspiel (invisible chess), providing three Monte Carlo methods of increasing strength for playing the game with little specific knowledge. We compare these Monte Carlo agents to the strongest known minimax-based Kriegspiel player, obtaining significantly better results with a considerably simpler logic and less domain-specific knowledge. 1

Planning-based techniques are a very powerful tool for automated story generation. However, as the number of possible actions increases, traditional planning techniques suffer from a combinatorial explosion due to large branching factors. In this work, we apply Monte Carlo Tree Search (MCTS) techniques to generate stories  in domains with large numbers of possible actions (100+). Our approach employs a Bayesian story evaluation method to guide the planning towards believable stories that reach a user defined goal. We generate stories in a novel domain with different type of story goals. Our approach shows an order of magnitude improvement in performance over traditional search techniques.

The core challenge facing search techniques when used to play Real-Time Strategy (RTS) games is the extensive combinatorial decision space. Several approaches were proposed to alleviate this dimensionality burden, using scripts or action probability distributions, based on expert knowledge. We propose to replace expert-authored scripts by a collection of smaller parametric scripts we call heuristics and use them to pre-select actions for Monte Carlo Tree Search (MCTS). The advantages of this proposal consist of granular control of the decision space and the ability to adapt the agent’s strategy in-game, all by altering the heuristics and their parameters. Experimentation results in μRTS using a proposed implementation have shown a significant performance gain over state-of-the-art agents.