Evaluation of a Multi-agent “Human-in-the-loop” Game Design System
Article No.: 19, Pages 1 - 26
Abstract
Designing games is a complicated and time-consuming process, where developing new levels for existing games can take weeks. Procedural content generation offers the potential to shorten this timeframe, however, automated design tools are not adopted widely in the game industry. This article presents an expert evaluation of a human-in-the-loop generative design approach for commercial game maps that incorporates multiple computational agents. The evaluation aims to gauge the extent to which such an approach could support and be accepted by human game designers and to determine whether the computational agents improve the overall design. To evaluate the approach, 11 game designers utilized the approach to design game levels with the computational agents both active and inactive. Eye-tracking, observational, and think-aloud data was collected to determine whether designers favored levels suggested by the computational agents. This data was triangulated with qualitative data from semi-structured interviews that were used to gather overall opinions of the approach. The eye-tracking data indicates that the participating game level designers showed a clear preference for levels suggested by the computational agents, however, expert designers in particular appeared to reject the idea that the computational agents are helpful. The perception of computational tools not being useful needs to be addressed if procedural content generation approaches are to fulfill their potential for the game industry.
References
[1]
Chad Adams, Hirav Parekh, and Sushil J. Louis. 2017. Procedural level design using an interactive cellular automata genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 85–86.
[2]
Saeema Ahmed, Ken M. Wallace, and Lucienne T. Blessing. 2003. Understanding the differences between how novice and experienced designers approach design tasks. Res. Eng. Des. 14, 1 (2003), 1–11.
[3]
Alberto Alvarez, Steve Dahlskog, Jose Font, Johan Holmberg, Chelsi Nolasco, and Axel Österman. 2018. Fostering creativity in the mixed-initiative evolutionary dungeon designer. In Proceedings of the 13th International Conference on the Foundations of Digital Games. ACM, 1–8.
[4]
Alexander Baldwin, Steve Dahlskog, Jose M. Font, and Johan Holmberg. 2017. Mixed-initiative procedural generation of dungeons using game design patterns. In Proceedings of the Conference on Computational Intelligence and Games. IEEE, 25–32.
[5]
Peter Bentley. 1999. Aspects of evolutionary design by computers. In Advances in Soft Computing, R. Roy, T. Furuhashi and P. K. Chawdhry (eds.). Springer, London, UK, 99–118.
[6]
Marcelo Bernal, John R. Haymaker, and Charles Eastman. 2015. On the role of computational support for designers in action. Des. Stud. 41, Part B (2015), 163–182.
[7]
L. Cardamone, D. Loiacono, and P. L. Lanzi. 2011. Interactive evolution for the procedural generation of tracks in a high-end racing game. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation. ACM, 395–402.
[8]
L. Cardamone, G. N. Yannakakis, J. Togelius, and P. L. Lanzi. 2001. Evolving interesting maps for a first person shooter. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer, 63–72.
[9]
Luigi Cardamone, Pier Luca Lanzi, and Daniele Loiacono. 2015. TrackGen: An interactive track generator for TORCS and Speed-Dreams. Appl. Soft Comput. 28 (2015), 550–558.
[10]
Elizabeth Charters. 2003. The use of think-aloud methods in qualitative research an introduction to think-aloud methods. Brock Educ. J. 12, 2 (2003), 68–82.
[11]
F. Cifaldi. 2006. Analysts: FPS “Most Attractive” genre for publishers. Retrieved from http://www.gamasutra.com/php-bin/news_index.php?story=8241.
[12]
A. M. Connor, T. J. Greig, and J. Kruse. 2017. Evaluating the impact of procedurally generated content on game immersion. Comput. Games J. 6, 4 (2017), 209–225.
[13]
A. M. Connor, T. J. Grieg, and J. Kruse. 2018. Evolutionary generation of game levels. EAI Trans. Creat. Technol. 5, 15 (2018), e4.
[14]
Michael Cook, Simon Colton, and Jeremy Gow. 2017. The ANGELINA videogame design system—Part II. IEEE Trans. Comput. Intell. AI Games 9, 3 (2017), 254–266.
[15]
Rui Craveirinha, Nuno Barreto, and Licinio Roque. 2016. Towards a taxonomy for the clarification of PCG actors’ roles. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. ACM, 244–253.
[16]
Daniel Michelon De Carli, Fernando Bevilacqua, Cesar Tadeu Pozzer, and Marcos Cordeiro d'Ornellas. 2011. A survey of procedural content generation techniques suitable to game development. In Proceedings of the Brazilian Symposium on Games and Digital Entertainment, IEEE Computer Society, 26–35.
[17]
Alena Denisova and Paul Cairns. 2015. The placebo effect in digital games: Phantom perception of adaptive artificial intelligence. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. ACM, 23–33.
[18]
Adrian Diaz-Furlong Hector and Luisa Solis-Gonzalez Cosio Ana. 2013. An approach to level design using procedural content generation and difficulty curves. In Proceedings of the Conference on Computational Intelligence in Games. IEEE, 1–8.
[19]
Andrew T. Duchowski. 2017. Eye tracking Methodology: Theory and Practice. Springer, London, UK.
[20]
Philip Galanter. 2012. Computational aesthetic evaluation: Past and future. In Computers and Creativity, J. McCormack and M. d'Inverno (Eds.). Springer, Berlin, 255–293.
[21]
Riccardo Galdieri, Alessandro Longobardi, Michele De Bonis, and Marcello Carrozzino. 2021. Users’ evaluation of procedurally generated game levels. In Proceedings of the International Conference on Augmented Reality, Virtual Reality and Computer Graphics. Springer, 44–52.
[22]
Edoardo Giacomello, Pier Luca Lanzi, and Daniele Loiacono. 2018. Doom level generation using generative adversarial networks. In Proceedings of the Games, Entertainment, Media Conference (GEM’18). IEEE, 316–323.
[23]
Joseph H. Goldberg and Xerxes P. Kotval. 1999. Computer interface evaluation using eye movements: Methods and constructs. Int. J. Ind. Ergon. 24, 6 (1999), 631–645.
[24]
Matthew Guzdial and Mark Riedl. 2018. Automated game design via conceptual expansion. In Proceedings of the 14th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 31–37.
[25]
Erin Jonathan Hastings, Ratan K. Guha, and Kenneth O. Stanley. 2009. Automatic content generation in the galactic arms race video game. IEEE Trans. Comput. Intell. AI Games 1, 4 (2009), 245–263.
[26]
Monique M. Hennink and Bonnie N. Kaiser. 2020. Saturation in qualitative research. In SAGE Research Methods Foundations, Paul Atkinson, Sara Delamont, Alexandru Cernat, Joseph W. Sakshaug and Richard A. Williams (Eds.). SAGE Publications Limited, Thousand Oaks, CA. Retrieved from https://methods.sagepub.com/foundations/saturation-in-qualitative-research.
[27]
Morten Hertzum. 2010. Images of usability. Int. J. Human–Comput. Interact. 26, 6 (2010), 567–600.
[28]
K. Hullett and J. Whitehead. 2010. Design patterns in FPS levels. In Proceedings of the 5th International Conference on the Foundations of Digital Games. ACM, 78–85.
[29]
Aki Järvinen. 2009. Games without Frontiers: Methods for Game Studies and Design. VDM Verlag, Saarbrücken, Germany.
[30]
Ming Jiang and Li Zhang. 2021. An interactive evolution strategy based deep convolutional generative adversarial network for 2D video game level procedural content generation. In Proceedings of the International Joint Conference on Neural Networks (IJCNN). IEEE, 1–6.
[31]
Anna Kantosalo and Hannu Toivonen. 2016. Modes for creative human-computer collaboration: Alternating and task-divided co-creativity. In Proceedings of the 7th International Conference on Computational Creativity. Association for Computational Creativity, 77–84.
[32]
Manolya Kavakli and John S. Gero. 2002. The structure of concurrent cognitive actions: A case study on novice and expert designers. Des. Stud. 23, 1 (2002), 25–40.
[33]
Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. 2020. PCGRL: Procedural content generation via reinforcement learning. In Proceedings of the 16th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 95–101.
[34]
Mohammad T. Khasawneh, Shannon R. Bowling, Xiaochun Jiang, Anand K. Gramopadhye, and Brian J. Melloy. 2003. A model for predicting human trust in automated systems. In Proceedings of the 8th Annual International Conference on Industrial Engineering – Theory, Applications and Practice. Citeseer, 216–222.
[35]
A. Kosorukoff. 2001. Human based genetic algorithm. In Proceedings of the International Conference on Systems, Man, and Cybernetics. IEEE, 3464–3469.
[36]
J. Kruse and A. M. Connor. 2015. Multi-agent evolutionary systems for the generation of complex virtual worlds. EAI Endors. Trans. Creat. Technol. 2, 5 (2015), e5.
[37]
J. Kruse, A. M. Connor, and S. Marks. 2021. An interactive multi-agent system for game design. Comput. Games J. 10, (2021), 41–63.
[38]
J. Kruse, R. Sosa, and A. M. Connor. 2016. Procedural urban environments for FPS games. In Proceedings of the Australasian Computer Science Week Multiconference. ACM, 1–5.
[39]
Jan Kruse. 2014. Interactive Evolutionary Computation in Design Applications for Virtual Worlds. Thesis. Auckland University of Technology. Retrieved from http://aut.researchgateway.ac.nz/handle/10292/8593.
[40]
P. L. Lanzi, D. Lomacono, and R. Stucchi. 2014. Evolving maps for match balancing in first person shooters. In Proceedings of the Conference on Computational Intelligence and Games. IEEE, 1–8.
[41]
Antonios Liapis, Gillian Smith, and Noor Shaker. 2016. Mixed-initiative content creation. In Procedural Content Generation in Games, Noor Haker, Julian Togelius, and Mark J. Nelson (Eds.). Springer, Cham, Switzerland, 195–214.
[42]
Antonios Liapis, Georgios N. Yannakakis, Constantine Alexopoulos, and Phil Lopes. 2016. Can computers foster human users’ creativity? Theory and praxis of mixed-initiative co-creativity. Digit. Cult. Educ. 8, 2 (2016), 136–153.
[43]
Antonios Liapis, Georgios N. Yannakakis, and Julian Togelius. 2013. Sentient sketchbook: Computer-assisted game level authoring. In Proceedings of the 8th International Conference on the Foundations of Digital Games. ACM, 213–220.
[44]
Antonios Liapis, Georgios Yannakakis, and Julian Togelius. 2013. Towards a generic method of evaluating game levels. In Proceedings of the 9th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 30–36.
[45]
Ken Museth. 2014. Hierarchical digital differential analyzer for efficient ray-marching in OpenVDB. In Proceedings of the ACM SIGGRAPH Talks. ACM, 1–1.
[46]
P. T. Ølsted, B. Ma, and S. Risi. 2015. Interactive evolution of levels for a competitive multiplayer FPS. In Proceedings of the Congress on Evolutionary Computation. IEEE, 1527–1534.
[47]
Lisa M. Osbeck and Stephen L. Antczak. 2021. Generalizability and qualitative research: A new look at an ongoing controversy. Qual. Psychol. 8, 1 (2021), 62.
[48]
Denis Pallez, Philippe Collard, Thierry Baccino, and Laurent Dumercy. 2007. Eye-tracking evolutionary algorithm to minimize user fatigue in IEC applied to interactive one-max problem. In Proceedings of the 9th Annual Conference Companion on Genetic and Evolutionary Computation. ACM, 2883–2886.
[49]
Yan Pei and Hideyuki Takagi. 2018. Research progress survey on interactive evolutionary computation. J. Ambient Intell. Humaniz. Comput. (2018), 1–14.
[50]
Michele Pirovano, Renato Mainetti, and Daniele Loiacono. 2015. Volcano: An interactive sword generator. In Proceedings of the Games Entertainment Media Conference. IEEE, 1–8.
[51]
Alexandru Predescu and Mariana Mocanu. 2020. A data driven survey of video games. In Proceedings of the 12th International Conference on Electronics, Computers and Artificial Intelligence (ECAI). IEEE, 1–6.
[52]
Willian M. P. Reis, Levi H. S. Lelis, and Ya'akov Kobi Gal. 2015. Human computation for procedural content generation in platform games. In Proceedings of the Conference on Computational Intelligence and Games. IEEE, 99–106.
[53]
Luiz Rodrigues and Jacques Brancher. 2019. Playing an educational game featuring procedural content generation: Which attributes impact players’ curiosity? RENOTE 17, 1 (2019), 254–263.
[54]
Johnny Saldaña. 2016. The Coding Manual for Qualitative Researchers (2nd ed.). Sage.
[55]
Jacob Schrum, Jake Gutierrez, Vanessa Volz, Jialin Liu, Simon Lucas, and Sebastian Risi. 2020. Interactive evolution and exploration within latent level-design space of generative adversarial networks. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 148–156.
[56]
Stefan Seidel, Nicholas Berente, Aron Lindberg, Kalle Lyytinen, and Jeffrey V. Nickerson. 2018. Autonomous tools and design: A triple-loop approach to human-machine learning. Commun. ACM 62, 1 (2018), 50–57.
[57]
Noor Shaker, Gillian Smith, and Georgios N. Yannakakis. 2016. Evaluating content generators. In Procedural Content Generation in Games, Noor Shaker, Julian Togelius, and Mark J. Nelson (Eds.). Springer, Cham, Switzerland, 215–224.
[58]
Tanya Short and Tarn Adams. 2017. Procedural Generation in Game Design. CRC Press, Boca Raton, FL.
[59]
Mike Stout. 2016. A beginner's guide to designing video game levels. Retrieved from https://gamedevelopment.tutsplus.com/tutorials/a-beginners-guide-to-designing-video-game-levels–cms-25662.
[60]
Adam Summerville, Julian R. H. Mariño, Sam Snodgrass, Santiago Ontañón, and Levi H. S. Lelis. 2017. Understanding Mario: An evaluation of design metrics for platformers. In Proceedings of the 12th International Conference on the Foundations of Digital Games. ACM, 1–10.
[61]
H. Takagi. 2001. Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation. Proc. IEEE 89, 9 (2001), 1275–1296.
[62]
J. Togelius, Mike Preuss, Nicola Beume, Simon Wessing, Johan Hagelbäck, Georgios N. Yannakakis, and Corrado Grappiolo. 2013. Controllable procedural map generation via multiobjective evolution. Genet. Program. Evolvable Mach. 14, 2 (2013), 245–277.
[63]
Julian Togelius, Renzo De Nardi, and Simon M. Lucas. 2006. Making racing fun through player modeling and track evolution. In Proceedings of the Workshop on Adaptive Approaches for Optimizing Player Satisfaction in Computer and Physical Games.
[64]
Julian Togelius, Emil Kastbjerg, David Schedl, and Georgios N. Yannakakis. 2011. What is procedural content generation? Mario on the borderline. In Proceedings of the 2nd International Workshop on Procedural Content Generation in Games. ACM, 1–6.
[65]
Eric J. Toy, Jaya V. H. H. Kummaragunta, and Jin Soung Yoo. 2018. Large-scale cross-country analysis of steam popularity. In Proceedings of the International Conference on Computational Science and Computational Intelligence. IEEE, 1054–1058.
[66]
Lev. S. Vygotskiĭ. 2012. Thought and Language. The MIT Press.
[67]
Georgios N. Yannakakis and J. Togelius. 2011. Experience-driven procedural content generation. IEEE Trans. Affect. Comput. 2, 3 (2011), 147–161.
[68]
Jichen Zhu, Antonios Liapis, Sebastian Risi, Rafael Bidarra, and G. Michael Youngblood. 2018. Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation. In Proceedings of the Conference on Computational Intelligence and Games. IEEE, 1–8.
Index Terms
- Evaluation of a Multi-agent “Human-in-the-loop” Game Design System
Recommendations
PCG-based game design: creating Endless Web
FDG '12: Proceedings of the International Conference on the Foundations of Digital GamesThis paper describes the creation of the game Endless Web, a 2D platforming game in which the player's actions determine the ongoing creation of the world she is exploring. Endless Web is an example of a PCG-based game: it uses procedural content ...
Supporting human-machine cooperative design in multi-agent systems
ACMOS'06: Proceedings of the 8th WSEAS international conference on Automatic control, modeling & simulationTo utilize the flexible problem solving capability of human designers, we aim at extending our previous approach for cooperative problem solving among computational agents toward a human-machine cooperative design system within the framework of Multi-...
Design patterns for multi-agent programming
Various agent-based programming languages and frameworks have been proposed to support the development of autonomous agents and multi-agent systems. They have provided a valuable contribution to the identification and operationalisation of agent ...
Comments
Information & Contributors
Information
Published In
September 2022
257 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 July 2022
Online AM: 30 April 2022
Accepted: 01 March 2022
Revised: 01 January 2022
Received: 01 June 2021
Published in TIIS Volume 12, Issue 3
Check for updates
Author Tags
Qualifiers
- Research-article
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 515Total Downloads
- Downloads (Last 12 months)157
- Downloads (Last 6 weeks)16
Reflects downloads up to 25 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format