AbadIA - sphere it krakow 2019
•
0 likes•114 views
The process of building an AI looks like it is so glamorous but is a long process, and at the end of the day, the tasks related to the AI model are just 5% or less of the project. We will see how to start an AI project from zero: defining the objectives, creating the architecture, building the game interfaces, massive data pipelines, defining model strategies, how to parallelize everything, etc. The "the abbey of crime" is an adamant 8-bit game. This game is more complicated than Montezuma Revenge and is a perfect challenge for an AI. Its complexity is about 10^1000 legal moves to solve it. As AI technology, we will use Reinforcement Learning using Deep Neural Networks and Monte Carlo Tree Search. The takeaways of this talk will be: understanding all the processes involved to create an AI and learning the basics of Reinforcement Learning.
Report
Share
AbadIA - sphere it krakow 2019
- 1. AbadIA Using Reinforcement Learning to play and solve the abbey of crime Juantomás García - Chief Envisioning Officer - Sngular
- 3. Who I am Juantomás García ( 0-) •Chief Envisioning Officer @ Sngular •GDEx2 (Google Developer Expert) for cloud and Machine Learning •#AbadIA Cheer Leader Others •Co-Author of the first Spanish free software book “La Pastilla Roja” •Former President of Hispalinux (Spanish Linux User Group) •Organizer of the Machine Learning Spain and GDG Cloud Madrid.
- 4. What we will see
- 6. ? My buddy Sebas told me: Is it possible to create an AI that can learn to play and solve the game? It’s a really crazy idea with an insane cult 8-bit game, but I like it “ ”
- 7. The Game:The Abbey of the Crime
- 8. First 8-bit RPG in pseudo 3D (2.5D) It was at 1987 and this game is a kind of legend in the video games world. Based in Umberto Eco book “In the name of the rose” Do you know the game? THE GAME
- 9. Size of the program: 87 Kb That include: • Code • Graphics • Music • Maps Do you know the game? THE GAME
- 10. The size of this avatar is: 87Kb Do you know the game? THE GAME
- 11. It was programmed using Z80 assembler for Amstrad CPC and Sinclair Spectrum ABOUT THE GAME Was programmed by Paco Menéndez Graphics was designed by Juan Delcán
- 12. The Plan
- 13. The plan: Make an AI that can learn to play and solve the game
- 14. What we had
- 16. Manuel Abadía disassembled the original Amstrad CPC game rom. He documented all the code, marked the graphics, sounds, etc. GitHub page: https://bit.ly/2I9TMZk Original Code
- 17. Current Version
- 18. Manuel Abadía make a port to C++ of the game using his special video games framework: Vigasoco Sebastian Blanes fork the project to use SDL. VigasocoSDL was born. Now the abbey can be played in chrome, PS3, CC, linux, MacOS, Windows, etc. VigasocoSDL GitHub page: https://github.com/luzbel/VigasocoSDL/wiki GAME EVOLUTION
- 19. How difficult is the Challenge
- 20. Number of atoms in our universe? HOW DIFFICULT IS THE CHALLENGE?
- 21. 10^80 Number of atoms in our universe (Hawking said there are more than one universe!!) HOW DIFFICULT IS THE CHALLENGE?
- 22. 10^80 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves? HOW DIFFICULT IS THE CHALLENGE?
- 23. 10^12010^80 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves HOW DIFFICULT IS THE CHALLENGE?
- 24. 10^12010^80 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves? HOW DIFFICULT IS THE CHALLENGE?
- 25. 10^12010^80 10^761 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves HOW DIFFICULT IS THE CHALLENGE?
- 26. 10^12010^80 10^761 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves AbadIA moves needs to solve the game? HOW DIFFICULT IS THE CHALLENGE?
- 27. 10^12010^80 10^761 10^10k Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves AbadIA legal moves where N is the depth of the game you wish to check. For N= 10000 is 10^10000 HOW DIFFICULT IS THE CHALLENGE?
- 28. Universe 100000000000000000000000000000000000000000000000000000000000000000000000000000000 Chess 1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 GO 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000 The Abbey of Crime 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 Just for the guys that not attended enought math classes.
- 29. The AI & Human Players only know if the game is solved after 10.000 actions.
- 30. HOW DIFFICULT IS THE CHALLENGE?
- 31. HOW DIFFICULT IS THE CHALLENGE?
- 32. HOW DIFFICULT IS THE CHALLENGE?
- 33. HOW DIFFICULT IS THE CHALLENGE?
- 34. First Steps
- 36. How to interact with the game. You will need lots of tools before starting to create the AI 1 2 3 How you get and save the game information. How to play millions of games. AI TOOLS
- 37. You will need send actions, get game info, reset it, etc. How to interact with the game?? HOW TO INTERACT
- 38. First version with too BeersHOW TO INTERACT
- 39. You may use a fake keyboard driver, a unix pipe, a socket, but do you know what is the really cool way to do this: HOW TO INTERACT
- 40. An embedded web server inside the game !!!! HOW TO INTERACT
- 41. Now we have a REST based API. Two way communication. We send actions - Moves - Resets - Save/Load states We got information - State dumps - Actions - Checkpoints HOW TO INTERACT WITH THE GAME
- 42. curl -X POST http:!//localhost:4477/abadIA/game/current/actions/RIGHT or curl -X GET http:!//localhost:4477/abadIA/game/current —H 'accept: application/json’ or Use python request library AS EASY AS
- 43. Demo game engine + web server + curl
- 45. At the beginning a laptop was enough. But very soon you need more CPU/ GPU. Then product like Google Cloud is your best ally. We had created a few Dockers, so now we can execute lots of instances of the game in parallel. note: If we use Google Cloud services like GKE, we can launch hundred of games in parallel. HOW TO SCALE IT
- 46. We recollect a lot of information - Game Info (timestamps, rewards, bonus, obsequium) - Games moves (state, action, reward, new_state) - Checkpoints (to restore the game at an interesting time) - ML Models (for recovering good models o just make a benchmark) GATHERING INFORMATION
- 47. It takes a lot of time to get all the parts working all together. Building tools, testing every piece, every option. Sometimes I feel like I was Mario Bros. GATHERING INFORMATION
- 49. Github Repo: https://www.github.com/LaAbadIAdelCrimen/ abadia-gym One the most frequently used tool is OpenAI Gym So we design an AbadIA gym The gym is a standard place to train and interact with Reinforcement Learning agents. In our project the gym is framework to wrap the game engine. PLAYGROUND FOR PLAYING
- 50. A game server with REST API An openAI Gym Enought hardware resources So what’s the next step SO WE HAVE
- 51. An small intro to Reinforcement Learning
- 52. A RL agent is a program that interacts with an environment, in our case a OpenAI gym for AbadIA, and learn from observations and rewards. CREATE A RL AGENT
- 53. And that looks like:
- 54. Demo Time + Video
- 56. Random Q*tables Q*Learning DQN CNN A3C MiniGames REINFORCEMENT LEARNING STRATEGIES
- 57. Current Version
- 58. Simple Neural Network Model
- 59. Monte Carlo Tree Search
- 61. Game Engine & Agent Containers
- 62. First we create the first 2 containers for: ● AbadIA Game Engine ● AbadIA Agent
- 64. Cluster Features: ● Preemtible ● n1-standard-1 (7€ month)
- 67. AutoGaming
- 69. Storage of the info: ● Game Information ● Actions - States - Rewards ● Snapshots ● Models
- 70. How to Integrate the all the information: ● Serverless ● Easy ● Fast
- 71. Advanced and Massive Analytics: ● Game Information ● Actions - States - Rewards ● Snapshots ● Even some ML (LRs)
- 72. Notebooks
- 73. How to manipulate and visualize with Datalabs or Colab: ● Notebooks ● Managed ● Integrated with Google Drive / Bigquery
- 74. AI Frameworks
- 75. How to training and serve models at scale: ● Tensorflow / Keras ● ML Engine ● AutoML
- 76. Orchestrating
- 77. How to execute hundreds of instances: ● Kubernetes ● Google Containers
- 78. Show me the code (how to collaborate)
- 80. Join US, we’ll share the glory with you
- 82. Thank You!!