Answers.Body: <p><b> About the environments </b> <br>
  For the controller part of your question, I would advice looking at openAI gym.
  <a href="https://www.gymlibrary.ml/content/environment_creation/" rel="nofollow noreferrer">https://www.gymlibrary.ml/content/environment_creation/</a> #how to make your own gym enviroment
  <br>
  Those gym enviroments work kinda like this</p>
  <pre><code>env = gym.make(&quot;ALE/SpaceInvaders-v5&quot;)
  observation = env.reset()
  
  observation, reward, done =env.step(action) 
  </code></pre>
  <p>Where your observation can be pixels,ram dumps, etc. The actions can be internally mapped to key presses. (but the internals don't really matter that much) Gym environments are just an easy way to then have your agent take in the observations and map them to actions. So your game can be stepped through so you don't have to worry about a complicated way to integrate the AI and keeping it synced.
  <br>
  There are already a bunch of artari games included (also ms pacman) <a href="https://www.gymlibrary.ml/environments/atari/" rel="nofollow noreferrer">https://www.gymlibrary.ml/environments/atari/</a></p>
  <p><b>About the AI</b></p>
  <p>Since you want to learn directly from the pixels or other more complex data types I advice you to read this:</p>
  <p>Deep Reinforcement Learning, a textbook (arXiv:2201.02135)
  <a href="https://arxiv.org/abs/2201.02135" rel="nofollow noreferrer">https://arxiv.org/abs/2201.02135</a></p>
  <p>If you just want some simple tutorial just read &quot;Hands-On Reinforcement Learning with Python&quot;, then you can try to implement something like DQN , with some cnn architecture. (similar to how they did in arXiv:1312.5602 &quot;Playing Atari with Deep Reinforcement Learning&quot;).
  Since DQN is model free, off policy and relatively easy to implement.
  (cool thing about off policy is that your agent doesnt have to interact with the environment as it doesnt need to sample &quot;experience&quot; using only it's current policy (way of choosing actions given a state) so you can even play the game itself and collect information (a,s,s',r) while playing and train your agent on that (to nudge it in the right direction if it gets stuck))</p>
  <p>if you're not a fan of gradient descent based methods, you can also use methods like ES (evolutionary strategies). As that way you can directly optimize for a reward. (it's a genetic algorithm that scales well in both compute and dimensionality)  (arXiv:1703.03864 Evolution Strategies as a
  Scalable Alternative to Reinforcement Learning)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T13:45:52.120Z
Answers.Text: (a,s,s',r) stands for action, state, next state, reward.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T18:57:01.700Z
Answers.Text: I "advice" you to ask for clarifications before providing an answer because the question and post are currently unclear.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:14:31.503Z
Answers.Text: It's quite clear what he wants, he is just very unfamiliar with rl. He is dissatisfied with the idea of simple AI that follows some simple rules and wants some separate agent that looks at pixels (like the player) and also can only do the same actions as the player (while poorly formulated it's quite clear). So seeing that I point him into the right direction being deep rl textbooks (since he wants the AI to learn pacman from the pixels).
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:20:31.080Z
Answers.Text: The OP writes "taking in pixel values **or something else.**", so why are you so sure that the OP wants something that takes (only) pixel values? The OP may be unfamiliar with RL, but that doesn't mean that the question is clear. The title says "how to create an AI controller for pacman", which is different from "create something like a separated **“controller” thread**". The OP has already looked into implementations of RL for pacman, so I assume he's not so unfamiliar with it, otherwise, they would not have understood anything in those implementations. In my view, the post is just unclear.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:24:36.143Z
Answers.Text: I don't give an answer that takes only pixel values. I point him to deep rl with that he has his controller that can interact with openAI gym to play his pacman directly from pixel values or ram states or whatever he wants.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:26:20.883Z
Answers.Text: It's clear that the controller thread is a separate thread/system interacting with the game in a complex way. (through observing pixels values or different (but the data type is complex not deeply integrated into the game))
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:27:17.830Z
Answers.Text: Ok, I would edit your post then not to start it with "Since you want to learn directly from the pixels". To me, it seems that the OP is suggesting to use multi-threading for learning and interaction (that seems to be the crux of the post), but I am not sure because the post is not fully clear.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:34:43.370Z
Answers.Text: You're right about the "Since you want to learn directly from the pixels" I changed that to a more fitting sentence.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:51:18.593Z
Answers.Text: "The OP may be unfamiliar with RL, but that doesn't mean that the question is clear". Just that someone can't find the correct terms doesn't mean they are not clear.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T19:54:30.370Z
Answers.Text: "The OP has already looked into implementations of RL for pacman, so I assume he's not so unfamiliar with it, otherwise, they would not have understood anything in those implementations". I point him clearly to deep rl (what is different), as classical rl has problems with high dimensional spaces (input spaces) like pixels or other complex datatypes as they tend to be tabular methods. (because of that they are generally like how OP said, like any other AI in games (meaning simple AI) that are deeply integrated with the game at hand to reduce the dimensionality )
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T11:58:09.247Z
Answers.Text: Hello there. I’m sorry the question is confusing, I’ll try to make it more clear. 
  Like @hal9000 said, simple AI is deeply integrated with the game at hand to reduce the dimensionality. I want to create an AI , which can look at the pixel values or “something” else, some other dataype, just like a human does. 
  I was asking resources around that. How can I create a controller - which runs on a separate thread - learn by taking in pixel values, and controls that game with that. Not by being a part of the “game” itself - like other simple AIs (like the enemies)
Answers.UserDisplayName: R3sist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T13:09:22.753Z
Answers.Text: @R3sist I think some of the confusion is that although the term "game AI" is often used for enemies in video games, and they may even use some old school AI such as pathfinding via A* search, many experts here probably won't even register that as being AI (despite the name)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-21T13:42:36.600Z
Answers.LastActivityDate: 2022-06-22T14:13:59.280Z
Answers.LastEditDate: 2022-06-22T14:13:59.280Z
Answers.UserDisplayName: hal9000
Body: <p>How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game?</p>
  <p>It takes the pixel values (or likewise) to learn, and play the game, just like a human does. Normally, AIs like enemies - are part of the game code, which learn and play by using the internal state of the game, just like the “enemy” AI.</p>
  <p>How can I create this abstraction between the game and the agent?</p>
  <p>The most obvious example would be : (I don’t know how viable this is) -
  The agent takes in pixel values of the game - learns - and sends “keyboard” inputs to that thread just like a human would, to play the game.</p>
  <p>Can someone point me in the right direction - as to how to do something like this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-21T18:46:51.870Z
Comments.Text: I don't understand this: "_where the “agent” which learns is part of the game code itself. Meaning it’s just like any other AI in the game, but it learns dynamically._". What do you mean by 1. "just like any other AI in the game" (i.e. which "any other AI in the game" are you referring to? Are you referring to rule-based bots?), 2. "is part of the game code itself"? Do you mean that there are no abstractions that separate the "game" from the "agent"? If that's the case, how is that not just a programming issue (i.e. some bad programmer that doesn't know what abstractions are)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-21T18:47:45.490Z
Comments.Text: From "separated “controller” thread", it seems that you were talking about creating multiple threads (one for the agent and one for the game, whatever that means).  If that's the case, rewrite your question to make that initial paragraph clearer. Maybe also provide the link to the code you were referring to. Which models or algorithms are you specifically referring to? Also, the title doesn't really match the problem or question in the title. One question per post, so the question in the title should be equal to the equation in the body. Edit your post to fix all these issues!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-21T18:53:42.960Z
Comments.Text: The answer below just seems a general answer to "how can I create a game AI agent". It doesn't even focus on pacman, threads, or whatever you're really asking here. It assumes that you want to learn from pixels, while you write "taking in pixel values **or something else**.". So, this post and the answer below couldn't be more problematic, and I don't have an idea of why people upvoted this!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-21T19:57:50.783Z
Comments.Text: Valid answer to a valid question. He just wants to know about deep rl (as he looks for something that dynamically plays games from pixel values (or other datatypes) and chooses actions like a player would (no tricks)), and not classical rl that is deeply integrated with the game (but learns dynamically). What is almost like simple rule based AI you generally find in games.
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-22T08:09:45.553Z
Comments.Text: Your right the question is quite confusing.
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-22T12:06:06.687Z
Comments.Text: I have edited the question, I hope it makes better sense now.
Comments.UserDisplayName: R3sist
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-22T13:16:11.103Z
Comments.Text: I think it is clearer what you are asking about now. The separate thread thing is a bit of a red herring (because in practice this is not the usual approach), but an interesting part of the question nonetheless. I am voting to reopen.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-22T13:36:32.853Z
Comments.Text: @R3sist do you want to know about the different types of AI or more like in you have the AI and you want to know how to integrate it with your game? (you might want to look at openAI gym and libraries based upon that) It has environments (like ms pacman, space invaders, etc.) where you can do obs= env.step(action) where obs can be pixels, ram dumps or else. But it doesnt implement the AI for you (the books and resources I had in my answer will help you with that).
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-22T14:07:06.373Z
Comments.Text: @R3sist I updated my answer, does it answer you question or is it missing something? :) Then I can improve my answer.
Comments.UserDisplayName: hal9000
Tags: <reinforcement-learning><deep-learning><deep-rl><q-learning>
Title: How do I create an AI controller for Pacman?
UserDisplayName: R3sist
ThreadText: # How do I create an AI controller for Pacman?
  
  (Asked by: username_0 on 2022-06-21)
  
  How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game?
  
  It takes the pixel values (or likewise) to learn, and play the game, just like a human does. Normally, AIs like enemies - are part of the game code, which learn and play by using the internal state of the game, just like the “enemy” AI.
  
  How can I create this abstraction between the game and the agent?
  
  The most obvious example would be : (I don’t know how viable this is) -
  The agent takes in pixel values of the game - learns - and sends “keyboard” inputs to that thread just like a human would, to play the game.
  
  Can someone point me in the right direction - as to how to do something like this?
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl, q-learning
  
  > **Comments:**
  > **username_1**: I don't understand this: "_where the “agent” which learns is part of the game code itself. Meaning it’s just like any other AI in the game, but it learns dynamically._". What do you mean by 1. "just like any other AI in the game" (i.e. which "any other AI in the game" are you referring to? Are you referring to rule-based bots?), 2. "is part of the game code itself"? Do you mean that there are no abstractions that separate the "game" from the "agent"? If that's the case, how is that not just a programming issue (i.e. some bad programmer that doesn't know what abstractions are)?  
  > **username_1**: From "separated “controller” thread", it seems that you were talking about creating multiple threads (one for the agent and one for the game, whatever that means).  If that's the case, rewrite your question to make that initial paragraph clearer. Maybe also provide the link to the code you were referring to. Which models or algorithms are you specifically referring to? Also, the title doesn't really match the problem or question in the title. One question per post, so the question in the title should be equal to the equation in the body. Edit your post to fix all these issues!  
  > **username_1**: The answer below just seems a general answer to "how can I create a game AI agent". It doesn't even focus on pacman, threads, or whatever you're really asking here. It assumes that you want to learn from pixels, while you write "taking in pixel values **or something else**.". So, this post and the answer below couldn't be more problematic, and I don't have an idea of why people upvoted this!  
  > **username_2**: Valid answer to a valid question. He just wants to know about deep rl (as he looks for something that dynamically plays games from pixel values (or other datatypes) and chooses actions like a player would (no tricks)), and not classical rl that is deeply integrated with the game (but learns dynamically). What is almost like simple rule based AI you generally find in games.  
  > **username_2**: Your right the question is quite confusing.  
  > **username_0**: I have edited the question, I hope it makes better sense now.  
  > **username_3**: I think it is clearer what you are asking about now. The separate thread thing is a bit of a red herring (because in practice this is not the usual approach), but an interesting part of the question nonetheless. I am voting to reopen.  
  > **username_2**: @username_0 do you want to know about the different types of AI or more like in you have the AI and you want to know how to integrate it with your game? (you might want to look at openAI gym and libraries based upon that) It has environments (like ms pacman, space invaders, etc.) where you can do obs= env.step(action) where obs can be pixels, ram dumps or else. But it doesnt implement the AI for you (the books and resources I had in my answer will help you with that).  
  > **username_2**: @username_0 I updated my answer, does it answer you question or is it missing something? :) Then I can improve my answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-21. Score: 2 | Accepted answer
  
  **About the environments**    
  For the controller part of your question, I would advice looking at openAI gym.
  <https://www.gymlibrary.ml/content/environment_creation/> #how to make your own gym enviroment
    
  Those gym enviroments work kinda like this
  
  ```
  env = gym.make("ALE/SpaceInvaders-v5")
  observation = env.reset()
  
  observation, reward, done =env.step(action) 
  
  ```
  
  Where your observation can be pixels,ram dumps, etc. The actions can be internally mapped to key presses. (but the internals don't really matter that much) Gym environments are just an easy way to then have your agent take in the observations and map them to actions. So your game can be stepped through so you don't have to worry about a complicated way to integrate the AI and keeping it synced.
    
  There are already a bunch of artari games included (also ms pacman) <https://www.gymlibrary.ml/environments/atari/>
  
  **About the AI**
  
  Since you want to learn directly from the pixels or other more complex data types I advice you to read this:
  
  Deep Reinforcement Learning, a textbook (arXiv:2201.02135)
  <https://arxiv.org/abs/2201.02135>
  
  If you just want some simple tutorial just read "Hands-On Reinforcement Learning with Python", then you can try to implement something like DQN , with some cnn architecture. (similar to how they did in arXiv:1312.5602 "Playing Atari with Deep Reinforcement Learning").
  Since DQN is model free, off policy and relatively easy to implement.
  (cool thing about off policy is that your agent doesnt have to interact with the environment as it doesnt need to sample "experience" using only it's current policy (way of choosing actions given a state) so you can even play the game itself and collect information (a,s,s',r) while playing and train your agent on that (to nudge it in the right direction if it gets stuck))
  
  if you're not a fan of gradient descent based methods, you can also use methods like ES (evolutionary strategies). As that way you can directly optimize for a reward. (it's a genetic algorithm that scales well in both compute and dimensionality) (arXiv:1703.03864 Evolution Strategies as a
  Scalable Alternative to Reinforcement Learning)
  
  > **Comments:**
  > **username_2**: (a,s,s',r) stands for action, state, next state, reward.  
  > **username_1**: I "advice" you to ask for clarifications before providing an answer because the question and post are currently unclear.  
  > **username_2**: It's quite clear what he wants, he is just very unfamiliar with rl. He is dissatisfied with the idea of simple AI that follows some simple rules and wants some separate agent that looks at pixels (like the player) and also can only do the same actions as the player (while poorly formulated it's quite clear). So seeing that I point him into the right direction being deep rl textbooks (since he wants the AI to learn pacman from the pixels).  
  > **username_1**: The OP writes "taking in pixel values **or something else.**", so why are you so sure that the OP wants something that takes (only) pixel values? The OP may be unfamiliar with RL, but that doesn't mean that the question is clear. The title says "how to create an AI controller for pacman", which is different from "create something like a separated **“controller” thread**". The OP has already looked into implementations of RL for pacman, so I assume he's not so unfamiliar with it, otherwise, they would not have understood anything in those implementations. In my view, the post is just unclear.  
  > **username_2**: I don't give an answer that takes only pixel values. I point him to deep rl with that he has his controller that can interact with openAI gym to play his pacman directly from pixel values or ram states or whatever he wants.  
  > **username_2**: It's clear that the controller thread is a separate thread/system interacting with the game in a complex way. (through observing pixels values or different (but the data type is complex not deeply integrated into the game))  
  > **username_1**: Ok, I would edit your post then not to start it with "Since you want to learn directly from the pixels". To me, it seems that the OP is suggesting to use multi-threading for learning and interaction (that seems to be the crux of the post), but I am not sure because the post is not fully clear.  
  > **username_2**: You're right about the "Since you want to learn directly from the pixels" I changed that to a more fitting sentence.  
  > **username_2**: "The OP may be unfamiliar with RL, but that doesn't mean that the question is clear". Just that someone can't find the correct terms doesn't mean they are not clear.  
  > **username_2**: "The OP has already looked into implementations of RL for pacman, so I assume he's not so unfamiliar with it, otherwise, they would not have understood anything in those implementations". I point him clearly to deep rl (what is different), as classical rl has problems with high dimensional spaces (input spaces) like pixels or other complex datatypes as they tend to be tabular methods. (because of that they are generally like how OP said, like any other AI in games (meaning simple AI) that are deeply integrated with the game at hand to reduce the dimensionality )  
  > **username_0**: Hello there. I’m sorry the question is confusing, I’ll try to make it more clear.  Like @username_2 said, simple AI is deeply integrated with the game at hand to reduce the dimensionality. I want to create an AI , which can look at the pixel values or “something” else, some other dataype, just like a human does.  I was asking resources around that. How can I create a controller - which runs on a separate thread - learn by taking in pixel values, and controls that game with that. Not by being a part of the “game” itself - like other simple AIs (like the enemies)  
  > **username_3**: @username_0 I think some of the confusion is that although the term "game AI" is often used for enemies in video games, and they may even use some old school AI such as pathfinding via A* search, many experts here probably won't even register that as being AI (despite the name)  
  
  ---
  
  