Body: <p>Assuming we had an unlimited time to train a model and a very powerful machine to use our model in real-time (hello quantum computer), I'd like to know why no one could achieve to build an AI able to play a FPS, using <strong>ONLY</strong> pixels shown on the screen.</p>
  
  <p><strong>Disclaimer:</strong> I am not tackling this problem and neither am I planning on doing such a thing, this is pure speculation and curiosity.</p>
  
  <hr>
  
  <p>I read this great article: <a href="https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14456" rel="nofollow noreferrer">playing FPS Games with Deep Reinforcement Learning (2017)</a> (<em>Guillaume Lample, Devendra Singh Chaplot</em>) where they achieve a 4/1 kills/death ratio on Doom against bots. <strong>But this is 3 years old now</strong>.</p>
  
  <p>Here is a picture of their model:</p>
  
  <p><a href="https://i.sstatic.net/pyHve.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pyHve.png" alt="model"></a></p>
  
  <p>But they made 2 assumptions that we, humans, do not make when we are playing for the first time to a new game like Call of Duty or Battlefield:</p>
  
  <ol>
  <li><p><strong>Game feature augmentation</strong>. To train a part of their model <strong>they used the game engine to know if there is (or not) an enemy in the frame</strong> they are processing. We obviously can't do this with CoD or Battlefield, and we, as human, just "learn" to recognize an enemy without these informations.</p></li>
  <li><p><strong>Changing textures of several maps</strong> while training to generalize better the model (see 5.2 of the paper linked previously). To summarize, they trained the model with 10 maps changing texture of some elements to make the model generalize better. Then they tested it on 3 unknown maps. In real world (ie in the scenario where we base our training/testing exclusively on the pixels of the screen), we can't train a model with different textures on the same map. And humans are able to play a deathmatch on an unknow map without re-learning everything (detect enemies, move, hide, reloading...). We just need to construct a 3D model of the map in our head to play our best.</p></li>
  </ol>
  
  <hr>
  
  <p>Their agent "divides the problem into <strong>two phases: navigation (exploring the map to collect items and find enemies) and action (fighting enemies when they are observed), and uses separate networks</strong> for each phase of the game".</p>
  
  <p>Would it be wise to use more than 2 models? Let's say:</p>
  
  <ul>
  <li>1 CNN to detect enemies</li>
  <li>1 model to deal with space features (position/navigation of the agent and the enemies)</li>
  <li>1 model to choose the actions given all data previous models have found?</li>
  </ul>
  
  <p>We could train them independently, at the same time.</p>
  
  <p>I think we'd get better result by processing manually some features (using computer vision techniques) like the minimap to get know positions of enemies and number of ammo to feed as input of the last model (action decider).</p>
  
  <p>But there are other problem we'd get: there is a delay between the frame we choose to pull the trigger, the time the bullet hit the enemy and the time the "reward" appears on the screen (ex: "100 points, kill [nameOfLateEnemy]" appears after the 3th bullet, and if there is ping because we are playing online it may appear 100ms after). <strong>How to use reinforcement learning when we don't know exactly which action was the one getting the reward?</strong> (we can move the agent while changing the lok directon while pulling the trigger at the same time. It's the combination of these actions that is making the agent kill an enemy).</p>
  
  <p>If the 2 assumptions they made were easy to get rid of, they would be discarded already.<br>
  However detecting enemies is basically a simple CNN, and making the navigation network generalize certainly have solutions I can't think of right now but some researchers should have in this 3-year gap between the paper and today.</p>
  
  <h2>So why isn't there a model playing CoD or Battlefield better than humans? What am I missing?</h2>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-13T02:46:27.667Z
Comments.Text: I'll think on this and form a thorough answer. Some thoughts: Such an agent might simply learn to become an aim bot and exploit certain features of the game environment. Example: in Black Ops 3 on Xbox 360 some people used an aim bot mod that didn't do anything fancy. It moved to the center of the map then looked as fast as it could in all directions (very fast compared to us). When ever it saw a player in the center screen it shot. I'd say to avoid an agent that learns these exploits we'd have to enforce things like a maximum look speed.
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-13T03:26:35.830Z
Comments.Text: You might be interested in reading [deep mind's quake playing](https://deepmind.com/blog/article/capture-the-flag-science) if you haven't. They also have a newer post about starcraft2, but that's not a fps.
Comments.UserDisplayName: juvian
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-16T10:47:47.990Z
Comments.Text: Did you get any answers? I'm also interested on this
Comments.UserDisplayName: JoanComasFdz
Tags: <machine-learning><deep-learning><game-ai><models>
Title: Why isn't there a model playing FPS like CoD or Battlefield already existing?
UserDisplayName: politinsa
ThreadText: # Why isn't there a model playing FPS like CoD or Battlefield already existing?
  
  (Asked by: username_0 on 2020-01-13)
  
  Assuming we had an unlimited time to train a model and a very powerful machine to use our model in real-time (hello quantum computer), I'd like to know why no one could achieve to build an AI able to play a FPS, using **ONLY** pixels shown on the screen.
  
  **Disclaimer:** I am not tackling this problem and neither am I planning on doing such a thing, this is pure speculation and curiosity.
  
  ---
  
  I read this great article: [playing FPS Games with Deep Reinforcement Learning (2017)](https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14456) (*Guillaume Lample, Devendra Singh Chaplot*) where they achieve a 4/1 kills/death ratio on Doom against bots. **But this is 3 years old now**.
  
  Here is a picture of their model:
  
  [![model](https://i.sstatic.net/pyHve.png)](https://i.sstatic.net/pyHve.png)
  
  But they made 2 assumptions that we, humans, do not make when we are playing for the first time to a new game like Call of Duty or Battlefield:
  
  1. **Game feature augmentation**. To train a part of their model **they used the game engine to know if there is (or not) an enemy in the frame** they are processing. We obviously can't do this with CoD or Battlefield, and we, as human, just "learn" to recognize an enemy without these informations.
  2. **Changing textures of several maps** while training to generalize better the model (see 5.2 of the paper linked previously). To summarize, they trained the model with 10 maps changing texture of some elements to make the model generalize better. Then they tested it on 3 unknown maps. In real world (ie in the scenario where we base our training/testing exclusively on the pixels of the screen), we can't train a model with different textures on the same map. And humans are able to play a deathmatch on an unknow map without re-learning everything (detect enemies, move, hide, reloading...). We just need to construct a 3D model of the map in our head to play our best.
  
  ---
  
  Their agent "divides the problem into **two phases: navigation (exploring the map to collect items and find enemies) and action (fighting enemies when they are observed), and uses separate networks** for each phase of the game".
  
  Would it be wise to use more than 2 models? Let's say:
  
  * 1 CNN to detect enemies
  * 1 model to deal with space features (position/navigation of the agent and the enemies)
  * 1 model to choose the actions given all data previous models have found?
  
  We could train them independently, at the same time.
  
  I think we'd get better result by processing manually some features (using computer vision techniques) like the minimap to get know positions of enemies and number of ammo to feed as input of the last model (action decider).
  
  But there are other problem we'd get: there is a delay between the frame we choose to pull the trigger, the time the bullet hit the enemy and the time the "reward" appears on the screen (ex: "100 points, kill [nameOfLateEnemy]" appears after the 3th bullet, and if there is ping because we are playing online it may appear 100ms after). **How to use reinforcement learning when we don't know exactly which action was the one getting the reward?** (we can move the agent while changing the lok directon while pulling the trigger at the same time. It's the combination of these actions that is making the agent kill an enemy).
  
  If the 2 assumptions they made were easy to get rid of, they would be discarded already.  
  However detecting enemies is basically a simple CNN, and making the navigation network generalize certainly have solutions I can't think of right now but some researchers should have in this 3-year gap between the paper and today.
  
  ## So why isn't there a model playing CoD or Battlefield better than humans? What am I missing?
  
  **Tags:** machine-learning, deep-learning, game-ai, models
  > **Comments:**
  > **username_1**: I'll think on this and form a thorough answer. Some thoughts: Such an agent might simply learn to become an aim bot and exploit certain features of the game environment. Example: in Black Ops 3 on Xbox 360 some people used an aim bot mod that didn't do anything fancy. It moved to the center of the map then looked as fast as it could in all directions (very fast compared to us). When ever it saw a player in the center screen it shot. I'd say to avoid an agent that learns these exploits we'd have to enforce things like a maximum look speed.  
  > **username_2**: You might be interested in reading [deep mind's quake playing](https://deepmind.com/blog/article/capture-the-flag-science) if you haven't. They also have a newer post about starcraft2, but that's not a fps.  
  > **username_3**: Did you get any answers? I'm also interested on this  
  
  ---
  
  ## 0 answers
  
  