Answers.Body: <p>I would approach following two parallel paths.  I should mention that I greatly prefer developing reinforcement learning systems in which I provide only pixel data and a reward along with the action space.  I prefer not to extract or engineer additional features.</p>
  <ol>
  <li><p>Use Pillow to crop out the game score (or whatever other things matter that are visible as text) from the game screen.  Build and train a CNN to convert from image to text to extract the data used to build reward information.</p>
  </li>
  <li><p>Build a reinforcement system that is fed the pixel data with whatever preprocessing, frame stacking, or other things you need for your training environment, leveraging the image to text data from step 1.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T13:27:31.073Z
Answers.Text: Concerning step 1 : It is not clear to me why training a CNN for this would be a better approach than using a pretrained (eventually fine-tuned) OCR system ?
Answers.UserDisplayName: kirua
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T14:44:34.893Z
Answers.Text: It's not better or worse.  The OP asked how someone might approach it.  This is the approach I would take.  Of course you can use an OCR system, but you must be mindful of unusual fonts used in a game, so at a minimum transfer learning would most likely be required.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T12:56:38.647Z
Answers.LastActivityDate: 2022-11-23T12:56:38.647Z
Answers.UserDisplayName: David Hoelzer
Body: <p>So I have some games that I like, and I'd like to create a net that can play them, just for fun. But I don't have their source code, so I can't just pull the information I want and create a state from them. I should have mentioned that most of the games I'd like the net to play are mobile games, and I am using bluestacks for emulation.</p>
  <p>What I thought of</p>
  <ul>
  <li>Finding <strong>alternative open-source versions.</strong></li>
  <li>Taking screenshots of in-game items and then <strong>using</strong> something like <strong>pyautogui</strong> to find them on the game window.</li>
  <li><strong>Using Cheat Engine</strong> to extract the info from memory.</li>
  </ul>
  <p>Why are those not viable options</p>
  <ul>
  <li>If there are open-source versions available, they are usually not that good, plus they are not the original game, so it doesn't fit my goal since <strong>I want the AI to play the actual game</strong> that most people are playing.</li>
  <li>This is A LOT of work; it only sometimes works, and <strong>if there's an update that changes the shapes of the textures, I need to do it all again.</strong> (colour itself would be fine, I could use grayscale). And <strong>the process of finding images on images is not that fast</strong>.</li>
  <li>This is also <strong>a horrible amount of work</strong>, and it's just <strong>not possible if you need a lot of information</strong>.</li>
  </ul>
  <p>For example, I want to create a Deep Q-Learning net to play the Subway Surfers game. But I don't have the source code, and no mods allow reading the game info.
  So how would you guys approach this issue? <strong>I don't want to recreate the games myself because of what I mentioned in &quot;Why those are not viable options&quot;.</strong> Any ideas and or solutions would be well appreciated.</p>
  <p>Thanks.</p>
  
Tags: <reinforcement-learning><dqn><game-ai>
Title: Train Deep Q-Learning Network on a game without source code
UserDisplayName: Skyr
ThreadText: # Train Deep Q-Learning Network on a game without source code
  
  (Asked by: username_0 on 2022-11-23)
  
  So I have some games that I like, and I'd like to create a net that can play them, just for fun. But I don't have their source code, so I can't just pull the information I want and create a state from them. I should have mentioned that most of the games I'd like the net to play are mobile games, and I am using bluestacks for emulation.
  
  What I thought of
  
  * Finding **alternative open-source versions.**
  * Taking screenshots of in-game items and then **using** something like **pyautogui** to find them on the game window.
  * **Using Cheat Engine** to extract the info from memory.
  
  Why are those not viable options
  
  * If there are open-source versions available, they are usually not that good, plus they are not the original game, so it doesn't fit my goal since **I want the AI to play the actual game** that most people are playing.
  * This is A LOT of work; it only sometimes works, and **if there's an update that changes the shapes of the textures, I need to do it all again.** (colour itself would be fine, I could use grayscale). And **the process of finding images on images is not that fast**.
  * This is also **a horrible amount of work**, and it's just **not possible if you need a lot of information**.
  
  For example, I want to create a Deep Q-Learning net to play the Subway Surfers game. But I don't have the source code, and no mods allow reading the game info.
  So how would you guys approach this issue? **I don't want to recreate the games myself because of what I mentioned in "Why those are not viable options".** Any ideas and or solutions would be well appreciated.
  
  Thanks.
  
  **Tags:** reinforcement-learning, dqn, game-ai
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-23. Score: 0 | Accepted answer
  
  I would approach following two parallel paths. I should mention that I greatly prefer developing reinforcement learning systems in which I provide only pixel data and a reward along with the action space. I prefer not to extract or engineer additional features.
  
  1. Use Pillow to crop out the game score (or whatever other things matter that are visible as text) from the game screen. Build and train a CNN to convert from image to text to extract the data used to build reward information.
  2. Build a reinforcement system that is fed the pixel data with whatever preprocessing, frame stacking, or other things you need for your training environment, leveraging the image to text data from step 1.
  
  > **Comments:**
  > **username_2**: Concerning step 1 : It is not clear to me why training a CNN for this would be a better approach than using a pretrained (eventually fine-tuned) OCR system ?  
  > **username_1**: It's not better or worse.  The OP asked how someone might approach it.  This is the approach I would take.  Of course you can use an OCR system, but you must be mindful of unusual fonts used in a game, so at a minimum transfer learning would most likely be required.  
  
  ---
  
  