Answers.Body: <p>You have implemented a simple <a href="https://getstream.io/blog/introduction-contextual-bandits/" rel="nofollow noreferrer">contextual bandit</a> solver, which is a machine learning algorithm. A few details may be different from a full implementation, but the key elements are:</p>
  
  <ul>
  <li><p>A choice of actions (click hit or don't click hit)</p></li>
  <li><p>A reward signal that can be observed after each action (+1 for a hit, 0 for nothing happens, -1 for an attack which misses)</p></li>
  <li><p>An observable state which affects the reward achievable (the position of the blue ball). For a contextual bandit, the state is not influenced by the action taken. This is true here.</p></li>
  <li><p>One thing that is different about your problem from a classic contextual bandit is that the next state <em>is</em> predictable from the current state (whilst in a pure bandit problem it should be entirely random). However, that's not too important to your problem here, and your solver is definitely following a contextual bandit approach.</p></li>
  <li><p>Your solver tests the score from trying different actions in each state, and narrows down the best action to take in each state. Your implementation is simple and "greedy" for a contextual bandit solver. A more typical solution would maintain an average result for each action and have a rule for how to explore actions in each state, so it could test whether results were reliable (this is very helpful with non-deterministic scenarios where bandit solvers are more often used).</p></li>
  </ul>
  
  <blockquote>
    <p>With each hit, the blue ball moves faster</p>
  </blockquote>
  
  <p>Unless you somehow limit the reaction time of the agent, this is not relevant to how you write the solver. You <em>could</em> change the rules affecting the agent to make it relevant in the same way as it would be for a human, e.g. deciding to click means the click happens 0.1 seconds later, and the state can include observations of position just now and several 0.02 seconds going back.</p>
  
  <p>In general, if you want to take this further, with more complex games and still learning how to control agent actions, you could look at simple reinforcement learning agents, such as Q-learning. If you are interested in the underlying theory of agents like this, then a good (and free) introductory text is <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton &amp; Barto "Reinforcement Learning: An Introduction"</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-29T21:50:45.987Z
Answers.Text: In my case, if the red ball change position, the solver don't work anymore... thank you, I will read this.
Answers.UserDisplayName: GIA
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-29T22:01:08.673Z
Answers.Text: @GuilhermeIA: An obvious extension to your solver there is to add the position of the red ball to the state. That adds another *dimension* so makes the problem a lot harder, as your learner will need to track far more states in order to learn that red and blue need to be coincident when it tries to hit.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-29T21:36:48.500Z
Answers.LastActivityDate: 2018-06-29T21:36:48.500Z
Answers.UserDisplayName: Neil Slater
Body: <p>I built a simple HTML game. In this game the goal is to click when the blue ball is above the red ball. If you hit, you get 1 point, if you miss, you lose 1 point. With each hit, the blue ball moves faster. <a href="https://codepen.io/iazzetta/pen/BVxzyr" rel="nofollow noreferrer">You can test the game here</a>.</p>
  
  <p><a href="https://i.sstatic.net/5L5pQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5L5pQ.png" alt="enter image description here"></a></p>
  
  <p>Without using machine learning, I would easily solve this problem by just clicking when the X, Y of the blue ball was on the X, Y of the red ball. Regardless of the time, knowing the positions of the 2 elements I could solve the problem of the game.</p>
  
  <blockquote>
    <p>However, if I wanted to create an AI to solve this problem, could I?
    How would it be? I'd really like to see the AI randomly wandering
    until it's perfect.</p>
  </blockquote>
  
  <h2>My way to solve the problem</h2>
  
  <p>I click many times and watch score. If score down, add to bad_positions. If actual position in bad_positions, not click. At first he misses many times, then starts to hit eternally. <strong>This is machine learning? Deep learning? Just a bot?</strong></p>
  
  <pre><code>var bad_positions = [];
  function train(){
    var pos = $ball.offset().left;
    var last_score = score;
    if (!bad_positions.includes(pos)) {
     $('#hit').click();
      if (score &lt; last_score){
        bad_positions.push(pos)
      } 
    }
  }
  </code></pre>
  
Tags: <machine-learning><deep-learning><game-ai><javascript>
Title: How to use Machine Learning with simple games?
UserDisplayName: GIA
ThreadText: # How to use Machine Learning with simple games?
  
  (Asked by: username_0 on 2018-06-29)
  
  I built a simple HTML game. In this game the goal is to click when the blue ball is above the red ball. If you hit, you get 1 point, if you miss, you lose 1 point. With each hit, the blue ball moves faster. You can test the game here.
  
  Without using machine learning, I would easily solve this problem by just clicking when the X, Y of the blue ball was on the X, Y of the red ball. Regardless of the time, knowing the positions of the 2 elements I could solve the problem of the game.
  
  However, if I wanted to create an AI to solve this problem, could I?
    How would it be? I'd really like to see the AI randomly wandering
    until it's perfect.
  
  My way to solve the problem
  I click many times and watch score. If score down, add to bad_positions. If actual position in bad_positions, not click. At first he misses many times, then starts to hit eternally. This is machine learning? Deep learning? Just a bot?
  var bad_positions = [];
  function train(){
    var pos = $ball.offset().left;
    var last_score = score;
    if (!bad_positions.includes(pos)) {
     $('#hit').click();
      if (score < last_score){
        bad_positions.push(pos)
      } 
    }
  }
  
  **Tags:** machine-learning, deep-learning, game-ai, javascript
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-06-29. Score: 1 | Accepted answer
  
  You have implemented a simple [contextual bandit](https://getstream.io/blog/introduction-contextual-bandits/) solver, which is a machine learning algorithm. A few details may be different from a full implementation, but the key elements are:
  
  * A choice of actions (click hit or don't click hit)
  * A reward signal that can be observed after each action (+1 for a hit, 0 for nothing happens, -1 for an attack which misses)
  * An observable state which affects the reward achievable (the position of the blue ball). For a contextual bandit, the state is not influenced by the action taken. This is true here.
  * One thing that is different about your problem from a classic contextual bandit is that the next state *is* predictable from the current state (whilst in a pure bandit problem it should be entirely random). However, that's not too important to your problem here, and your solver is definitely following a contextual bandit approach.
  * Your solver tests the score from trying different actions in each state, and narrows down the best action to take in each state. Your implementation is simple and "greedy" for a contextual bandit solver. A more typical solution would maintain an average result for each action and have a rule for how to explore actions in each state, so it could test whether results were reliable (this is very helpful with non-deterministic scenarios where bandit solvers are more often used).
  
  > With each hit, the blue ball moves faster
  
  Unless you somehow limit the reaction time of the agent, this is not relevant to how you write the solver. You *could* change the rules affecting the agent to make it relevant in the same way as it would be for a human, e.g. deciding to click means the click happens 0.1 seconds later, and the state can include observations of position just now and several 0.02 seconds going back.
  
  In general, if you want to take this further, with more complex games and still learning how to control agent actions, you could look at simple reinforcement learning agents, such as Q-learning. If you are interested in the underlying theory of agents like this, then a good (and free) introductory text is [Sutton & Barto "Reinforcement Learning: An Introduction"](http://incompleteideas.net/book/the-book-2nd.html)
  
  > **Comments:**
  > **username_0**: In my case, if the red ball change position, the solver don't work anymore... thank you, I will read this.  
  > **username_1**: @GuilhermeIA: An obvious extension to your solver there is to add the position of the red ball to the state. That adds another *dimension* so makes the problem a lot harder, as your learner will need to track far more states in order to learn that red and blue need to be coincident when it tries to hit.  
  
  ---
  
  