Answers.Body: <p>You don't have &quot;9 possible states&quot; in your problem, you have those 9 board squares multiplied by all possible values that the numbers within them can take, in all combinations. If those are discrete integers within bounds, then your space is finite - as an example if each square could contain the number <span class="math-container">$1$</span>,<span class="math-container">$2$</span> or <span class="math-container">$3$</span> then your state space would have size <span class="math-container">$3^9 \times 9 = 177147$</span> (the extra <span class="math-container">$\times 9$</span> is because you also need to track which square the agent is in, and that can occur in any combination with the square values). If instead the ranges are unbound, or numbers real-valued, then your state space is infinite.</p>
  <p>It is still possible to solve RL problems with infinite state space. As your action space is simple and discrete, you can still use value-based methods.</p>
  <p>The step you need to make is to look into approximation schemes. <a href="http://www.incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton &amp; Barto</a> has a useful section on the simpler approximation schemes that you can use with linear approximators (or even the most basic state aggregation might work for you). In second edition that is chapters 9 and 10.
  .</p>
  <p>A very common approach with your kind of problem is to use a neural network and DQN model (Q-learning adapted to work with a neural network). If you search for tutorials on working with DQN, you should find hundreds, so you can filter down to something relevant and useful to you.</p>
  <p>The very basics of DQN are:</p>
  <ul>
  <li>It's Q-learning, but instead of the action value table, you will train a neural network to estimate <span class="math-container">$Q(s,a)$</span></li>
  <li>Combining neural networks with off-policy RL can be unstable, so there are a couple of important changes to learning process:
  <ul>
  <li>Instead of immediately updating from the most recent experience, all experience is buffered in a large table and random sampled mini-batches are taken from that table to train with.</li>
  <li>Instead of using the current estimator to bootstrap from, an old copy of it is used and updated occasionally (e.g. once every 1000 steps)</li>
  </ul>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T16:31:50.137Z
Answers.Text: DQN sounds like it is the answer. Thank you for the information.
Answers.UserDisplayName: Eric
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T09:18:24.200Z
Answers.LastActivityDate: 2023-02-21T13:38:36.280Z
Answers.LastEditDate: 2023-02-21T13:38:36.280Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am new to reinforcement learning and having a hard time making the leap from tutorials to real world problems.</p>
  <p>I'm trying to figure out how to deal with a board game with 9 squares each square representing a state, so 9 possible states, but each state will also have one or more numeric values associated with it and the numeric values will be different in each episode of the game. In my example the agent can only see the &quot;features?&quot; of its current state.  The only possible actions are to up, down, left, right. The initial policy is set to give the agent a 25% chance of moving in each direction I also want the policy to consider the numeric variables associated with the current state. How do I do this without creating an infinite amount of state action pairs?</p>
  
Tags: <reinforcement-learning><board-games>
Title: How to account for a numeric variable in a state in RL?
UserDisplayName: Eric
ThreadText: # How to account for a numeric variable in a state in RL?
  
  (Asked by: username_0 on 2023-02-21)
  
  I am new to reinforcement learning and having a hard time making the leap from tutorials to real world problems.
  
  I'm trying to figure out how to deal with a board game with 9 squares each square representing a state, so 9 possible states, but each state will also have one or more numeric values associated with it and the numeric values will be different in each episode of the game. In my example the agent can only see the "features?" of its current state. The only possible actions are to up, down, left, right. The initial policy is set to give the agent a 25% chance of moving in each direction I also want the policy to consider the numeric variables associated with the current state. How do I do this without creating an infinite amount of state action pairs?
  
  **Tags:** reinforcement-learning, board-games
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-21. Score: 2 | Accepted answer
  
  You don't have "9 possible states" in your problem, you have those 9 board squares multiplied by all possible values that the numbers within them can take, in all combinations. If those are discrete integers within bounds, then your space is finite - as an example if each square could contain the number $1$,$2$ or $3$ then your state space would have size $3^9 \times 9 = 177147$ (the extra $\times 9$ is because you also need to track which square the agent is in, and that can occur in any combination with the square values). If instead the ranges are unbound, or numbers real-valued, then your state space is infinite.
  
  It is still possible to solve RL problems with infinite state space. As your action space is simple and discrete, you can still use value-based methods.
  
  The step you need to make is to look into approximation schemes. [Sutton & Barto](http://www.incompleteideas.net/book/the-book-2nd.html) has a useful section on the simpler approximation schemes that you can use with linear approximators (or even the most basic state aggregation might work for you). In second edition that is chapters 9 and 10.
  .
  
  A very common approach with your kind of problem is to use a neural network and DQN model (Q-learning adapted to work with a neural network). If you search for tutorials on working with DQN, you should find hundreds, so you can filter down to something relevant and useful to you.
  
  The very basics of DQN are:
  
  * It's Q-learning, but instead of the action value table, you will train a neural network to estimate $Q(s,a)$
  * Combining neural networks with off-policy RL can be unstable, so there are a couple of important changes to learning process:
    + Instead of immediately updating from the most recent experience, all experience is buffered in a large table and random sampled mini-batches are taken from that table to train with.
    + Instead of using the current estimator to bootstrap from, an old copy of it is used and updated occasionally (e.g. once every 1000 steps)
  
  > **Comments:**
  > **username_0**: DQN sounds like it is the answer. Thank you for the information.  
  
  ---
  
  