Answers.Body: <p>Binning of inputs are of course possible but it is a balancing act that may never outperform a network operating on continuous values.</p>
  <p>Coarse binning may lose information necessary to accurately solve the problem.  Imagine backup sensors on a car.  If your goal is to get as close to the wall behind you without touching it, you can do it with a beeping sensor (distance binning sensor) but you won’t be as accurate as with a camera (continuous).</p>
  <p>On the other end of the spectrum with many bins one tempts the curse of dimensionality.  This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T15:05:08.420Z
Answers.Text: Ok i understood, thanks. Relatively to this "On the other end of the spectrum with many bins one tempts the curse of dimensionality. This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.". Discretizing the state variables is not increasing the number of state variables is reducing the information of each variable.
Answers.UserDisplayName: Vitor Martins
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T15:27:54.823Z
Answers.Text: I wrote this very early in the morning for me.  When I read descritizing my brain went to one hot encoding of the continuous input into bins which does tempt the curse.  On the other hand, turning the continuous input into a step wise function (ie, round(x,0)) does not tempt the curse, you are correct.  This nestles up to NN quantization and it's trade-offs.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T13:30:56.447Z
Answers.LastActivityDate: 2024-04-10T14:46:57.547Z
Answers.LastEditDate: 2024-04-10T14:46:57.547Z
Answers.UserDisplayName: foreverska
Body: <p>For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete state and action spaces.</p>
  <p>In some examples of DQN use for solving Cart Pole gym environment, they use directly each state variable as input for the neural network (NN), therefore my doubt is: Can this input state variables for the NN be discretized in bins allowing to solve the cartpole environment ?</p>
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-10T11:51:11.240Z
Comments.Text: You could discretise the continuous output into bins and then use a tabular lookup table but you would either a) need to use very coarse discretisations, at which point you'd probably lose too much information to be able to solve the problem well b) require a very fine discretisation that would lead to a combinatorial blow up in number of states required to store in the lookup table.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-10T13:15:57.047Z
Comments.Text: So what you're telling me is that is possible to discretize the state space and send that "bins" to the NN, though there're some problems of lossing information to effectively learn the goal ?
Comments.UserDisplayName: Vitor Martins
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-10T16:35:03.867Z
Comments.Text: oh, sorry, I didn't realise that you wanted to pass the discretised input into the NN. Why would you want to do this? I thought you wanted to learn in a tabular way.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-15T15:08:13.333Z
Comments.Text: To use DQN algorithm. In a case that have to much discretized states and actions, i think that DQN should perform better than a Q-Learning algorithm.
Comments.UserDisplayName: Vitor Martins
Tags: <reinforcement-learning><deep-rl><q-learning><dqn>
Title: Can DQN learn with discrete state spaces?
UserDisplayName: Vitor Martins
ThreadText: # Can DQN learn with discrete state spaces?
  
  (Asked by: username_0 on 2024-04-10)
  
  For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete state and action spaces.
  
  In some examples of DQN use for solving Cart Pole gym environment, they use directly each state variable as input for the neural network (NN), therefore my doubt is: Can this input state variables for the NN be discretized in bins allowing to solve the cartpole environment ?
  
  Thanks.
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, dqn
  
  > **Comments:**
  > **username_1**: You could discretise the continuous output into bins and then use a tabular lookup table but you would either a) need to use very coarse discretisations, at which point you'd probably lose too much information to be able to solve the problem well b) require a very fine discretisation that would lead to a combinatorial blow up in number of states required to store in the lookup table.  
  > **username_0**: So what you're telling me is that is possible to discretize the state space and send that "bins" to the NN, though there're some problems of lossing information to effectively learn the goal ?  
  > **username_1**: oh, sorry, I didn't realise that you wanted to pass the discretised input into the NN. Why would you want to do this? I thought you wanted to learn in a tabular way.  
  > **username_0**: To use DQN algorithm. In a case that have to much discretized states and actions, i think that DQN should perform better than a Q-Learning algorithm.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-04-10. Score: 1 | Accepted answer
  
  Binning of inputs are of course possible but it is a balancing act that may never outperform a network operating on continuous values.
  
  Coarse binning may lose information necessary to accurately solve the problem. Imagine backup sensors on a car. If your goal is to get as close to the wall behind you without touching it, you can do it with a beeping sensor (distance binning sensor) but you won’t be as accurate as with a camera (continuous).
  
  On the other end of the spectrum with many bins one tempts the curse of dimensionality. This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.
  
  > **Comments:**
  > **username_0**: Ok i understood, thanks. Relatively to this "On the other end of the spectrum with many bins one tempts the curse of dimensionality. This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.". Discretizing the state variables is not increasing the number of state variables is reducing the information of each variable.  
  > **username_2**: I wrote this very early in the morning for me.  When I read descritizing my brain went to one hot encoding of the continuous input into bins which does tempt the curse.  On the other hand, turning the continuous input into a step wise function (ie, round(x,0)) does not tempt the curse, you are correct.  This nestles up to NN quantization and it's trade-offs.  
  
  ---
  
  