Body: <p>For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done <a href="https://www.datamachinist.com/reinforcement-learning/part-6-q-learning-for-continuous-state-problems/" rel="nofollow noreferrer">here</a>. However, according to <a href="https://core.ac.uk/download/pdf/301653434.pdf" rel="nofollow noreferrer">van Hasselt</a> from his <a href="https://books.google.com.br/books/about/Reinforcement_Learning.html?id=YPjNuvrJR0MC&amp;redir_esc=y" rel="nofollow noreferrer">book</a>, this discretization changes the problem into a partially observable MDP (POMDP), and this is understandable. And I know POMDPs require special treatment from the vanilla Q-learning we are used to (observation space, belief states, etc).</p>
  <p>But my question is: is there a specific technical reason why a discretized-state problem (which is now POMDP) should be solved using POMDP algorithms, instead of plainly constructing a vanilla Q-table using the discretized states (i.e. the buckets from discretization)? In other words, is there a disadvantage in not using POMDP algorithms to tackle the discretized-state problem?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T19:35:19.730Z
Comments.Text: I don't see why would state discretization that you propose cause your MDP to be partially observable, do you mind clarifying your conclusion ?
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T19:38:13.143Z
Comments.Text: When you discretize, several true states will be observed as the same. For instance, if we discretize all numbers from 0 to 0.1 as state 0, then 0.05 will be observed as state 0, and so will 0.097.
Comments.UserDisplayName: O'Jhene
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-25T19:42:15.157Z
Comments.Text: That's more of a problem of insufficient discretization than the method itself being inherently flawed. You can always refine your discretization until your policy is "constant" over your discretization tile, so that you don't have an issue where sparse discretization might be a problem.
Comments.UserDisplayName: Brale
Tags: <reinforcement-learning><q-learning><markov-decision-process><pomdp>
Title: Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?
UserDisplayName: O'Jhene
ThreadText: # Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?
  
  (Asked by: username_0 on 2021-02-25)
  
  For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done [here](https://www.datamachinist.com/reinforcement-learning/part-6-q-learning-for-continuous-state-problems/). However, according to [van Hasselt](https://core.ac.uk/download/pdf/301653434.pdf) from his [book](https://books.google.com.br/books/about/Reinforcement_Learning.html?id=YPjNuvrJR0MC&redir_esc=y), this discretization changes the problem into a partially observable MDP (POMDP), and this is understandable. And I know POMDPs require special treatment from the vanilla Q-learning we are used to (observation space, belief states, etc).
  
  But my question is: is there a specific technical reason why a discretized-state problem (which is now POMDP) should be solved using POMDP algorithms, instead of plainly constructing a vanilla Q-table using the discretized states (i.e. the buckets from discretization)? In other words, is there a disadvantage in not using POMDP algorithms to tackle the discretized-state problem?
  
  **Tags:** reinforcement-learning, q-learning, markov-decision-process, pomdp
  > **Comments:**
  > **username_1**: I don't see why would state discretization that you propose cause your MDP to be partially observable, do you mind clarifying your conclusion ?  
  > **username_0**: When you discretize, several true states will be observed as the same. For instance, if we discretize all numbers from 0 to 0.1 as state 0, then 0.05 will be observed as state 0, and so will 0.097.  
  > **username_1**: That's more of a problem of insufficient discretization than the method itself being inherently flawed. You can always refine your discretization until your policy is "constant" over your discretization tile, so that you don't have an issue where sparse discretization might be a problem.  
  
  ---
  
  ## 0 answers
  
  