Body: <p>I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the <span class="math-container">$Q(s, a)$</span> after each action taken). But this 'large space' realization has now disappointed me and I read a lot of stuff on the internet. I have the following confusions.</p>
  <p>I saw the 'approximation' term for the 'large space' scenario (for example, in <a href="https://towardsdatascience.com/function-approximation-in-reinforcement-learning-85a4864d566" rel="nofollow noreferrer">this Medium blog post</a>). But what it is <em><strong>exactly</strong></em>? I can't reduce the states I have nor can I club together different states and update the Q values. So, what is it I should do when they say 'approximate'? If it is the <span class="math-container">$Q(s,a)$</span> we approximate, then won't we anyway do for each state <span class="math-container">$s$</span> as and when it is encountered? How does this help in a 'large space' scenario?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T14:47:00.203Z
Comments.Text: The only thing that is missing from your question is maybe some specific context. You say "I saw the 'approximation' term for the 'large space' scenario". Can you please quote the excerpt (and provide the link to the source) where people use the term "approximation" in this specific context. It may be necessary to answer this question more appropriately, although I think it may already be answerable. After that, I will reopen your post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T15:01:25.637Z
Comments.Text: Actually, I saw like lot of sources which are more for continuous state spaces and they say approximation. For large state spaces. I saw the term 'discretization'. But I couldn't put all the details here and thought that people would eventually point this out in answers. So, if I am try to put a link here it would look like aah, it is for continuous space and you are asking for large state space. I am confused and so I asked the way it is.
Comments.UserDisplayName: knowledge_seeker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T15:24:15.967Z
Comments.Text: Discretisation of a continuous space is one form of approximation (one of the simplest, but can be very effective depending on the environment).
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><q-learning><value-functions><function-approximation><continuous-state-spaces>
Title: What do we actually 'approximate' when dealing with large state spaces in Q-learning?
UserDisplayName: knowledge_seeker
ThreadText: # What do we actually 'approximate' when dealing with large state spaces in Q-learning?
  
  (Asked by: username_0 on 2021-11-19)
  
  I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the $Q(s, a)$ after each action taken). But this 'large space' realization has now disappointed me and I read a lot of stuff on the internet. I have the following confusions.
  
  I saw the 'approximation' term for the 'large space' scenario (for example, in [this Medium blog post](https://towardsdatascience.com/function-approximation-in-reinforcement-learning-85a4864d566)). But what it is ***exactly***? I can't reduce the states I have nor can I club together different states and update the Q values. So, what is it I should do when they say 'approximate'? If it is the $Q(s,a)$ we approximate, then won't we anyway do for each state $s$ as and when it is encountered? How does this help in a 'large space' scenario?
  
  **Tags:** reinforcement-learning, q-learning, value-functions, function-approximation, continuous-state-spaces
  > **Comments:**
  > **username_1**: The only thing that is missing from your question is maybe some specific context. You say "I saw the 'approximation' term for the 'large space' scenario". Can you please quote the excerpt (and provide the link to the source) where people use the term "approximation" in this specific context. It may be necessary to answer this question more appropriately, although I think it may already be answerable. After that, I will reopen your post.  
  > **username_0**: Actually, I saw like lot of sources which are more for continuous state spaces and they say approximation. For large state spaces. I saw the term 'discretization'. But I couldn't put all the details here and thought that people would eventually point this out in answers. So, if I am try to put a link here it would look like aah, it is for continuous space and you are asking for large state space. I am confused and so I asked the way it is.  
  > **username_2**: Discretisation of a continuous space is one form of approximation (one of the simplest, but can be very effective depending on the environment).  
  
  ---
  
  ## 0 answers
  
  