Answers.Body: <blockquote>
  <p>Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?</p>
  </blockquote>
  <p>Individual problems don't get any more complex, but the scope of <em>solvable</em> environments increases due to research and discovery of better or more apt methods.</p>
  <p>Using deep RL methods with large neural nets can be a lot less efficient for solving simple problems. So tabular methods still have their place there.</p>
  <p>Practically, if your state/action space (number of states times number of actions) is small enough to fit a Q table in memory, and it is possible to visit all relevant state/action pairs multiple times in a relatively short time, then tabular methods offer guarantees of convergence that approximate methods cannot. So tabular approaches are often preferred if they are appropriate.</p>
  <p>Many interesting, cutting edge problems that are relevant to AI, such as autonomous robots acting in the real world, do not fit the tabular approach. In that sense, the approach is &quot;obsolete&quot; in that it no longer provides challenging research topics for practical AI (there are still unanswered theoretical questions, such as proof of convergence for Monte Carlo control).</p>
  <p>It is still worth understanding tabular value-based methods in detail, because they form the foundations of the more complex deep learning methods. In some sense they represent ideal solutions that deep RL tries to approximate, and the design of tabular solutions can be the inspiration for changes and adjustments to neural-network methods.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-21T20:46:45.610Z
Answers.Text: Not many people understand the convergence proof shown in neuro-dynamic programming. I'm not even sure the professors I had which worked on this area remembered much of it.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-21T20:07:24.420Z
Answers.LastActivityDate: 2020-07-21T20:07:24.420Z
Answers.UserDisplayName: Neil Slater
Body: <p>While learning RL, I came across some problems where the Q-matrix that I need to make is very very large. I am not sure if it is ever practical. Then I research and came to this conclusion that using the tabular method is not the only way, in fact, it is a very less powerful tool as compared to other methods such as deep RL methods.</p>
  <p>Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?</p>
  
Tags: <reinforcement-learning><deep-learning>
Title: Are tabular reinforcement learning methods obsolete (or getting obsolete)?
UserDisplayName: SJa
ThreadText: # Are tabular reinforcement learning methods obsolete (or getting obsolete)?
  
  (Asked by: username_0 on 2020-07-21)
  
  While learning RL, I came across some problems where the Q-matrix that I need to make is very very large. I am not sure if it is ever practical. Then I research and came to this conclusion that using the tabular method is not the only way, in fact, it is a very less powerful tool as compared to other methods such as deep RL methods.
  
  Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?
  
  **Tags:** reinforcement-learning, deep-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-21. Score: 3 | Accepted answer
  
  > Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?
  
  Individual problems don't get any more complex, but the scope of *solvable* environments increases due to research and discovery of better or more apt methods.
  
  Using deep RL methods with large neural nets can be a lot less efficient for solving simple problems. So tabular methods still have their place there.
  
  Practically, if your state/action space (number of states times number of actions) is small enough to fit a Q table in memory, and it is possible to visit all relevant state/action pairs multiple times in a relatively short time, then tabular methods offer guarantees of convergence that approximate methods cannot. So tabular approaches are often preferred if they are appropriate.
  
  Many interesting, cutting edge problems that are relevant to AI, such as autonomous robots acting in the real world, do not fit the tabular approach. In that sense, the approach is "obsolete" in that it no longer provides challenging research topics for practical AI (there are still unanswered theoretical questions, such as proof of convergence for Monte Carlo control).
  
  It is still worth understanding tabular value-based methods in detail, because they form the foundations of the more complex deep learning methods. In some sense they represent ideal solutions that deep RL tries to approximate, and the design of tabular solutions can be the inspiration for changes and adjustments to neural-network methods.
  
  > **Comments:**
  > **username_2**: Not many people understand the convergence proof shown in neuro-dynamic programming. I'm not even sure the professors I had which worked on this area remembered much of it.  
  
  ---
  
  