Answers.Body: <p>A full Bellman update can be intractable. For instance, if your state space or action space are continuous, the full Bellman update is intractable. You can try to solve this by discretizing, but if your state space is large this will also be intractable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T15:15:37.543Z
Answers.Text: Would it also make sense to prefer MC in MDPs where episodes run for a very long time, since it will take Bellman updates far longer to propagate back to the start state?
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T17:00:13.353Z
Answers.Text: Not necessarily, I believe you'd experience the same issue with MC. I believe in general if you can do bellman updates, you probably should.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T17:58:54.720Z
Answers.Text: Interesting. In MC, all states in an episode are updated based on the terminal state. So if the reward signal is sparse and only present in terminal states, like in a game, I think the signal would propogate faster, since in Bellman updates, the reward propogates back one state at a time
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T18:15:25.167Z
Answers.Text: Don't forget, in MC you have to wait an entire episode before an update. Credit assignment will be difficult one way or the other, I believe.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T15:09:29.047Z
Answers.LastActivityDate: 2020-07-09T15:09:29.047Z
Answers.UserDisplayName: harwiltz
Body: <p>Sample-based algorithms, like Monte Carlo Algorithms and TD-Learning, are often presented as useful since they do not require a transition model.</p>
  <p>Assuming I do have access to a transition model, are there any reasons one might want to use sample-based methods instead of performing a full Bellman update?</p>
  
Tags: <reinforcement-learning><monte-carlo-methods><temporal-difference-methods><bellman-equations><dynamic-programming>
Title: If the transition model is available, why would we use sample-based algorithms?
UserDisplayName: chessprogrammer
ThreadText: # If the transition model is available, why would we use sample-based algorithms?
  
  (Asked by: username_0 on 2020-07-09)
  
  Sample-based algorithms, like Monte Carlo Algorithms and TD-Learning, are often presented as useful since they do not require a transition model.
  
  Assuming I do have access to a transition model, are there any reasons one might want to use sample-based methods instead of performing a full Bellman update?
  
  **Tags:** reinforcement-learning, monte-carlo-methods, temporal-difference-methods, bellman-equations, dynamic-programming
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-09. Score: 1
  
  A full Bellman update can be intractable. For instance, if your state space or action space are continuous, the full Bellman update is intractable. You can try to solve this by discretizing, but if your state space is large this will also be intractable.
  
  > **Comments:**
  > **username_0**: Would it also make sense to prefer MC in MDPs where episodes run for a very long time, since it will take Bellman updates far longer to propagate back to the start state?  
  > **username_1**: Not necessarily, I believe you'd experience the same issue with MC. I believe in general if you can do bellman updates, you probably should.  
  > **username_0**: Interesting. In MC, all states in an episode are updated based on the terminal state. So if the reward signal is sparse and only present in terminal states, like in a game, I think the signal would propogate faster, since in Bellman updates, the reward propogates back one state at a time  
  > **username_1**: Don't forget, in MC you have to wait an entire episode before an update. Credit assignment will be difficult one way or the other, I believe.  
  
  ---
  
  