Answers.Body: <p>I think pseudocode was made for tabular case with an assumption of deterministic environment. <span class="math-container">$Model(s, a)$</span> would then be a table with information of the next state and reward after taking action <span class="math-container">$a$</span> from state <span class="math-container">$s$</span>. The size of that table would be same as the size of Q table. Because the environment is deterministic you wouldn't take a random sample because there is only one possible transition so you would take the transition remembered in model table.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-10T11:16:22.990Z
Answers.Text: Thanks for your response. Do you mean that if we are in state s and take action a, we will be always in the state `S_new` in the next state? It seems somehow weird!
Answers.UserDisplayName: Amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-10T13:49:20.710Z
Answers.Text: Yes, it's weird because it was made with an assumption of deterministic environment which is quite optimistic and unreasonable. I never tried it myself but i believe if you have stochastic environment, for tabular case, you could remember all the different transitions that happened and also have a count of how many times those transitions happened, then you could calculate probabilities of those transitions happening from those counts and do planning with that. Maybe you could do sampling like you suggested but based on those probabilities or maybe some kind of dynamic programming updates.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-10T18:28:35.087Z
Answers.Text: Thanks. Have you ever seen a paper that used the stochastic version? In the deterministic version, does the model use the last visit of next state and reward?
Answers.UserDisplayName: Amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-10T19:11:20.533Z
Answers.Text: As for the paper, google search has led me to [this](https://lenabayeva.files.wordpress.com/2015/03/aamasreport2011.pdf) . I haven't read the paper but it looks like exactly what you are looking for. As for the deterministic environment, yes, model simply uses last observed next state and reward.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-11T14:02:58.340Z
Answers.Text: Dear @Brale 
  
  I really appreciate you.
Answers.UserDisplayName: Amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-09T10:46:55.373Z
Answers.LastActivityDate: 2019-01-09T10:46:55.373Z
Answers.UserDisplayName: Brale
Body: <p>I'm reading the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton). </p>
  
  <p>The authors provide the pseudocode of the <em>prioritized sweeping</em> algorithm, but I do not know what is the meaning of <code>Model(s, a)</code>. Does it mean that <code>Model(s, a)</code> is the history of rewards gained when we are in state <code>s</code> and the action <code>a</code> is taken? </p>
  
  <p>Does <code>R, S_new = Model(s,a)</code> mean that we should take a random sample from rewards gained in state <code>s</code> and action <code>a</code> is taken?</p>
  
Tags: <reinforcement-learning><sutton-barto><prioritized-sweeping>
Title: What is the meaning of Model(s, a) in the prioritized sweeping algorithm?
UserDisplayName: Amin
ThreadText: # What is the meaning of Model(s, a) in the prioritized sweeping algorithm?
  
  (Asked by: username_0 on 2019-01-09)
  
  I'm reading the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).
  
  The authors provide the pseudocode of the *prioritized sweeping* algorithm, but I do not know what is the meaning of `Model(s, a)`. Does it mean that `Model(s, a)` is the history of rewards gained when we are in state `s` and the action `a` is taken?
  
  Does `R, S_new = Model(s,a)` mean that we should take a random sample from rewards gained in state `s` and action `a` is taken?
  
  **Tags:** reinforcement-learning, sutton-barto, prioritized-sweeping
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-09. Score: 3
  
  I think pseudocode was made for tabular case with an assumption of deterministic environment. $Model(s, a)$ would then be a table with information of the next state and reward after taking action $a$ from state $s$. The size of that table would be same as the size of Q table. Because the environment is deterministic you wouldn't take a random sample because there is only one possible transition so you would take the transition remembered in model table.
  
  > **Comments:**
  > **username_0**: Thanks for your response. Do you mean that if we are in state s and take action a, we will be always in the state `S_new` in the next state? It seems somehow weird!  
  > **username_1**: Yes, it's weird because it was made with an assumption of deterministic environment which is quite optimistic and unreasonable. I never tried it myself but i believe if you have stochastic environment, for tabular case, you could remember all the different transitions that happened and also have a count of how many times those transitions happened, then you could calculate probabilities of those transitions happening from those counts and do planning with that. Maybe you could do sampling like you suggested but based on those probabilities or maybe some kind of dynamic programming updates.  
  > **username_0**: Thanks. Have you ever seen a paper that used the stochastic version? In the deterministic version, does the model use the last visit of next state and reward?  
  > **username_1**: As for the paper, google search has led me to [this](https://lenabayeva.files.wordpress.com/2015/03/aamasreport2011.pdf) . I haven't read the paper but it looks like exactly what you are looking for. As for the deterministic environment, yes, model simply uses last observed next state and reward.  
  > **username_0**: Dear @username_1   I really appreciate you.  
  
  ---
  
  