Answers.Body: <blockquote>
  <p>My question is if I should select state_action pairs by theirs immediate reward or should I select them by the episode reward?</p>
  </blockquote>
  <p>By the return (sum of all rewards) from the whole episode. A lot of decisions made in &quot;good&quot; episodes do not lead to immediate rewards, but instead transition towards states where better rewards are possible.</p>
  <p>In retrospect, you do not know whether any single action was a good choice, but with the cross entropy method (CEM) you rely on the fact that on average the better episodes will contain more good decisions than the worse episodes, so you train the policy neural network on the state, action pairs from the elite episodes as if all the decisions were correct. This will not be true, but will hopefully be true more often than by chance, so the policy should improve.</p>
  <p>This can be a noisy, high variance approach with any RL method. CEM is one of the most sensitive to noise and variance. However, the taxi environment is deterministic, and that makes using CEM more reasonable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-22T11:59:39.740Z
Answers.Text: So, I should select all the state-action pairs in each episode with a total_reward bigger than the treshold. Am I ok?
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-22T12:33:48.303Z
Answers.Text: @HermesMorales Yes. If an episode is "elite", then all the state/action pairs in it should be considered good for training data.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-21T22:06:34.997Z
Answers.LastActivityDate: 2021-07-22T11:42:43.620Z
Answers.LastEditDate: 2021-07-22T11:42:43.620Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the threshold.</p>
  <p>My question is: should I select state-action pairs by their immediate reward or by the episode reward?</p>
  <p>I am applying the method to a craft environment interesting to me and I have been studying an example applying the OpenAI's Gym taxi environment, but I do not fully understand the code.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-22T10:39:31.823Z
Comments.Text: Hello. I've edited your post in order to clarify what your question is by putting it in the title. Please, make sure that the current version of your post is still consistent with the original one. If not, feel free to edit your post again to fix that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-22T11:58:19.030Z
Comments.Text: Although the edition is Ok, I am not sure if monte-carlo-methods as tag is correct.
Comments.UserDisplayName: Hermes Morales
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-22T12:36:34.737Z
Comments.Text: It is Monte Carlo in the general sense of simulating outcomes by sampling from random variables (in this case only the policy). I would not have used `monte-carlo-methods` here, CEM is not a variant of Monte Carlo control. Although you could consider REINFORCE algorithm strongly related to CEM, and an "upgrade" of it.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-22T14:47:53.103Z
Comments.Text: It may be a good idea to provide a reference/link to the CEM you're referring to, just to make sure we're talking about the same thing.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-22T16:56:25.470Z
Comments.Text: @NeilSlater  I am refining my skills related to Reinforcement Learning after doing some online courses in Coursera. I intend to apply these knowledge area to the adaptation of livestock farmers to climate variability. As an excercise, I am trying to deep my expertise in the different kind of  RL and related algorithms. For that, initially I must master the environment construction under different perspectives, tabular and approximation methods, discrete and continuous state and action spaces, and solidify my familiarity with the discipline
Comments.UserDisplayName: Hermes Morales
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-22T16:59:53.330Z
Comments.Text: @nbro It would be a very good new to have some comments on my very initial progress. For that I must also improve my Github knowledge in order to publish the different versions that I'll be producing in the next months. For the moment I do not have any working solution....I'll let you know as soon as I have any concrete version.
Comments.UserDisplayName: Hermes Morales
Tags: <reinforcement-learning><rewards><monte-carlo-methods><return>
Title: In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
UserDisplayName: Hermes Morales
ThreadText: # In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
  
  (Asked by: username_0 on 2021-07-21)
  
  I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the threshold.
  
  My question is: should I select state-action pairs by their immediate reward or by the episode reward?
  
  I am applying the method to a craft environment interesting to me and I have been studying an example applying the OpenAI's Gym taxi environment, but I do not fully understand the code.
  
  **Tags:** reinforcement-learning, rewards, monte-carlo-methods, return
  
  > **Comments:**
  > **username_1**: Hello. I've edited your post in order to clarify what your question is by putting it in the title. Please, make sure that the current version of your post is still consistent with the original one. If not, feel free to edit your post again to fix that.  
  > **username_0**: Although the edition is Ok, I am not sure if monte-carlo-methods as tag is correct.  
  > **username_2**: It is Monte Carlo in the general sense of simulating outcomes by sampling from random variables (in this case only the policy). I would not have used `monte-carlo-methods` here, CEM is not a variant of Monte Carlo control. Although you could consider REINFORCE algorithm strongly related to CEM, and an "upgrade" of it.  
  > **username_1**: It may be a good idea to provide a reference/link to the CEM you're referring to, just to make sure we're talking about the same thing.  
  > **username_0**: @username_2  I am refining my skills related to Reinforcement Learning after doing some online courses in Coursera. I intend to apply these knowledge area to the adaptation of livestock farmers to climate variability. As an excercise, I am trying to deep my expertise in the different kind of  RL and related algorithms. For that, initially I must master the environment construction under different perspectives, tabular and approximation methods, discrete and continuous state and action spaces, and solidify my familiarity with the discipline  
  > **username_0**: @username_1 It would be a very good new to have some comments on my very initial progress. For that I must also improve my Github knowledge in order to publish the different versions that I'll be producing in the next months. For the moment I do not have any working solution....I'll let you know as soon as I have any concrete version.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-07-21. Score: 1 | Accepted answer
  
  > My question is if I should select state\_action pairs by theirs immediate reward or should I select them by the episode reward?
  
  By the return (sum of all rewards) from the whole episode. A lot of decisions made in "good" episodes do not lead to immediate rewards, but instead transition towards states where better rewards are possible.
  
  In retrospect, you do not know whether any single action was a good choice, but with the cross entropy method (CEM) you rely on the fact that on average the better episodes will contain more good decisions than the worse episodes, so you train the policy neural network on the state, action pairs from the elite episodes as if all the decisions were correct. This will not be true, but will hopefully be true more often than by chance, so the policy should improve.
  
  This can be a noisy, high variance approach with any RL method. CEM is one of the most sensitive to noise and variance. However, the taxi environment is deterministic, and that makes using CEM more reasonable.
  
  > **Comments:**
  > **username_0**: So, I should select all the state-action pairs in each episode with a total_reward bigger than the treshold. Am I ok?  
  > **username_2**: @username_0 Yes. If an episode is "elite", then all the state/action pairs in it should be considered good for training data.  
  
  ---
  
  