Body: <p>I wrote two programs that simulated 10000 episodes in gym environment CartPole-v0.</p>
  <p>The first program takes random moves in every steps in each episode. The average reward over 10000 episodes is 22.1582.</p>
  <p>The second program uses a random policy in each episode. For each episode, initialize a 4 by 2 matrix <span class="math-container">$M$</span> with random numbers from a uniform distribution on <span class="math-container">$[0,1)$</span> that maps state observations to action values. Then choose the action with higher value in each step. The average reward over 10000 episodes is 46.8291.</p>
  <p>The linear mapping given by <span class="math-container">$M$</span> covers only a portion of the search space so it seems the way actions are selected in the first program is more &quot;random&quot; than the second program. How can we go about explaining the huge discrepancy in the average rewards obtained by the two methods?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T09:30:45.290Z
Comments.Text: Cartpole has far more than 4 by 2 state/actions. How are you mapping from state to lookup of action value in your 4 by 2 table? I would guess that the 2 corresponds to "move left, move right" evaluation. But how are you altering the state in order to look it up in the table?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T02:52:59.187Z
Comments.Text: Matrix $M$ is not a lookup table. It is a matrix that maps a vector of size 4 (representing the state) to a vector of size 2 (representing the action values)
Comments.UserDisplayName: artils1997
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T10:14:10.753Z
Comments.Text: So you are using the matrix to multiply the state vector? Essentially that is a linear value approximation. Except the values will be meaningless/wrong, so essentially it's a randomised policy with a few inflection points between left/right actions
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T13:55:49.657Z
Comments.Text: Yes they are randomly generated policies. Basically, I am just wondering why randomly generated policies, on average, perform better than taking random actions in terms of the average returns from each episode.
Comments.UserDisplayName: artils1997
Tags: <reinforcement-learning><gym>
Title: Difference in average rewards between taking random actions and following random policies
UserDisplayName: artils1997
ThreadText: # Difference in average rewards between taking random actions and following random policies
  
  (Asked by: username_0 on 2021-01-16)
  
  I wrote two programs that simulated 10000 episodes in gym environment CartPole-v0.
  
  The first program takes random moves in every steps in each episode. The average reward over 10000 episodes is 22.1582.
  
  The second program uses a random policy in each episode. For each episode, initialize a 4 by 2 matrix $M$ with random numbers from a uniform distribution on $[0,1)$ that maps state observations to action values. Then choose the action with higher value in each step. The average reward over 10000 episodes is 46.8291.
  
  The linear mapping given by $M$ covers only a portion of the search space so it seems the way actions are selected in the first program is more "random" than the second program. How can we go about explaining the huge discrepancy in the average rewards obtained by the two methods?
  
  **Tags:** reinforcement-learning, gym
  > **Comments:**
  > **username_1**: Cartpole has far more than 4 by 2 state/actions. How are you mapping from state to lookup of action value in your 4 by 2 table? I would guess that the 2 corresponds to "move left, move right" evaluation. But how are you altering the state in order to look it up in the table?  
  > **username_0**: Matrix $M$ is not a lookup table. It is a matrix that maps a vector of size 4 (representing the state) to a vector of size 2 (representing the action values)  
  > **username_1**: So you are using the matrix to multiply the state vector? Essentially that is a linear value approximation. Except the values will be meaningless/wrong, so essentially it's a randomised policy with a few inflection points between left/right actions  
  > **username_0**: Yes they are randomly generated policies. Basically, I am just wondering why randomly generated policies, on average, perform better than taking random actions in terms of the average returns from each episode.  
  
  ---
  
  ## 0 answers
  
  