Answers.Body: <p>The two implementations you posted are different, but they do represent the same <span class="math-container">$\epsilon$</span>-greedy policy.</p>
  <p>The first function returns an array <code>A</code> which contains the probabilities of each action choice. In order to get an action choice you need to run something like <code>np.random.choice(np.arange(len(A)), p=A)</code> to sample from it.</p>
  <p>The second function directly returns an action choice.</p>
  <p>You might want the policy probabilities from the first approach for some kinds of reinforcement learning methods. For instance, <em>importance sampling</em> for off-policy learning needs to divide target policy probability by behaviour policy probability. Having a policy method that returns the full distribution of action choices is a generic way to represent a policy, allowing you to work with the policy in other contexts than taking an action.</p>
  <p>You might otherwise prefer the second direct action choice because then you do not need to add extra code to make that choice in the inner loop of the learning algorithm. Technically this is not just representing the policy, but actually sampling from the policy as well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-01T08:02:01.920Z
Answers.LastActivityDate: 2023-12-01T08:51:53.790Z
Answers.LastEditDate: 2023-12-01T08:51:53.790Z
Answers.UserDisplayName: Neil Slater
Body: <p>According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:</p>
  <p><span class="math-container">$$
  \pi(a|s) = 
  \begin{cases} 
  \frac{\epsilon}{|A|} + 1 - \epsilon &amp; \text{if } a = \arg\max_{a'} Q(s, a') \\\\
  \frac{\epsilon}{|A|} &amp; \text{otherwise}
  \end{cases}
  $$</span></p>
  <p>I followed an exercise from</p>
  <p><a href="https://github.com/dennybritz/reinforcement-learning/blob/master/TD/Q-Learning%20Solution.ipynb" rel="nofollow noreferrer">https://github.com/dennybritz/reinforcement-learning/blob/master/TD/Q-Learning%20Solution.ipynb</a></p>
  <p>where the policy is implemented in Python like this:</p>
  <pre><code>def policy_fn(observation):
      A = np.ones(nA, dtype=float) * epsilon / nA
      best_action = np.argmax(Q[observation])
      A[best_action] += (1.0 - epsilon)
      return A
  </code></pre>
  <p>I wonder why this implementation is different from:</p>
  <pre><code>def policy_fn(observation):
      p = np.random.rand()
      if p &gt; epsilon:
          return np.argmax(Q[observation])
      return np.random.randint(0, nA)
  </code></pre>
  <p>At least, I get different results for the final action-state pairs.
  In my opinion, the best action gets sampled with a probability of <span class="math-container">$1 -\epsilon$</span>. Otherwise, each action gets chosen with the same probability.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-30T15:06:34.110Z
Comments.Text: they look equivalent to me. let's look at what happens: with probability $\epsilon$, an action is chosen uniformly at random. That means that all actions have a probability of $\epsilon / |\mathcal{A}|$ of being selected. However, with probability $1 - \epsilon$, we select the greedy action. So, the probability of the greedy action being chosen is $\epsilon / |\mathcal{A}| + 1 - \epsilon$. Both the functions look to implement the same thing, where the first returns the probability of each action being selected (and thus should then be sampled from) and the latter returns the action directly.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><comparison><python><epsilon-greedy-policy>
Title: Why are these two implementations of the $\epsilon$-greedy policy different?
UserDisplayName: kklaw
ThreadText: # Why are these two implementations of the $\epsilon$-greedy policy different?
  
  (Asked by: username_0 on 2023-11-30)
  
  According to the book Reinforcement Learning An Introduction, the epsilon greedy policy can generally implemented as:
  
  $$
  \pi(a|s) =
  \begin{cases}
  \frac{\epsilon}{|A|} + 1 - \epsilon & \text{if } a = \arg\max\_{a'} Q(s, a') \\\\
  \frac{\epsilon}{|A|} & \text{otherwise}
  \end{cases}
  $$
  
  I followed an exercise from
  
  <https://github.com/dennybritz/reinforcement-learning/blob/master/TD/Q-Learning%20Solution.ipynb>
  
  where the policy is implemented in Python like this:
  
  ```
  def policy_fn(observation):
      A = np.ones(nA, dtype=float) * epsilon / nA
      best_action = np.argmax(Q[observation])
      A[best_action] += (1.0 - epsilon)
      return A
  
  ```
  
  I wonder why this implementation is different from:
  
  ```
  def policy_fn(observation):
      p = np.random.rand()
      if p > epsilon:
          return np.argmax(Q[observation])
      return np.random.randint(0, nA)
  
  ```
  
  At least, I get different results for the final action-state pairs.
  In my opinion, the best action gets sampled with a probability of $1 -\epsilon$. Otherwise, each action gets chosen with the same probability.
  
  **Tags:** reinforcement-learning, comparison, python, epsilon-greedy-policy
  
  > **Comments:**
  > **username_1**: they look equivalent to me. let's look at what happens: with probability $\epsilon$, an action is chosen uniformly at random. That means that all actions have a probability of $\epsilon / |\mathcal{A}|$ of being selected. However, with probability $1 - \epsilon$, we select the greedy action. So, the probability of the greedy action being chosen is $\epsilon / |\mathcal{A}| + 1 - \epsilon$. Both the functions look to implement the same thing, where the first returns the probability of each action being selected (and thus should then be sampled from) and the latter returns the action directly.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-12-01. Score: 1
  
  The two implementations you posted are different, but they do represent the same $\epsilon$-greedy policy.
  
  The first function returns an array `A` which contains the probabilities of each action choice. In order to get an action choice you need to run something like `np.random.choice(np.arange(len(A)), p=A)` to sample from it.
  
  The second function directly returns an action choice.
  
  You might want the policy probabilities from the first approach for some kinds of reinforcement learning methods. For instance, *importance sampling* for off-policy learning needs to divide target policy probability by behaviour policy probability. Having a policy method that returns the full distribution of action choices is a generic way to represent a policy, allowing you to work with the policy in other contexts than taking an action.
  
  You might otherwise prefer the second direct action choice because then you do not need to add extra code to make that choice in the inner loop of the learning algorithm. Technically this is not just representing the policy, but actually sampling from the policy as well.
  
  
  ---
  
  