Answers.Body: <p>I would argue it is just stochastic because it chooses the current best action with probability <span class="math-container">$1-\epsilon+\epsilon/|A|$</span> and then selects randomly among the rest of the actions with the remaining probability <span class="math-container">$\epsilon/|A|$</span>, where <span class="math-container">$A$</span> is the action space. The current best action is updated over time with running averages and may be the same one in the long run if it is truly a stationary bandit environment, but it will still explore.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T19:16:53.317Z
Answers.Text: Technically it selects the current best action with probability $1 - \epsilon + \epsilon / \vert A \vert$. In the case where we select actions uniformly at random (with probability $\epsilon$), one of the options is generally again that we randomly select the best action :)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T03:30:30.923Z
Answers.Text: @DennisSoemers, can you explain how you got the ϵ/|A| part?
Answers.UserDisplayName: asdfasdfsdf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T09:45:59.007Z
Answers.Text: @Melanol With probability $\epsilon$, we sample one action uniformly at random from a set of $\vert A \vert$ actions. Sampling "uniformly at random" means we assign equal probabilities to all those actions, which means that each of those actions (including the "best" one) gets a probability of $1 / \vert A \vert$. That gets multiplied by $\epsilon$ because the whole sampling-uniformly-at-random thing only happens with probability $\epsilon$
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T14:26:37.503Z
Answers.LastActivityDate: 2022-07-06T19:41:04.430Z
Answers.LastEditDate: 2022-07-06T19:41:04.430Z
Answers.UserDisplayName: sma
Answers.Body: <p>Yes - you can think of an epsilon-greedy policy as a mixture of a policy that chooses an action at random (the stochastic part) and a possibly deterministic policy used otherwise. The value of epsilon gives the weight of the random component, and <span class="math-container">$1-\epsilon$</span> that of the other component.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T08:44:44.283Z
Answers.LastActivityDate: 2022-07-06T08:44:44.283Z
Answers.UserDisplayName: mikkola
Answers.Body: <p>When a policy is stochastic, it means taking actions will be done based on probabilities. For example, in deterministic policy (in the case of navigation for four actions), if an agent takes upward action, it will go up and etc. However, in stochastic policy, if an agent takes upward, it may go up with 80% of probability and go right with 20% of possibility. Thus, in epsilon-greedy based policies, we choose actions in exploration mode randomly, but it does not mean it is stochastic.
  I tried to explain short and clear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T14:06:34.833Z
Answers.LastActivityDate: 2022-07-06T14:06:34.833Z
Answers.UserDisplayName: Pouyan
Body: <p>Even though if exploration doesn't happen, it's deterministic.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-06T15:34:04.460Z
Comments.Text: I found this is the book "Intro to RL": " In general, policies may be stochastic, specifying
  probabilities for each action." So, according to the book, it is not, but I was supposed to use a stochastic policy in n-step Sarsa, instead, I used epsilon-greedy, and the algorithm still learns.
Comments.UserDisplayName: asdfasdfsdf
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-06T20:26:25.747Z
Comments.Text: I am not sure I understand the purpose of this question. Are you asking "how much stochasticity there must be in a policy in order to be considered a stochastic policy?" How do you want to measure that? Or are you asking whether a policy that sometimes is not stochastic should be considered stochastic or not? It's a matter of convention, but I would say that it makes more sense to consider this policy stochastic (because it's sometimes stochastic) than deterministic (because it's not always deterministic). Maybe you can classify policies into classes of stochasticity.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-06T20:29:58.150Z
Comments.Text: My point is - If you define determinism as a "[view where all events are determined completely by previously existing causes](https://en.wikipedia.org/wiki/Determinism)", then $\epsilon$-greedy is not deterministic, so it must be the opposite? It may also be worth pointing out that we could consider $\epsilon$-greedy actually an infinite set of policies, where at one extreme you may have something that is completely random, at at the other extreme you have something that is completely deterministic.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><definitions><policies><epsilon-greedy-policy>
Title: If a policy is epsilon-greedy, is it technically stochastic?
UserDisplayName: asdfasdfsdf
ThreadText: # If a policy is epsilon-greedy, is it technically stochastic?
  
  (Asked by: username_0 on 2022-07-06)
  
  Even though if exploration doesn't happen, it's deterministic.
  
  **Tags:** reinforcement-learning, definitions, policies, epsilon-greedy-policy
  
  > **Comments:**
  > **username_0**: I found this is the book "Intro to RL": " In general, policies may be stochastic, specifying probabilities for each action." So, according to the book, it is not, but I was supposed to use a stochastic policy in n-step Sarsa, instead, I used epsilon-greedy, and the algorithm still learns.  
  > **username_1**: I am not sure I understand the purpose of this question. Are you asking "how much stochasticity there must be in a policy in order to be considered a stochastic policy?" How do you want to measure that? Or are you asking whether a policy that sometimes is not stochastic should be considered stochastic or not? It's a matter of convention, but I would say that it makes more sense to consider this policy stochastic (because it's sometimes stochastic) than deterministic (because it's not always deterministic). Maybe you can classify policies into classes of stochasticity.  
  > **username_1**: My point is - If you define determinism as a "[view where all events are determined completely by previously existing causes](https://en.wikipedia.org/wiki/Determinism)", then $\epsilon$-greedy is not deterministic, so it must be the opposite? It may also be worth pointing out that we could consider $\epsilon$-greedy actually an infinite set of policies, where at one extreme you may have something that is completely random, at at the other extreme you have something that is completely deterministic.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2022-07-06. Score: 3
  
  I would argue it is just stochastic because it chooses the current best action with probability $1-\epsilon+\epsilon/|A|$ and then selects randomly among the rest of the actions with the remaining probability $\epsilon/|A|$, where $A$ is the action space. The current best action is updated over time with running averages and may be the same one in the long run if it is truly a stationary bandit environment, but it will still explore.
  
  > **Comments:**
  > **username_3**: Technically it selects the current best action with probability $1 - \epsilon + \epsilon / \vert A \vert$. In the case where we select actions uniformly at random (with probability $\epsilon$), one of the options is generally again that we randomly select the best action :)  
  > **username_0**: @username_3, can you explain how you got the ϵ/|A| part?  
  > **username_3**: @Melanol With probability $\epsilon$, we sample one action uniformly at random from a set of $\vert A \vert$ actions. Sampling "uniformly at random" means we assign equal probabilities to all those actions, which means that each of those actions (including the "best" one) gets a probability of $1 / \vert A \vert$. That gets multiplied by $\epsilon$ because the whole sampling-uniformly-at-random thing only happens with probability $\epsilon$  
  
  ---
  
  ### Answer by username_4 on 2022-07-06. Score: 2
  
  Yes - you can think of an epsilon-greedy policy as a mixture of a policy that chooses an action at random (the stochastic part) and a possibly deterministic policy used otherwise. The value of epsilon gives the weight of the random component, and $1-\epsilon$ that of the other component.
  
  
  ---
  
  ### Answer by username_5 on 2022-07-06. Score: 0
  
  When a policy is stochastic, it means taking actions will be done based on probabilities. For example, in deterministic policy (in the case of navigation for four actions), if an agent takes upward action, it will go up and etc. However, in stochastic policy, if an agent takes upward, it may go up with 80% of probability and go right with 20% of possibility. Thus, in epsilon-greedy based policies, we choose actions in exploration mode randomly, but it does not mean it is stochastic.
  I tried to explain short and clear.
  
  
  ---
  
  