Answers.Body: <p>For this, we will need game theory.</p>
  <p>In game theory, an optimal strategy is one that cannot be exploited by the opponent <em>even if they know your strategy</em>.</p>
  <p>Let's say you want a strategy where your move selection is not based on what happened before (so you are not trying to model your opponent, or trick them into believing you will always play scissors and then throw them off, anything like that).
  A strategy will look like <span class="math-container">$(P, S, R)$</span>, where <span class="math-container">$P, S, R \in [0, 1], P+S+R = 1$</span>. You select paper with probability <span class="math-container">$P$</span>, scissors with probability <span class="math-container">$S$</span>, rock with probability <span class="math-container">$R$</span>.
  Now, if your probabilities are a bit uneven (for example <span class="math-container">$(0.5, 0.2, 0.3)$</span>) an opponent can abuse that strategy. If your opponent plays with probabilities <span class="math-container">$(p, s, r)$</span>, their expected reward (counting +1 for win, -1 for loss, 0 for draw) would be <span class="math-container">$0.5(s - r) + 0.2(r - p) + 0.3(p - s) = 0.1p + 0.2s - 0.3r$</span>. If they wish to maximize their wins, they would play scissors all the time against you, and expect to have a distinct advantage over you.</p>
  <p>In general, for a strategy <span class="math-container">$(P, S, R)$</span> for you and <span class="math-container">$(p, s, r)$</span> for your opponent, your opponent's winnings would be <span class="math-container">$P(s - r) + S(r - p) + R(p - s) =  p(R-S) + s(P-R) + r(S - P)$</span>. If all the partial derivatives of this, with respect to <span class="math-container">$p$</span>, <span class="math-container">$s$</span> and <span class="math-container">$r$</span> are 0, the opponent has no way to maximize his winnings; they would have no incentive to play a particular move over any other move.
  This occurs when <span class="math-container">$P = S = R = \frac13$</span>.</p>
  <p>That's basically how to approach game theory: find a strategy so your opponent has no incentive to choose one action over another. The approach seems a bit counter-intuitive at first (you're trying to find the optimal strategy for your opponent instead of for yourself) but it works for many similar problems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-27T20:40:49.447Z
Answers.Text: @DuttaA This answer doesn't make that assumption anywhere. In fact the "choose equally distributed randomly" strategy will work the same against any kind of strategy.
Answers.UserDisplayName: Helena
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-27T20:47:17.727Z
Answers.Text: @DuttaA The expression uses unknown variables for the opponents probabilities  (p,s,r). It only uses constant values for the own example strategy (0.5, 0.2, 0.3) .
Answers.UserDisplayName: Helena
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-27T21:00:22.433Z
Answers.Text: @DuttaA You are right, in reality you don't know each others strategy. Because you cannot know what kind of opponent you will get, the best thing you can do is to assume your strongest possible foe.
Answers.UserDisplayName: Helena
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-29T09:25:37.090Z
Answers.Text: @Helena Assuming a strong foe can make you underperform. You should maximize your expected reward using a probability distribution over the opponents. E.g., when playing with children, you should not use random play.
Answers.UserDisplayName: HappyFace
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-27T16:22:39.593Z
Answers.LastActivityDate: 2020-08-27T16:22:39.593Z
Answers.UserDisplayName: Robby Goetschalckx
Body: <p>A deterministic policy in the rock-paper-scissors game can be easily exploited by the opponent - by doing just the right sequence of moves to defeat the agent. More often than not, I've <em>heard</em> that <strong>a random policy</strong> is the <strong>optimal</strong> policy in this case - but the argument seems a little informal.</p>
  <p>Could someone please expound on this, possibly adding more mathematical details and intuition? I guess the case I'm referring to is that of a game between <strong>two RL agents</strong>, but I'd be happy to learn about other cases too. Thanks!</p>
  <p>EDIT: When would a random policy be optimal in this case?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-27T13:48:11.700Z
Comments.Text: Interesting - if it's two RL agents playing against each other, the deterministic policy is definitely not optimal right? RL agents can adapt actively!
Comments.UserDisplayName: stoic-santiago
Tags: <reinforcement-learning><game-theory><optimal-policy>
Title: What's the optimal policy in the rock-paper-scissors game?
UserDisplayName: stoic-santiago
ThreadText: # What's the optimal policy in the rock-paper-scissors game?
  
  (Asked by: username_0 on 2020-08-27)
  
  A deterministic policy in the rock-paper-scissors game can be easily exploited by the opponent - by doing just the right sequence of moves to defeat the agent. More often than not, I've *heard* that **a random policy** is the **optimal** policy in this case - but the argument seems a little informal.
  
  Could someone please expound on this, possibly adding more mathematical details and intuition? I guess the case I'm referring to is that of a game between **two RL agents**, but I'd be happy to learn about other cases too. Thanks!
  
  EDIT: When would a random policy be optimal in this case?
  
  **Tags:** reinforcement-learning, game-theory, optimal-policy
  > **Comments:**
  > **username_0**: Interesting - if it's two RL agents playing against each other, the deterministic policy is definitely not optimal right? RL agents can adapt actively!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-27. Score: 6 | Accepted answer
  
  For this, we will need game theory.
  
  In game theory, an optimal strategy is one that cannot be exploited by the opponent *even if they know your strategy*.
  
  Let's say you want a strategy where your move selection is not based on what happened before (so you are not trying to model your opponent, or trick them into believing you will always play scissors and then throw them off, anything like that).
  A strategy will look like $(P, S, R)$, where $P, S, R \in [0, 1], P+S+R = 1$. You select paper with probability $P$, scissors with probability $S$, rock with probability $R$.
  Now, if your probabilities are a bit uneven (for example $(0.5, 0.2, 0.3)$) an opponent can abuse that strategy. If your opponent plays with probabilities $(p, s, r)$, their expected reward (counting +1 for win, -1 for loss, 0 for draw) would be $0.5(s - r) + 0.2(r - p) + 0.3(p - s) = 0.1p + 0.2s - 0.3r$. If they wish to maximize their wins, they would play scissors all the time against you, and expect to have a distinct advantage over you.
  
  In general, for a strategy $(P, S, R)$ for you and $(p, s, r)$ for your opponent, your opponent's winnings would be $P(s - r) + S(r - p) + R(p - s) = p(R-S) + s(P-R) + r(S - P)$. If all the partial derivatives of this, with respect to $p$, $s$ and $r$ are 0, the opponent has no way to maximize his winnings; they would have no incentive to play a particular move over any other move.
  This occurs when $P = S = R = \frac13$.
  
  That's basically how to approach game theory: find a strategy so your opponent has no incentive to choose one action over another. The approach seems a bit counter-intuitive at first (you're trying to find the optimal strategy for your opponent instead of for yourself) but it works for many similar problems.
  
  > **Comments:**
  > **username_2**: @DuttaA This answer doesn't make that assumption anywhere. In fact the "choose equally distributed randomly" strategy will work the same against any kind of strategy.  
  > **username_2**: @DuttaA The expression uses unknown variables for the opponents probabilities  (p,s,r). It only uses constant values for the own example strategy (0.5, 0.2, 0.3) .  
  > **username_2**: @DuttaA You are right, in reality you don't know each others strategy. Because you cannot know what kind of opponent you will get, the best thing you can do is to assume your strongest possible foe.  
  > **username_3**: @Helena Assuming a strong foe can make you underperform. You should maximize your expected reward using a probability distribution over the opponents. E.g., when playing with children, you should not use random play.  
  
  ---
  
  