Answers.Body: <p>My understanding of your question is, you have 2 designs:</p>
  
  <ol>
  <li>A deterministic policy that outputs 2 scalar for x and y respectively.</li>
  <li>A value function that outputs the probability of each pixel in the 2D grid.</li>
  </ol>
  
  <p>If you choose the max of softmax on (2.), you'll get the same deterministic policy as (1.), assuming there are some tie-breaking designs. So I don't think choosing the max of softmax is a great policy.</p>
  
  <p>So, the question becomes "deterministic policy vs. value function". Since the value function have more information, we can have special designs on exploration.</p>
  
  <p>In (1.), if you use <span class="math-container">$\epsilon$</span>-greedy on the discrete output, It'll take a random move with <span class="math-container">$\epsilon$</span> probability. When exploring (instead of exploiting), all actions except the best action will have the same probability.</p>
  
  <p>However, in (2.), the probability can act like a value function (the value of <span class="math-container">$Q(s, a) \forall a$</span> in a certain state <span class="math-container">$s$</span>), and you can use something like upper confidence bound (UCB) instead of <span class="math-container">$\epsilon$</span>-greedy. If you are using neural networks to output this probability distribution, you can <a href="https://github.com/dennybritz/reinforcement-learning/issues/34" rel="nofollow noreferrer">add an entropy term in the loss function like in A3C</a> to encourage exploration.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-18T16:34:58.253Z
Answers.Text: That makes sense. Only, If you use softmax on (2.), and do a random sample on that distribution, you do get a non-deterministic output right? Or am I incorrect?
Answers.UserDisplayName: SumakuTension
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-18T18:44:51.813Z
Answers.Text: @SumakuTension Oh yes, I meant choosing the max value of the softmaxed output isn't a good policy. Performing a random sample on the distribution does make it non-deterministic.
Answers.UserDisplayName: J3soon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-18T18:46:57.807Z
Answers.Text: I've edited the answer, thanks for the correction!
Answers.UserDisplayName: J3soon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-18T13:06:44.600Z
Answers.LastActivityDate: 2019-12-18T18:46:22.643Z
Answers.LastEditDate: 2019-12-18T18:46:22.643Z
Answers.UserDisplayName: J3soon
Body: <p>I have a question about a reinforcement learning problem. </p>
  
  <p>I'm training an agent to add or delete pixels in a [12 x 12] 2D space (going to be 3D in the future).
  Its action space consists of two discrete outputs: x[0-12] and y[0-12].</p>
  
  <p>What would be the value of instead outputting a (continuous) probabilistic output representation, like the [12 x 12] space with each pixel as a probability, and sampling from it. E.g. a softmax function applied to 144 (12*12) output nodes.</p>
  
  <p>My environment is deterministic itself: taking action  in state  always results in the same next state ′.</p>
  
  <p>I understand that this may be more difficult to train since the output space becomes continuous instead of discrete, and therefore bigger, but does stochastic/probabilistic output have any benefits over 1 discrete output?</p>
  
  <p>Thanks!</p>
  
Tags: <reinforcement-learning><stochastic-policy>
Title: What's the value of making the RL agent's output stochastic opposed to deterministic?
UserDisplayName: SumakuTension
ThreadText: # What's the value of making the RL agent's output stochastic opposed to deterministic?
  
  (Asked by: username_0 on 2019-12-15)
  
  I have a question about a reinforcement learning problem.
  
  I'm training an agent to add or delete pixels in a [12 x 12] 2D space (going to be 3D in the future).
  Its action space consists of two discrete outputs: x[0-12] and y[0-12].
  
  What would be the value of instead outputting a (continuous) probabilistic output representation, like the [12 x 12] space with each pixel as a probability, and sampling from it. E.g. a softmax function applied to 144 (12\*12) output nodes.
  
  My environment is deterministic itself: taking action in state always results in the same next state ′.
  
  I understand that this may be more difficult to train since the output space becomes continuous instead of discrete, and therefore bigger, but does stochastic/probabilistic output have any benefits over 1 discrete output?
  
  Thanks!
  
  **Tags:** reinforcement-learning, stochastic-policy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-18. Score: 2 | Accepted answer
  
  My understanding of your question is, you have 2 designs:
  
  1. A deterministic policy that outputs 2 scalar for x and y respectively.
  2. A value function that outputs the probability of each pixel in the 2D grid.
  
  If you choose the max of softmax on (2.), you'll get the same deterministic policy as (1.), assuming there are some tie-breaking designs. So I don't think choosing the max of softmax is a great policy.
  
  So, the question becomes "deterministic policy vs. value function". Since the value function have more information, we can have special designs on exploration.
  
  In (1.), if you use $\epsilon$-greedy on the discrete output, It'll take a random move with $\epsilon$ probability. When exploring (instead of exploiting), all actions except the best action will have the same probability.
  
  However, in (2.), the probability can act like a value function (the value of $Q(s, a) \forall a$ in a certain state $s$), and you can use something like upper confidence bound (UCB) instead of $\epsilon$-greedy. If you are using neural networks to output this probability distribution, you can [add an entropy term in the loss function like in A3C](https://github.com/dennybritz/reinforcement-learning/issues/34) to encourage exploration.
  
  > **Comments:**
  > **username_0**: That makes sense. Only, If you use softmax on (2.), and do a random sample on that distribution, you do get a non-deterministic output right? Or am I incorrect?  
  > **username_1**: @SumakuTension Oh yes, I meant choosing the max value of the softmaxed output isn't a good policy. Performing a random sample on the distribution does make it non-deterministic.  
  > **username_1**: I've edited the answer, thanks for the correction!  
  
  ---
  
  