Answers.Body: <p>Firstly, I would suggest you <strong>do not use softmax for exploration</strong>, because it does not imply the model's uncertainty. Training with softmax and cross-entropy, your model may be very confident, but wrongly, because of overfitting.</p>
  <p>Another reason why you should not use softmax to measure uncertainty is that your estimate of the variance (optimal estimate of the variance is the average error) may be quite low, explaining the low entropy of your model. However, there maybe exist regions (holes), where your model is highly uncertain (large error).</p>
  <p>Thus, you should try other options for measuring uncertainty.</p>
  <ol>
  <li><p>One way to measure model (or <em>epistemic</em>) uncertainty is to use a Bayesian Network estimating the whole distribution <span class="math-container">$p(\theta | D)$</span>, where \theta are your parameters and D is the collected dataset of experience. The entropy of <span class="math-container">$p(\theta | D)$</span> tells us the model uncertainty.
  <em><strong>Why Bayesian Network?</strong></em> Intuitively, if our estimator says all θs are equally likely given your data, then it means that you have no idea of what the model really is. On the other hand, if it says there is one only θ, that can possibly explain D, then you have high confidence in the model.</p>
  </li>
  <li><p>Another way to measure epistemic uncertainty is to use bootstrap ensembling. Instead of estimating the uncertainty of every single parameter in the Bayesian Net, learn N different networks! Use bootstrapping in order to generate N datasets (of same size as D) by resampling with replacement. Then train each network on a corresponding generated dataset. Then in testing, you can either sample uniformly a network (<span class="math-container">$p(\theta | D) = \frac{1}{N}\sum_i^N{\delta({\theta}_i)}$</span>)  or just average all networks predictions.</p>
  </li>
  </ol>
  <p><strong>For exploration</strong> (in Deep RL), there are a lot of different options. I recommend you check <strong>information gain, Thompson sampling and pseudo-counts</strong>. However, if you were using the standard deep RL agents, like DQN etc, you could begin with the standard ε-greedy behavior policy, ignoring that this is not an optimal exploration policy for the most cases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T09:53:23.290Z
Answers.LastActivityDate: 2021-05-02T09:53:23.290Z
Answers.UserDisplayName: ddaedalus
Body: <p>Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like (0,0,1,0,0,0) or continuous softmax like (0.1, 0.15, 0.45, 0.25, 0,5, 0.1)??</p>
  <p>if the answer is continuous, what algorithm do you suggest? like generating random numbers between 0 and 1 and then using softmax? (this algorithm mostly provides close numbers and I think it's not the correct way)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T12:38:20.843Z
Comments.Text: This post is unclear to me. What do you mean by "what random samples should we provide?" Provide to what? Moreover, what "random samples" are you talking about? Samples of what?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T16:28:12.110Z
Comments.Text: Random samples of possible actions for exploration. In the exploitation, the neural network will provide the action for us but in the exploration, we have to generate these samples of action ourselves and feed them to the algorithm. The actual question is for a neural network with softmax in output, these random actions should be binary or continuous.
Comments.UserDisplayName: fardis nadimi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T17:23:42.400Z
Comments.Text: If by samples you mean "actions" (which is what you just said), then I don't understand the question "actions should be continuous or binary".
Comments.UserDisplayName: nbro
Tags: <neural-networks><reinforcement-learning><tensorflow><python><softmax>
Title: Exploration for softmax should be binary or continuous softmax?
UserDisplayName: fardis nadimi
ThreadText: # Exploration for softmax should be binary or continuous softmax?
  
  (Asked by: username_0 on 2021-05-01)
  
  Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like (0,0,1,0,0,0) or continuous softmax like (0.1, 0.15, 0.45, 0.25, 0,5, 0.1)??
  
  if the answer is continuous, what algorithm do you suggest? like generating random numbers between 0 and 1 and then using softmax? (this algorithm mostly provides close numbers and I think it's not the correct way)
  
  **Tags:** neural-networks, reinforcement-learning, tensorflow, python, softmax
  > **Comments:**
  > **username_1**: This post is unclear to me. What do you mean by "what random samples should we provide?" Provide to what? Moreover, what "random samples" are you talking about? Samples of what?  
  > **username_0**: Random samples of possible actions for exploration. In the exploitation, the neural network will provide the action for us but in the exploration, we have to generate these samples of action ourselves and feed them to the algorithm. The actual question is for a neural network with softmax in output, these random actions should be binary or continuous.  
  > **username_1**: If by samples you mean "actions" (which is what you just said), then I don't understand the question "actions should be continuous or binary".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-05-02. Score: 0
  
  Firstly, I would suggest you **do not use softmax for exploration**, because it does not imply the model's uncertainty. Training with softmax and cross-entropy, your model may be very confident, but wrongly, because of overfitting.
  
  Another reason why you should not use softmax to measure uncertainty is that your estimate of the variance (optimal estimate of the variance is the average error) may be quite low, explaining the low entropy of your model. However, there maybe exist regions (holes), where your model is highly uncertain (large error).
  
  Thus, you should try other options for measuring uncertainty.
  
  1. One way to measure model (or *epistemic*) uncertainty is to use a Bayesian Network estimating the whole distribution $p(\theta | D)$, where \theta are your parameters and D is the collected dataset of experience. The entropy of $p(\theta | D)$ tells us the model uncertainty.
     ***Why Bayesian Network?*** Intuitively, if our estimator says all θs are equally likely given your data, then it means that you have no idea of what the model really is. On the other hand, if it says there is one only θ, that can possibly explain D, then you have high confidence in the model.
  2. Another way to measure epistemic uncertainty is to use bootstrap ensembling. Instead of estimating the uncertainty of every single parameter in the Bayesian Net, learn N different networks! Use bootstrapping in order to generate N datasets (of same size as D) by resampling with replacement. Then train each network on a corresponding generated dataset. Then in testing, you can either sample uniformly a network ($p(\theta | D) = \frac{1}{N}\sum\_i^N{\delta({\theta}\_i)}$) or just average all networks predictions.
  
  **For exploration** (in Deep RL), there are a lot of different options. I recommend you check **information gain, Thompson sampling and pseudo-counts**. However, if you were using the standard deep RL agents, like DQN etc, you could begin with the standard ε-greedy behavior policy, ignoring that this is not an optimal exploration policy for the most cases.
  
  
  ---
  
  