Body: <p>I have written a code for an RL agent such that at each state the model calculates the probabilities of all possible actions and samples one action randomly to proceed further. To acheive this, I have written the following code</p>
  <pre><code>act_prob_dist = tfp.distributions.Categorical(probs=act_probs)
  action = act_prob_dist.sample()
  </code></pre>
  <p>It is working fine in the initial stages of training. Once the model has learnt about the particular state really well and the probability of one particular action has increased significantly than the others, the <code>sample()</code> call is picking is the same action every time. For example, when the action probabilities of a particular state are</p>
  <pre><code>tf.Tensor(
  [[0.05213022 0.06613996 0.4933109  0.02918373 0.04188393 0.04100212
    0.03228914 0.00716161 0.08877521 0.02158365 0.04645196 0.07092285
    0.00916469]], shape=(1, 13), dtype=float32)
  </code></pre>
  <p>The model is sampling an action randomly. After decent iterations of learning the action probabilities became</p>
  <pre><code>tf.Tensor(
  [[1.12852089e-12 1.54888698e-06 6.40413802e-08 1.03480375e-11
    2.05246806e-08 2.17290430e-09 1.04494591e-09 5.20959872e-11
    9.99995708e-01 1.26053008e-08 6.85156265e-10 1.70332885e-06
    9.99039457e-07]], shape=(1, 13), dtype=float32)
  </code></pre>
  <p>The model started picking index with high probability every time(index 8 in this case). The documentation reads <code>Generate samples of the specified shape.</code> I'm assuming it implies the choosing happens randomly. Can someone please explain why same action is being chosen in my case?<br/>
  PS: <code>tf.version</code> is returning <code>tensorflow._api.v2.version</code></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-26T11:56:32.477Z
Comments.Text: Hello. Welcome to Artificial Intelligence Stack Exchange. It seems to me that this is or could be a programming issue. Let me know if I am wrong or right. Generally, questions about programming problems are off-topic here and should probably be asked on Stack Overflow. Here, we focus on the **theoretical, philosophical, social** and **historical** aspects of Artificial Intelligence. Please, take the time to read [our on-topic page](https://ai.stackexchange.com/help/on-topic). So, if you think this is not a programming problem, you should edit your post to clarify that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-27T14:31:16.760Z
Comments.Text: I guess you are right. I posted it on stack overflow but got no responses. So I thought there might be something wrong with the way I'm structuring my agent and this platform might be the right place for me to ask
Comments.UserDisplayName: MonkeyDLuffy
Tags: <reinforcement-learning><deep-learning><tensorflow><deep-rl><tensorflow-probability>
Title: tfp.Distributions.Categorical.sample() is picking the same action everytime after certain iterations
UserDisplayName: MonkeyDLuffy
ThreadText: # tfp.Distributions.Categorical.sample() is picking the same action everytime after certain iterations
  
  (Asked by: username_0 on 2021-11-26)
  
  I have written a code for an RL agent such that at each state the model calculates the probabilities of all possible actions and samples one action randomly to proceed further. To acheive this, I have written the following code
  act_prob_dist = tfp.distributions.Categorical(probs=act_probs)
  action = act_prob_dist.sample()
  
  It is working fine in the initial stages of training. Once the model has learnt about the particular state really well and the probability of one particular action has increased significantly than the others, the sample() call is picking is the same action every time. For example, when the action probabilities of a particular state are
  tf.Tensor(
  [[0.05213022 0.06613996 0.4933109  0.02918373 0.04188393 0.04100212
    0.03228914 0.00716161 0.08877521 0.02158365 0.04645196 0.07092285
    0.00916469]], shape=(1, 13), dtype=float32)
  
  The model is sampling an action randomly. After decent iterations of learning the action probabilities became
  tf.Tensor(
  [[1.12852089e-12 1.54888698e-06 6.40413802e-08 1.03480375e-11
    2.05246806e-08 2.17290430e-09 1.04494591e-09 5.20959872e-11
    9.99995708e-01 1.26053008e-08 6.85156265e-10 1.70332885e-06
    9.99039457e-07]], shape=(1, 13), dtype=float32)
  
  The model started picking index with high probability every time(index 8 in this case). The documentation reads Generate samples of the specified shape. I'm assuming it implies the choosing happens randomly. Can someone please explain why same action is being chosen in my case?
  PS: tf.version is returning tensorflow._api.v2.version
  
  **Tags:** reinforcement-learning, deep-learning, tensorflow, deep-rl, tensorflow-probability
  > **Comments:**
  > **username_1**: Hello. Welcome to Artificial Intelligence Stack Exchange. It seems to me that this is or could be a programming issue. Let me know if I am wrong or right. Generally, questions about programming problems are off-topic here and should probably be asked on Stack Overflow. Here, we focus on the **theoretical, philosophical, social** and **historical** aspects of Artificial Intelligence. Please, take the time to read [our on-topic page](https://ai.stackexchange.com/help/on-topic). So, if you think this is not a programming problem, you should edit your post to clarify that.  
  > **username_0**: I guess you are right. I posted it on stack overflow but got no responses. So I thought there might be something wrong with the way I'm structuring my agent and this platform might be the right place for me to ask  
  
  ---
  
  ## 0 answers
  
  