Body: <blockquote>
    <p>In the information theory, the entropy is a measure of uncertainty in
    some system. Being applied to agent policy, entropy shows how much the
    agent is uncertain about which action to make. In math notation,
    entropy of the policy is defined  as : <span class="math-container">$$H(\pi) = -\sum \pi(a|s) \log
   \pi(a|s)$$</span> The value of entropy is always greater than zero and has a
    single maximum when the policy is uniform. In other words, all actions
    have the same probability. Entropy becomes minimal when our policy has
    1 for some action and 0 for all others, which means that the agent is
    absolutely sure what to do. To prevent our agent from being stuck in
    the local minimum, we are subtracting the entropy from the loss
    function, punishing the agent for being too certain about the action
    to take.</p>
  </blockquote>
  
  <p>The above excerpt is from Maxim Lapan in the book <em>Deep Reinforcement Learning Hands-on</em> page 254.</p>
  
  <p>In code, it might look like : </p>
  
  <pre><code> optimizer.zero_grad()
   logits= PG_network(batch_states_ts)
   log_prob = F.log_softmax(logits, dim=1)
   log_prob_actions = batch_scales_ts * log_prob[range(params["batch_size"]), batch_actions_ts]
   loss_policy = -log_prob_actions.mean()
  
   prob = F.softmax(logits, dim=1)
   entropy = -(prob * log_prob).sum(dim=1).mean()
   entropy_loss = params["entropy_beta"] * entropy
   loss = loss_policy - entropy_loss
  </code></pre>
  
  <p>I know that a disadvantage of using policy gradient is our agent can be stuck at a local minimum. Can you explain <strong>mathematically</strong> why subtracting the entropy from our policy will  <em>prevent our agent from being stuck in the local minimum</em> ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-25T14:49:00.757Z
Comments.Text: In other words, what author wanted to say is that entropy helps exploration. If you don't explore you might get suboptimal policy. Local minimum is an (overused) term to signify something suboptimal, in this case policy. Using entropy term will not prevent getting suboptimal policies but it might help getting better policies because you're exploring more.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-25T14:53:36.383Z
Comments.Text: @Brale_ It says "subtracting the entropy from our policy will **prevent** our agent from being stuck in the local minimum". So after the training we should always get the optimal policy or close. This is what I understand
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-25T15:13:28.310Z
Comments.Text: Well no, if it was that easy to always get optimal policy just by adding entropy reinforcement learning would be solved. You're taking it too literally, it just means you're helping exploration by adding entropy.
Comments.UserDisplayName: Brale
Tags: <reinforcement-learning><python><policy-gradients>
Title: Subtracting the entropy from our policy gradient will prevent our agent from being stuck in the local minimum?
UserDisplayName: jgauth
ThreadText: # Subtracting the entropy from our policy gradient will prevent our agent from being stuck in the local minimum?
  
  (Asked by: username_0 on 2020-04-25)
  
  In the information theory, the entropy is a measure of uncertainty in
    some system. Being applied to agent policy, entropy shows how much the
    agent is uncertain about which action to make. In math notation,
    entropy of the policy is defined  as : $$H(\pi) = -\sum \pi(a|s) \log
   \pi(a|s)$$ The value of entropy is always greater than zero and has a
    single maximum when the policy is uniform. In other words, all actions
    have the same probability. Entropy becomes minimal when our policy has
    1 for some action and 0 for all others, which means that the agent is
    absolutely sure what to do. To prevent our agent from being stuck in
    the local minimum, we are subtracting the entropy from the loss
    function, punishing the agent for being too certain about the action
    to take.
  
  The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 254.
  In code, it might look like : 
   optimizer.zero_grad()
   logits= PG_network(batch_states_ts)
   log_prob = F.log_softmax(logits, dim=1)
   log_prob_actions = batch_scales_ts * log_prob[range(params["batch_size"]), batch_actions_ts]
   loss_policy = -log_prob_actions.mean()
  
   prob = F.softmax(logits, dim=1)
   entropy = -(prob * log_prob).sum(dim=1).mean()
   entropy_loss = params["entropy_beta"] * entropy
   loss = loss_policy - entropy_loss
  
  I know that a disadvantage of using policy gradient is our agent can be stuck at a local minimum. Can you explain mathematically why subtracting the entropy from our policy will  prevent our agent from being stuck in the local minimum ?
  
  **Tags:** reinforcement-learning, python, policy-gradients
  > **Comments:**
  > **username_1**: In other words, what author wanted to say is that entropy helps exploration. If you don't explore you might get suboptimal policy. Local minimum is an (overused) term to signify something suboptimal, in this case policy. Using entropy term will not prevent getting suboptimal policies but it might help getting better policies because you're exploring more.  
  > **username_0**: @Brale_ It says "subtracting the entropy from our policy will **prevent** our agent from being stuck in the local minimum". So after the training we should always get the optimal policy or close. This is what I understand  
  > **username_1**: Well no, if it was that easy to always get optimal policy just by adding entropy reinforcement learning would be solved. You're taking it too literally, it just means you're helping exploration by adding entropy.  
  
  ---
  
  ## 0 answers
  
  