Answers.Body: <p>The main point here is that you can write <span class="math-container">$Q(s, a|\theta) = R = \theta^\top \phi(s, a)$</span>. For more details on this, you can read up on Policy Gradients (Chapter 13) from the 2nd edition of the Sutton and Barto book (in fact the expression you're looking for is equation 13.9)</p>
  
  <p>For simplicity, I'm setting <span class="math-container">$\beta=1$</span>, but you can always put it in once you get the idea. Therefore, the expression for <span class="math-container">$\pi(s,a,\theta)$</span> (I'm including <span class="math-container">$\theta$</span> here to make the dependence of <span class="math-container">$\pi$</span> explicit) is now:
  <span class="math-container">$$
  \pi(s,a,\theta) = \frac{exp(\theta^\top \phi(s, a))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} = \frac{A}{B}
  $$</span></p>
  
  <p>Also, it is standard to compute <span class="math-container">$\nabla_\theta\log \pi(s,a,\theta)$</span> instead of <span class="math-container">$\nabla_\theta \pi(s,a,\theta)$</span> (although you totally can), so I'll go ahead and do that. To differentiate <span class="math-container">$\log \pi(s,a,\theta)$</span> with respect to the parameters, <span class="math-container">$\theta$</span>, let's do some calculus:
  <span class="math-container">$$
  \nabla_\theta \log \pi(s,a,\theta) = \nabla_\theta \log \frac{A}{B} = \nabla_\theta \log A - \nabla_\theta \log B \\
  = \nabla_\theta \left[ \theta^\top \phi(s,a) \right] - \nabla_\theta \left[ \log \sum_{a'} exp(\theta^\top \phi(s, a')) \right] \\
  = \phi(s,a) - \frac{1}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \sum_{b} exp(\theta^\top \phi(s, b)) \phi(s,b) \\
  = \phi(s,a) - \sum_{b} \frac{exp(\theta^\top \phi(s, b))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \phi(s,b) \\
  = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)
  $$</span></p>
  
  <p>Therefore, you can write <span class="math-container">$\nabla_\theta\log \pi(s,a,\theta) = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-15T20:49:26.900Z
Answers.Text: I completely forgot about this question, managed to understand it a few days later exactly the way you explained, so it's great for future people to have this answer public and accepted!
  Thanks for the contribution
Answers.UserDisplayName: Dominus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-15T09:55:01.773Z
Answers.LastActivityDate: 2018-12-15T09:55:01.773Z
Answers.UserDisplayName: shaabhishek
Body: <p>I'm struggling with an inverse reinforcement learning problem which seems to appear quite often around the literature, yet I can't find any resources explaining it.</p>
  
  <p>The problem is that of calculating the gradient of a Boltzmann policy distribution over the reward weights <span class="math-container">$\theta$</span>:</p>
  
  <p><span class="math-container">$$\displaystyle\pi(s,a)=\frac{\exp(\beta\cdot Q(s,a|\theta))}{\sum_{a'}\exp(\beta\cdot Q(s,a'|\theta')}$$</span></p>
  
  <p>The <span class="math-container">$\theta$</span> are a linear parametrization of the reward function, such that</p>
  
  <p><span class="math-container">$$\displaystyle R = \theta^T\phi(s,a)$$</span></p>
  
  <p>where <span class="math-container">$\phi(s,a)$</span> are features of the state space. In the simplest of the case, one could take <span class="math-container">$\phi_i(s,a) = \delta(s,i)$</span>, that is, the feature space is just an indicator function of the state space.</p>
  
  <p>A lot of algorithms simply state to calculate the gradient, but that doesn't seem that trivial, and I'm not managing to infer from the <a href="https://github.com/jmacglashan/burlap/blob/master/src/main/java/burlap/behavior/singleagent/learnfromdemo/mlirl/support/BoltzmannPolicyGradient.java" rel="nofollow noreferrer">bits of code I found online</a>.</p>
  
  <p>Some of the papers using this kind of methods are <a href="http://www.icml-2011.org/papers/478_icmlpaper.pdf" rel="nofollow noreferrer">Apprenticeship Learning About Multiple Intentions</a> (2011), by Monica Babes-Vroman et al, and <a href="https://papers.nips.cc/paper/4479-map-inference-for-bayesian-inverse-reinforcement-learning" rel="nofollow noreferrer">MAP Inference for Bayesian Inverse Reinforcement Learning</a> (2011), by Jaedeug Choi et al.</p>
  
Tags: <reinforcement-learning><optimization><gradient-descent>
Title: How can we calculate the gradient of the Boltzmann policy over reward function?
UserDisplayName: Dominus
ThreadText: # How can we calculate the gradient of the Boltzmann policy over reward function?
  
  (Asked by: username_0 on 2018-07-14)
  
  I'm struggling with an inverse reinforcement learning problem which seems to appear quite often around the literature, yet I can't find any resources explaining it.
  
  The problem is that of calculating the gradient of a Boltzmann policy distribution over the reward weights $\theta$:
  
  $$\displaystyle\pi(s,a)=\frac{\exp(\beta\cdot Q(s,a|\theta))}{\sum\_{a'}\exp(\beta\cdot Q(s,a'|\theta')}$$
  
  The $\theta$ are a linear parametrization of the reward function, such that
  
  $$\displaystyle R = \theta^T\phi(s,a)$$
  
  where $\phi(s,a)$ are features of the state space. In the simplest of the case, one could take $\phi\_i(s,a) = \delta(s,i)$, that is, the feature space is just an indicator function of the state space.
  
  A lot of algorithms simply state to calculate the gradient, but that doesn't seem that trivial, and I'm not managing to infer from the [bits of code I found online](https://github.com/jmacglashan/burlap/blob/master/src/main/java/burlap/behavior/singleagent/learnfromdemo/mlirl/support/BoltzmannPolicyGradient.java).
  
  Some of the papers using this kind of methods are [Apprenticeship Learning About Multiple Intentions](http://www.icml-2011.org/papers/478_icmlpaper.pdf) (2011), by Monica Babes-Vroman et al, and [MAP Inference for Bayesian Inverse Reinforcement Learning](https://papers.nips.cc/paper/4479-map-inference-for-bayesian-inverse-reinforcement-learning) (2011), by Jaedeug Choi et al.
  
  **Tags:** reinforcement-learning, optimization, gradient-descent
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-15. Score: 2 | Accepted answer
  
  The main point here is that you can write $Q(s, a|\theta) = R = \theta^\top \phi(s, a)$. For more details on this, you can read up on Policy Gradients (Chapter 13) from the 2nd edition of the Sutton and Barto book (in fact the expression you're looking for is equation 13.9)
  
  For simplicity, I'm setting $\beta=1$, but you can always put it in once you get the idea. Therefore, the expression for $\pi(s,a,\theta)$ (I'm including $\theta$ here to make the dependence of $\pi$ explicit) is now:
  $$
  \pi(s,a,\theta) = \frac{exp(\theta^\top \phi(s, a))}{\sum\_{a'} exp(\theta^\top \phi(s, a'))} = \frac{A}{B}
  $$
  
  Also, it is standard to compute $\nabla\_\theta\log \pi(s,a,\theta)$ instead of $\nabla\_\theta \pi(s,a,\theta)$ (although you totally can), so I'll go ahead and do that. To differentiate $\log \pi(s,a,\theta)$ with respect to the parameters, $\theta$, let's do some calculus:
  $$
  \nabla\_\theta \log \pi(s,a,\theta) = \nabla\_\theta \log \frac{A}{B} = \nabla\_\theta \log A - \nabla\_\theta \log B \\
  = \nabla\_\theta \left[ \theta^\top \phi(s,a) \right] - \nabla\_\theta \left[ \log \sum\_{a'} exp(\theta^\top \phi(s, a')) \right] \\
  = \phi(s,a) - \frac{1}{\sum\_{a'} exp(\theta^\top \phi(s, a'))} \sum\_{b} exp(\theta^\top \phi(s, b)) \phi(s,b) \\
  = \phi(s,a) - \sum\_{b} \frac{exp(\theta^\top \phi(s, b))}{\sum\_{a'} exp(\theta^\top \phi(s, a'))} \phi(s,b) \\
  = \phi(s,a) - \sum\_{b} \pi(s,b,\theta) \phi(s,b)
  $$
  
  Therefore, you can write $\nabla\_\theta\log \pi(s,a,\theta) = \phi(s,a) - \sum\_{b} \pi(s,b,\theta) \phi(s,b)$
  
  > **Comments:**
  > **username_0**: I completely forgot about this question, managed to understand it a few days later exactly the way you explained, so it's great for future people to have this answer public and accepted! Thanks for the contribution  
  
  ---
  
  