Answers.Body: <p>This is <a href="https://en.wikipedia.org/wiki/Jensen%27s_inequality" rel="nofollow noreferrer">Jensen's inequality</a> at work.</p>
  <p>First of all, note that the first line can be rewritten as an expectation</p>
  <p><span class="math-container">$$\sum_{s} p(s \mid \mathbf{a}) \log p(\mathbf{y} \mid s, \mathbf{a}) = \mathbb{E}_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})]$$</span></p>
  <p>Then Jensen's inequality gives (Note that a log function is a <strong>concave</strong> function so gives the opposite inequality to what is normally given when explaining Jensen's inequality with respect to convex functions):</p>
  <p><span class="math-container">$$\mathbb{E}_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})] \leq \log\mathbb{E}_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] $$</span></p>
  <p>and then finally you can rewrite the Expectation as a summation.</p>
  <p><span class="math-container">$$\log\mathbb{E}_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] = \log \sum_{s} p(s \mid \mathbf{a}) p(\mathbf{y} \mid s, \mathbf{a})$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-26T13:11:11.887Z
Answers.LastActivityDate: 2021-04-30T09:38:49.557Z
Answers.LastEditDate: 2021-04-30T09:38:49.557Z
Answers.UserDisplayName: James
Body: <p>How is the jump from line 1 to line 2 done in equation 10 of <a href="https://arxiv.org/pdf/1502.03044.pdf" rel="nofollow noreferrer">Show, Attend and Tell</a>?</p>
  <p><a href="https://i.sstatic.net/j4kt9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/j4kt9.png" alt="enter image description here" /></a></p>
  <p>While we're at it, another thing that might be muddying the waters for me is that I'm not clear on what the sum is over. I know that <span class="math-container">$s$</span> is indexed as <span class="math-container">$s_{t,i}$</span>, but the one-hot trick from line 2 to 3 makes me believe that the sum is over just <span class="math-container">$i$</span>.</p>
  
Tags: <papers><math><attention><evidence-lower-bound>
Title: How is the variational lower bound for hard attention derived in Show, Attend and Tell
UserDisplayName: Alexander Soare
ThreadText: # How is the variational lower bound for hard attention derived in Show, Attend and Tell
  
  (Asked by: username_0 on 2021-04-26)
  
  How is the jump from line 1 to line 2 done in equation 10 of [Show, Attend and Tell](https://arxiv.org/pdf/1502.03044.pdf)?
  
  [![enter image description here](https://i.sstatic.net/j4kt9.png)](https://i.sstatic.net/j4kt9.png)
  
  While we're at it, another thing that might be muddying the waters for me is that I'm not clear on what the sum is over. I know that $s$ is indexed as $s\_{t,i}$, but the one-hot trick from line 2 to 3 makes me believe that the sum is over just $i$.
  
  **Tags:** papers, math, attention, evidence-lower-bound
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-26. Score: 2 | Accepted answer
  
  This is [Jensen's inequality](https://en.wikipedia.org/wiki/Jensen%27s_inequality) at work.
  
  First of all, note that the first line can be rewritten as an expectation
  
  $$\sum\_{s} p(s \mid \mathbf{a}) \log p(\mathbf{y} \mid s, \mathbf{a}) = \mathbb{E}\_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})]$$
  
  Then Jensen's inequality gives (Note that a log function is a **concave** function so gives the opposite inequality to what is normally given when explaining Jensen's inequality with respect to convex functions):
  
  $$\mathbb{E}\_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})] \leq \log\mathbb{E}\_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] $$
  
  and then finally you can rewrite the Expectation as a summation.
  
  $$\log\mathbb{E}\_{p(s|a)}[ p(\mathbf{y} \mid s, \mathbf{a})] = \log \sum\_{s} p(s \mid \mathbf{a}) p(\mathbf{y} \mid s, \mathbf{a})$$
  
  
  ---
  
  