Answers.Body: <p>It's common in machine learning to do this <strong>log-trick</strong>, i.e. rather than optimizing <span class="math-container">$f(\mathbf{x})$</span>, you optimize <span class="math-container">$\log f(\mathbf{x})$</span>.</p>
  <p>There are 3 main reasons why we (can) do this.</p>
  <ol>
  <li><p>When your objective function is the product of multiple probabilities (or, more generally, small numbers), i.e. <span class="math-container">$f(\mathbf{x}) = \prod_{i=1}^N p(x_i)$</span>, then <span class="math-container">$\log f(\mathbf{x}) = \log \left( \prod_{i=1}^N p(x_i) \right) = \sum_{i=1}^N \log p(x_i)$</span> (see <a href="https://courses.lumenlearning.com/boundless-algebra/chapter/working-with-logarithms/" rel="nofollow noreferrer">this</a>), which is more numerical stable because we got rid of multiplication of possibly very small numbers, which can lead to <a href="https://gregorygundersen.com/blog/2020/02/09/log-sum-exp/" rel="nofollow noreferrer">underflow</a>.</p>
  </li>
  <li><p>For the same reason, we can also compute derivatives more easily, as we can simply compute the derivatives of each component (because derivatives are linear)</p>
  </li>
  <li><p>The logarithm is monotonically increasing, so <span class="math-container">$\log f(\mathbf{x})$</span> has the same optima as <span class="math-container">$f(\mathbf{x})$</span> (simple proof <a href="https://math.stackexchange.com/a/3053138/168764">here</a>).</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T23:45:24.520Z
Answers.LastActivityDate: 2021-12-16T11:02:02.810Z
Answers.LastEditDate: 2021-12-16T11:02:02.810Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Because Binary Cross-Entropy Loss is a commonly used loss function for binary classification problems in machine learning:</p>
  <p><span class="math-container">$$
  L = - \frac{1}{N} \sum_{i=1}^{N} \left[ y_i \cdot \log(\hat{y_i}) + (1 - y_i) \cdot \log(1 - \hat{y_i}) \right]
  $$</span></p>
  <ul>
  <li><span class="math-container">$N$</span> is the number of samples.</li>
  <li><span class="math-container">$y_i$</span> is the true label of the <span class="math-container">$i$</span>-th sample (0 or 1).</li>
  <li><span class="math-container">$\hat{y_i}$</span> is the predicted probability for the <span class="math-container">$i$</span>-th sample (i.e., the probability of predicting 1).</li>
  </ul>
  <p>If the true label <span class="math-container">$y_i=0$</span>, the loss is <span class="math-container">$\log(1 - \hat{y}_i)$</span>, and similarly, if <span class="math-container">$y_i=1$</span>, the loss is <span class="math-container">$\log(\hat{y}_i)$</span>.</p>
  <p>By dividing the samples based on their true labels, we arrive at the form used in GANs:</p>
  <p><span class="math-container">$$
  \mathbb{E}_{\boldsymbol{x} \sim p_{\text {data }}(\boldsymbol{x})}[\log D(\boldsymbol{x})]+\mathbb{E}_{\boldsymbol{z} \sim p_{\boldsymbol{z}}(\boldsymbol{z})}[\log (1-D(G(\boldsymbol{z})))]
  $$</span></p>
  <p>As for why Binary Cross-Entropy Loss is chosen for binary classification problems, that is a broader question.</p>
  <p>Additionally, it's important to note that GANs do not only train the discriminator but also the generator. The choice of such a joint optimization objective can be explained by the fact that the generator's optimization goal is to minimize the Jensen-Shannon (JS) divergence between the real data distribution <span class="math-container">$p_{\text{data}}$</span> and the generated data distribution <span class="math-container">$p_g$</span>.</p>
  <p>The original paper proves that when optimizing the discriminator <span class="math-container">$D$</span>, fixing <span class="math-container">$G$</span>, it becomes a maximization problem. The optimal solution for the discriminator is:</p>
  <p><span class="math-container">$$
  D^*_G(\boldsymbol{x}) = \frac{p_{\text{data}}(\boldsymbol{x})}{p_{\text{data}}(\boldsymbol{x}) + p_g(\boldsymbol{x})}
  $$</span></p>
  <p>Substituting <span class="math-container">$D^*_G$</span> into <span class="math-container">$V(D, G)$</span> gives:</p>
  <p><span class="math-container">$$
  V(D^*_G, G) = \mathbb{E}_{\boldsymbol{x} \sim p_{\text{data}}} \left[ \log \frac{p_{\text{data}}(\boldsymbol{x})}{p_{\text{data}}(\boldsymbol{x}) + p_g(\boldsymbol{x})} \right] + \mathbb{E}_{\boldsymbol{x} \sim p_g} \left[ \log \frac{p_g(\boldsymbol{x})}{p_{\text{data}}(\boldsymbol{x}) + p_g(\boldsymbol{x})} \right]
  $$</span></p>
  <p>This simplifies to:</p>
  <p><span class="math-container">$$
  V(D^*_G, G) = -\log 4 + 2 \cdot JS(p_{\text{data}} \| p_g)
  $$</span></p>
  <p>Therefore, the optimization objective <span class="math-container">$\min_G V(D^*_G, G)$</span> is equivalent to minimizing the JS divergence:</p>
  <p><span class="math-container">$$
  \min_G JS(p_{\text{data}} \| p_g)
  $$</span></p>
  <p>However, with the alternative optimization objective you mentioned, after a similar derivation, we find:</p>
  <p><span class="math-container">$$
  V'(D^*, G) = \mathbb{E}_{\boldsymbol{x} \sim p_{\text{data}}} \left[ \frac{p_{\text{data}}}{p_{\text{data}} + p_g} \right] + \mathbb{E}_{\boldsymbol{x} \sim p_g} \left[ \frac{p_g}{p_{\text{data}} + p_g} \right] = 1
  $$</span></p>
  <p>Unlike the original GAN, the alternative loss function does not directly express a divergence measure between <span class="math-container">$p_{\text{data}}$</span> and <span class="math-container">$p_g$</span>. The original GAN provides a symmetric and meaningful distribution alignment measure through JS divergence, while the alternative loss function lacks this theoretical foundation and cannot guarantee that <span class="math-container">$p_{\text{data}} = p_g$</span> at the optimum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-20T08:21:38.863Z
Answers.LastActivityDate: 2024-09-20T08:21:38.863Z
Answers.UserDisplayName: XiaoBanni
Body: <p>The minimax equation for generative adversarial networks</p>
  <p><span class="math-container">$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\boldsymbol{z}\sim p_{\boldsymbol{z}}(\boldsymbol{z})}[\log(1 - D(G(\boldsymbol{z}))]  $$</span></p>
  <p>Why do we use logarithms instead of just</p>
  <p><span class="math-container">$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[ D(\boldsymbol{x})] + \mathbb{E}_{\boldsymbol{z}\sim p_{\boldsymbol{z}}(\boldsymbol{z})}[(1 - D(G(\boldsymbol{z}))]  $$</span></p>
  
Tags: <objective-functions><generative-adversarial-networks><logarithm>
Title: Why are logarithms used in GANs minimax equation?
UserDisplayName: James Ellis
ThreadText: # Why are logarithms used in GANs minimax equation?
  
  (Asked by: username_0 on 2021-12-15)
  
  The minimax equation for generative adversarial networks
  
  $$\min\_G \max\_D V(D,G) = \mathbb{E}\_{\boldsymbol{x}\sim p\_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}\_{\boldsymbol{z}\sim p\_{\boldsymbol{z}}(\boldsymbol{z})}[\log(1 - D(G(\boldsymbol{z}))] $$
  
  Why do we use logarithms instead of just
  
  $$\min\_G \max\_D V(D,G) = \mathbb{E}\_{\boldsymbol{x}\sim p\_{data}(\boldsymbol{x})}[ D(\boldsymbol{x})] + \mathbb{E}\_{\boldsymbol{z}\sim p\_{\boldsymbol{z}}(\boldsymbol{z})}[(1 - D(G(\boldsymbol{z}))] $$
  
  **Tags:** objective-functions, generative-adversarial-networks, logarithm
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-12-15. Score: 2 | Accepted answer
  
  It's common in machine learning to do this **log-trick**, i.e. rather than optimizing $f(\mathbf{x})$, you optimize $\log f(\mathbf{x})$.
  
  There are 3 main reasons why we (can) do this.
  
  1. When your objective function is the product of multiple probabilities (or, more generally, small numbers), i.e. $f(\mathbf{x}) = \prod\_{i=1}^N p(x\_i)$, then $\log f(\mathbf{x}) = \log \left( \prod\_{i=1}^N p(x\_i) \right) = \sum\_{i=1}^N \log p(x\_i)$ (see [this](https://courses.lumenlearning.com/boundless-algebra/chapter/working-with-logarithms/)), which is more numerical stable because we got rid of multiplication of possibly very small numbers, which can lead to [underflow](https://gregorygundersen.com/blog/2020/02/09/log-sum-exp/).
  2. For the same reason, we can also compute derivatives more easily, as we can simply compute the derivatives of each component (because derivatives are linear)
  3. The logarithm is monotonically increasing, so $\log f(\mathbf{x})$ has the same optima as $f(\mathbf{x})$ (simple proof [here](https://math.stackexchange.com/a/3053138/168764)).
  
  
  ---
  
  ### Answer by username_2 on 2024-09-20. Score: 0
  
  Because Binary Cross-Entropy Loss is a commonly used loss function for binary classification problems in machine learning:
  
  $$
  L = - \frac{1}{N} \sum\_{i=1}^{N} \left[ y\_i \cdot \log(\hat{y\_i}) + (1 - y\_i) \cdot \log(1 - \hat{y\_i}) \right]
  $$
  
  * $N$ is the number of samples.
  * $y\_i$ is the true label of the $i$-th sample (0 or 1).
  * $\hat{y\_i}$ is the predicted probability for the $i$-th sample (i.e., the probability of predicting 1).
  
  If the true label $y\_i=0$, the loss is $\log(1 - \hat{y}\_i)$, and similarly, if $y\_i=1$, the loss is $\log(\hat{y}\_i)$.
  
  By dividing the samples based on their true labels, we arrive at the form used in GANs:
  
  $$
  \mathbb{E}\_{\boldsymbol{x} \sim p\_{\text {data }}(\boldsymbol{x})}[\log D(\boldsymbol{x})]+\mathbb{E}\_{\boldsymbol{z} \sim p\_{\boldsymbol{z}}(\boldsymbol{z})}[\log (1-D(G(\boldsymbol{z})))]
  $$
  
  As for why Binary Cross-Entropy Loss is chosen for binary classification problems, that is a broader question.
  
  Additionally, it's important to note that GANs do not only train the discriminator but also the generator. The choice of such a joint optimization objective can be explained by the fact that the generator's optimization goal is to minimize the Jensen-Shannon (JS) divergence between the real data distribution $p\_{\text{data}}$ and the generated data distribution $p\_g$.
  
  The original paper proves that when optimizing the discriminator $D$, fixing $G$, it becomes a maximization problem. The optimal solution for the discriminator is:
  
  $$
  D^\*\_G(\boldsymbol{x}) = \frac{p\_{\text{data}}(\boldsymbol{x})}{p\_{\text{data}}(\boldsymbol{x}) + p\_g(\boldsymbol{x})}
  $$
  
  Substituting $D^\*\_G$ into $V(D, G)$ gives:
  
  $$
  V(D^\*\_G, G) = \mathbb{E}\_{\boldsymbol{x} \sim p\_{\text{data}}} \left[ \log \frac{p\_{\text{data}}(\boldsymbol{x})}{p\_{\text{data}}(\boldsymbol{x}) + p\_g(\boldsymbol{x})} \right] + \mathbb{E}\_{\boldsymbol{x} \sim p\_g} \left[ \log \frac{p\_g(\boldsymbol{x})}{p\_{\text{data}}(\boldsymbol{x}) + p\_g(\boldsymbol{x})} \right]
  $$
  
  This simplifies to:
  
  $$
  V(D^\*\_G, G) = -\log 4 + 2 \cdot JS(p\_{\text{data}} \| p\_g)
  $$
  
  Therefore, the optimization objective $\min\_G V(D^\*\_G, G)$ is equivalent to minimizing the JS divergence:
  
  $$
  \min\_G JS(p\_{\text{data}} \| p\_g)
  $$
  
  However, with the alternative optimization objective you mentioned, after a similar derivation, we find:
  
  $$
  V'(D^\*, G) = \mathbb{E}\_{\boldsymbol{x} \sim p\_{\text{data}}} \left[ \frac{p\_{\text{data}}}{p\_{\text{data}} + p\_g} \right] + \mathbb{E}\_{\boldsymbol{x} \sim p\_g} \left[ \frac{p\_g}{p\_{\text{data}} + p\_g} \right] = 1
  $$
  
  Unlike the original GAN, the alternative loss function does not directly express a divergence measure between $p\_{\text{data}}$ and $p\_g$. The original GAN provides a symmetric and meaningful distribution alignment measure through JS divergence, while the alternative loss function lacks this theoretical foundation and cannot guarantee that $p\_{\text{data}} = p\_g$ at the optimum.
  
  
  ---
  
  