Body: <p>I found a rather odd piece of code in a 3.8k star repo of the well known StyleGAN 2 paper.</p>
  <p>In the loss function they use the <a href="https://github.com/NVlabs/stylegan2-ada-pytorch/blob/d72cc7d041b42ec8e806021a205ed9349f87c6a4/training/loss.py#L130C1-L132C1" rel="nofollow noreferrer">following expression</a>:</p>
  <pre><code>with torch.autograd.profiler.record_function(name + '_backward'):
      (real_logits * 0 + loss_Dreal + loss_Dr1).mean().mul(gain).backward()
  </code></pre>
  <p>(They use something similar in <a href="https://github.com/NVlabs/stylegan2-ada-pytorch/blob/d72cc7d041b42ec8e806021a205ed9349f87c6a4/training/loss.py#L92" rel="nofollow noreferrer">line 92</a>)</p>
  <p>Multiplying <code>real_logits</code> with <code>0</code> seems to me like unnecessary computation overhead. However, the same code is adapted by other researches like the authors of &quot;MAT: Mask-Aware Transformer for Large Hole Image Inpainting&quot;. See <a href="https://github.com/fenglinglwb/MAT/blob/d273d891ecdad2e1df106516423a75bc45b2d800/losses/loss.py#L168" rel="nofollow noreferrer">here</a>:</p>
  <pre><code>with torch.autograd.profiler.record_function(name + '_backward'):
      ((real_logits + real_logits_stg1) * 0 + loss_Dreal + loss_Dreal_stg1 + loss_Dr1 + loss_Dr1_stg1).mean().mul(gain).backward()
  </code></pre>
  <p>The only hypothesis that I have is that they want a quick way of initializing a matrix of a specific shape filled with zeros. However, this makes no sense as everything ends up in the <code>mean()</code> anyways.</p>
  <p>What other purpose does this fulfill?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-15T13:23:28.650Z
Comments.Text: The `.backwards()` is probably not callable, if you created the tensor from zeros. multiplying with zero maybe keeps that intact?
Comments.UserDisplayName: N. Kiefer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-15T14:40:10.663Z
Comments.Text: the scalar output is the very same when omitting the zero term. But there must be a clever side effect during back propagation :-/
Comments.UserDisplayName: Klops
Tags: <generative-adversarial-networks><loss>
Title: StyleGAN 2 multiplies loss components with zero, why?
UserDisplayName: Klops
ThreadText: # StyleGAN 2 multiplies loss components with zero, why?
  
  (Asked by: username_0 on 2024-02-15)
  
  I found a rather odd piece of code in a 3.8k star repo of the well known StyleGAN 2 paper.
  In the loss function they use the following expression:
  with torch.autograd.profiler.record_function(name + '_backward'):
      (real_logits * 0 + loss_Dreal + loss_Dr1).mean().mul(gain).backward()
  
  (They use something similar in line 92)
  Multiplying real_logits with 0 seems to me like unnecessary computation overhead. However, the same code is adapted by other researches like the authors of "MAT: Mask-Aware Transformer for Large Hole Image Inpainting". See here:
  with torch.autograd.profiler.record_function(name + '_backward'):
      ((real_logits + real_logits_stg1) * 0 + loss_Dreal + loss_Dreal_stg1 + loss_Dr1 + loss_Dr1_stg1).mean().mul(gain).backward()
  
  The only hypothesis that I have is that they want a quick way of initializing a matrix of a specific shape filled with zeros. However, this makes no sense as everything ends up in the mean() anyways.
  What other purpose does this fulfill?
  
  **Tags:** generative-adversarial-networks, loss
  > **Comments:**
  > **username_1**: The `.backwards()` is probably not callable, if you created the tensor from zeros. multiplying with zero maybe keeps that intact?  
  > **username_0**: the scalar output is the very same when omitting the zero term. But there must be a clever side effect during back propagation :-/  
  
  ---
  
  ## 0 answers
  
  