Answers.Body: <p>That's interesting. I'm guessing it has to do with the way PyTorch handles weight initialization. For linear layers, the bias range is set at the scale of <span class="math-container">$1/\sqrt{\text{in_features}}$</span> via the kaiming initialization function.</p>
  <p>In your case, your input features is zero, so you get an undetermined scale. So I am guessing somewhere in the PyTorch code, this gets converted to or interpreted as zero (which is of course not mathematically correct but a common practice).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-15T15:59:34.977Z
Answers.LastActivityDate: 2022-10-15T15:59:34.977Z
Answers.UserDisplayName: PeaBrane
Body: <p>I'm following <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear" rel="nofollow noreferrer">https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear</a></p>
  <p>Documentation allows for <code>bias=True</code> term, but they don't say where this parameter is actually stored. I wanted to clarify.</p>
  <p>You would expect <code>nn.Linear(1, 1, bias=True)</code> to have 1 free parameter (if you draw it out you have 1 input going to 1 output, so only 1 weight). But this isn't the case. An extra bias term is implicitly added to the model, to get 2 parameters. They probably should have said that, but this is how it works.</p>
  <p>With this framework in mind, we get to my question:</p>
  <p>Why does declaring <code>0</code> as the input size and <code>m</code> as the output size initialize all the bias components to <code>0</code>? It doesn't make sense why they would do this.</p>
  <p>For instance,</p>
  <pre><code>&gt;&gt;&gt; for p in nn.Linear(0, 4, bias=True).parameters(): print(p)
  
  Parameter containing:
  tensor([], size=(4, 0), requires_grad=True)
  Parameter containing:
  tensor([0., 0., 0., 0.], requires_grad=True)
  
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-14T07:46:44.787Z
Comments.Text: I don't understand why you expect only one parameter in the first case with `bias=True`. You have `activation = weight * input + bias`. One input, one output, everything is scalar. The layer has two parameters: the weight, and the bias.
Comments.UserDisplayName: maxy
Tags: <machine-learning><pytorch>
Title: Why does a zero-input network initialize all bias terms to 0? (pytorch)
UserDisplayName: Mondo Duke
ThreadText: # Why does a zero-input network initialize all bias terms to 0? (pytorch)
  
  (Asked by: username_0 on 2022-10-14)
  
  I'm following https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear
  Documentation allows for bias=True term, but they don't say where this parameter is actually stored. I wanted to clarify.
  You would expect nn.Linear(1, 1, bias=True) to have 1 free parameter (if you draw it out you have 1 input going to 1 output, so only 1 weight). But this isn't the case. An extra bias term is implicitly added to the model, to get 2 parameters. They probably should have said that, but this is how it works.
  With this framework in mind, we get to my question:
  Why does declaring 0 as the input size and m as the output size initialize all the bias components to 0? It doesn't make sense why they would do this.
  For instance,
  >>> for p in nn.Linear(0, 4, bias=True).parameters(): print(p)
  
  Parameter containing:
  tensor([], size=(4, 0), requires_grad=True)
  Parameter containing:
  tensor([0., 0., 0., 0.], requires_grad=True)
  
  **Tags:** machine-learning, pytorch
  > **Comments:**
  > **username_1**: I don't understand why you expect only one parameter in the first case with `bias=True`. You have `activation = weight * input + bias`. One input, one output, everything is scalar. The layer has two parameters: the weight, and the bias.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-15. Score: 2 | Accepted answer
  
  That's interesting. I'm guessing it has to do with the way PyTorch handles weight initialization. For linear layers, the bias range is set at the scale of $1/\sqrt{\text{in\_features}}$ via the kaiming initialization function.
  
  In your case, your input features is zero, so you get an undetermined scale. So I am guessing somewhere in the PyTorch code, this gets converted to or interpreted as zero (which is of course not mathematically correct but a common practice).
  
  
  ---
  
  