Answers.Body: <blockquote>
  <p>I know I can manually set the initial weights to zero between the different sub-networks, which works fine initially - but during training the error-correction ends up connecting the various networks together.</p>
  </blockquote>
  <p>This is the approach I would take as well. To prevent them from connecting during training, you should freeze the weights of the zero-entries, for example, by masking out the frozen parameters during each forward pass: <a href="https://discuss.pytorch.org/t/how-to-freeze-part-of-the-weight-matrix-during-training/137389" rel="nofollow noreferrer">https://discuss.pytorch.org/t/how-to-freeze-part-of-the-weight-matrix-during-training/137389</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-03T13:52:50.417Z
Answers.LastActivityDate: 2023-08-03T13:52:50.417Z
Answers.UserDisplayName: Alexander Wan
Body: <p>This might sound like a slightly odd question, but is there a good technique to make a fully connected neural network act as if it's partially connected - by having certain edges not propagate a signal either forward or back?</p>
  <p>I have several small neural networks, each of which is structurally identical - they all have the same inputs, all have a single output, and the same number and size of hidden layers.</p>
  <p>I was hoping to take advantage of hardware acceleration by putting them in the same NN, so rather than having five networks with 5 input, 1 output and a single hidden layer of 20 neurones each, I'd have a single network with 5 inputs, 5 outputs, and a hidden layer with 100 neurones.</p>
  <p>But I need them to not cross-contaminate one-another - to operate as if they were distinct networks and not learn based on one-another.</p>
  <p>I know I can manually set the initial weights to zero between the different sub-networks, which works fine initially - but during training the error-correction ends up connecting the various networks together.</p>
  <p>It occurs to me that I <em>might</em> be able to have another matrix very similar to weights, but this time being a simple '1' (open) or '0' (closed) multiplier I could attach to the learning rate - 'closed' links would be multiplied by 0 during training, resulting in no change.</p>
  <p>But if anyone knows if there's a different technique that works, I'd love to know.</p>
  
Tags: <neural-networks>
Title: Fully Connected Neural Networks That Act Partially Connected
UserDisplayName: Steve Harding
ThreadText: # Fully Connected Neural Networks That Act Partially Connected
  
  (Asked by: username_0 on 2023-08-03)
  
  This might sound like a slightly odd question, but is there a good technique to make a fully connected neural network act as if it's partially connected - by having certain edges not propagate a signal either forward or back?
  
  I have several small neural networks, each of which is structurally identical - they all have the same inputs, all have a single output, and the same number and size of hidden layers.
  
  I was hoping to take advantage of hardware acceleration by putting them in the same NN, so rather than having five networks with 5 input, 1 output and a single hidden layer of 20 neurones each, I'd have a single network with 5 inputs, 5 outputs, and a hidden layer with 100 neurones.
  
  But I need them to not cross-contaminate one-another - to operate as if they were distinct networks and not learn based on one-another.
  
  I know I can manually set the initial weights to zero between the different sub-networks, which works fine initially - but during training the error-correction ends up connecting the various networks together.
  
  It occurs to me that I *might* be able to have another matrix very similar to weights, but this time being a simple '1' (open) or '0' (closed) multiplier I could attach to the learning rate - 'closed' links would be multiplied by 0 during training, resulting in no change.
  
  But if anyone knows if there's a different technique that works, I'd love to know.
  
  **Tags:** neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-03. Score: 1
  
  > I know I can manually set the initial weights to zero between the different sub-networks, which works fine initially - but during training the error-correction ends up connecting the various networks together.
  
  This is the approach I would take as well. To prevent them from connecting during training, you should freeze the weights of the zero-entries, for example, by masking out the frozen parameters during each forward pass: <https://discuss.pytorch.org/t/how-to-freeze-part-of-the-weight-matrix-during-training/137389>.
  
  
  ---
  
  