Body: <p>Does anyone have any pointers to resources about the properties of randomly initialized neural networks (with no training)? I'm guessing this might depend on the network architecture and initialization scheme, but I'm most interested in properties that seem to be mostly true across architectures and initializations.</p>
  <p>In particular, I'm wondering what the distributions over outputs look like. Intuitively, one would expect most output distributions to be random, but I'm sure there are also &quot;adversarial examples&quot; that could be crafted which leads the NN to output highly-confident predictions.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-19T20:17:08.237Z
Comments.Text: In your questions, you specify untrained networks. So, the distribution will remain random. But, one work which sounds like this one is - Feedback alignment (https://arxiv.org/abs/1609.01596).
Comments.UserDisplayName: Abhishek Verma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-27T18:16:21.233Z
Comments.Text: Thanks for the pointer! However, this doesn't quite answer my question (_what kind of_ randomness): say you have all weights initialized as random normals(0,1), and RELUs, and your data distribution is uniform over a box. What will be the distribution of outputs f(x) where x ~ data-distribution and f(•) is the neural network? Wondering if this kind of analysis (formal or empirical) has been done anywhere
Comments.UserDisplayName: mdc
Tags: <neural-networks><weights-initialization>
Title: Distributions over outputs for randomly initialized neural networks
UserDisplayName: mdc
ThreadText: # Distributions over outputs for randomly initialized neural networks
  
  (Asked by: username_0 on 2022-09-18)
  
  Does anyone have any pointers to resources about the properties of randomly initialized neural networks (with no training)? I'm guessing this might depend on the network architecture and initialization scheme, but I'm most interested in properties that seem to be mostly true across architectures and initializations.
  
  In particular, I'm wondering what the distributions over outputs look like. Intuitively, one would expect most output distributions to be random, but I'm sure there are also "adversarial examples" that could be crafted which leads the NN to output highly-confident predictions.
  
  **Tags:** neural-networks, weights-initialization
  
  > **Comments:**
  > **username_1**: In your questions, you specify untrained networks. So, the distribution will remain random. But, one work which sounds like this one is - Feedback alignment (https://arxiv.org/abs/1609.01596).  
  > **username_0**: Thanks for the pointer! However, this doesn't quite answer my question (_what kind of_ randomness): say you have all weights initialized as random normals(0,1), and RELUs, and your data distribution is uniform over a box. What will be the distribution of outputs f(x) where x ~ data-distribution and f(•) is the neural network? Wondering if this kind of analysis (formal or empirical) has been done anywhere  
  
  ---
  
  ## 0 answers
  
  