Body: <p>I am beginner in deep learning and currently training a few neural networks (Pytorch) for problems in audio and speech. For my tasks, simple feed-forward networks are working well enough. I use basic layers like Linear, ReLU and Softmax with nll loss. I have tested a few initialization schemes provided by Pytorch and noticed that initialization has significant (but not high) effect in the speed of training and the final accuracy of the model. I am currently using <code>torch.nn.init.kaiming_uniform_</code> for initialization.</p>
  <p>In my understanding, all these are data independent initialization schemes. I would like to try something that is data dependent. I saw a few pre-training strategies with unsupervised learning followed by supervised learning, but they seem overly complicated.</p>
  <p>I am looking for something simple where I can use (preferably a fraction of the) training data to 'tweak' weights to better positions before the training. Are there any such strategies?</p>
  <p><strong>Addendum-1</strong>:
  Current initialization schemes (AFAIK) are mostly random values with control over range or energy to prevent values from dying down or blowing up. My aim is to further improve the starting point of training by taking account training data (or at least some of it). I am thinking of something like this. We pass a few batches of training data through the initial network and collect statistics on neuron outputs. Based on this, we identify the misbehaving neurons and tweak the weights and biases to reduce such issues so as to improve the training speed or accuracy. Is there anything of that kind?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T17:50:47.343Z
Comments.Text: Hello. Can you clarify why you want to do this? Wouldn't this basically then bias the training process based on the data? Maybe you could use a separate dataset (from the training dataset) for initialising the weights: in this case, it would probably be less biased, but I still don't understand why you want to do this.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T22:14:21.063Z
Comments.Text: @nbro, kindly see Addendum-1.
Comments.UserDisplayName: Axeon Thra
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T22:26:21.440Z
Comments.Text: Based on your addendum, there doesn't seem to be a big difference between what you seem to want and just training the neural network.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-06T01:56:22.750Z
Comments.Text: What you describe seems just like training (where tweaking problematic weights correspond to taking gradient descent steps) as @nbro has pointed out. Regardless, there's something called meta-learning, and it seems related to me. In meta-learning, you're trying to initialize weights in a way such that, the model can be adapted to different downstream tasks quickly.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-06T09:52:33.143Z
Comments.Text: @nbro, The key difference is that we are not using back-propagation to minimize a 'differentiable' cost function. Instead, we can tweak the weights using a bunch of heuristics that may or may not be differentiable.
Comments.UserDisplayName: Axeon Thra
Tags: <deep-learning><weights-initialization>
Title: What are strategies for data driven weights initialization?
UserDisplayName: Axeon Thra
ThreadText: # What are strategies for data driven weights initialization?
  
  (Asked by: username_0 on 2021-09-04)
  
  I am beginner in deep learning and currently training a few neural networks (Pytorch) for problems in audio and speech. For my tasks, simple feed-forward networks are working well enough. I use basic layers like Linear, ReLU and Softmax with nll loss. I have tested a few initialization schemes provided by Pytorch and noticed that initialization has significant (but not high) effect in the speed of training and the final accuracy of the model. I am currently using `torch.nn.init.kaiming_uniform_` for initialization.
  
  In my understanding, all these are data independent initialization schemes. I would like to try something that is data dependent. I saw a few pre-training strategies with unsupervised learning followed by supervised learning, but they seem overly complicated.
  
  I am looking for something simple where I can use (preferably a fraction of the) training data to 'tweak' weights to better positions before the training. Are there any such strategies?
  
  **Addendum-1**:
  Current initialization schemes (AFAIK) are mostly random values with control over range or energy to prevent values from dying down or blowing up. My aim is to further improve the starting point of training by taking account training data (or at least some of it). I am thinking of something like this. We pass a few batches of training data through the initial network and collect statistics on neuron outputs. Based on this, we identify the misbehaving neurons and tweak the weights and biases to reduce such issues so as to improve the training speed or accuracy. Is there anything of that kind?
  
  **Tags:** deep-learning, weights-initialization
  > **Comments:**
  > **username_1**: Hello. Can you clarify why you want to do this? Wouldn't this basically then bias the training process based on the data? Maybe you could use a separate dataset (from the training dataset) for initialising the weights: in this case, it would probably be less biased, but I still don't understand why you want to do this.  
  > **username_0**: @nbro, kindly see Addendum-1.  
  > **username_1**: Based on your addendum, there doesn't seem to be a big difference between what you seem to want and just training the neural network.  
  > **username_2**: What you describe seems just like training (where tweaking problematic weights correspond to taking gradient descent steps) as @nbro has pointed out. Regardless, there's something called meta-learning, and it seems related to me. In meta-learning, you're trying to initialize weights in a way such that, the model can be adapted to different downstream tasks quickly.  
  > **username_0**: @nbro, The key difference is that we are not using back-propagation to minimize a 'differentiable' cost function. Instead, we can tweak the weights using a bunch of heuristics that may or may not be differentiable.  
  
  ---
  
  ## 0 answers
  
  