Answers.Body: <p><em>but produce the same result.</em> this is a fake news going around the ML community, they will have comparable performances, but the predictions won't be the same.</p>
  <p>And no, initialization does not aim to get good initial performances, but to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on)</p>
  <p>So, to answer to your question: _ the more complex a dataset is, the more difficult it will be to find adequate initial random weights_ no, it's pretty much impossible to find a random initial guess for neural network that works even for the simplest of the task, mainly due to stacked non linearities</p>
  <p>The only initialization that is based on good guesses is called pretraining, but it's far from being random</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T18:56:11.257Z
Answers.Text: Thanks for clarifying some points!  But I have a question: I know that the initial weights initialized randomly are to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on) as you said.  However, if this initialization is not good, would this hinder the training process?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T19:48:50.093Z
Answers.Text: @willTheJ yes, suppose to have normalized your feature in 0-1 (so they are all positive), and you initialize your weights with negative numbers, then $relu(Wx+b)$ will always be 0, thus no learning will happen at all. So yes, initialization is "an art", because it can most likely hinder your training (mainly the initial part), however now we have good guesses for most of the known activation functions
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T19:55:11.610Z
Answers.Text: Thanks for ReLu's example!. For example, the ReLu would produce output 0 if the value is negative, but what if for example it was the Sigmoid, would no learning occur either?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T23:12:54.920Z
Answers.Text: @willTheJ you can still have problems, for example initializing with big values your parameters $W,b$, doing $\sigma(Wx+b)$ will lead you to the tails of the sigmoid function, which has pretty much 0 gradient, so the learning will be very very slow
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T18:32:32.453Z
Answers.LastActivityDate: 2023-11-19T18:32:32.453Z
Answers.UserDisplayName: Alberto
Body: <p>I know that neural networks don't need to have exactly the same weights, and that different networks trained on the same problem may result in different final weights, but produce comparable results. And i know that the initial weights initialized randomly are used to help the optimization procedures.</p>
  <p>In multilayer perceptron neural networks, during training, using a dataset with a simpler pattern such as Iris, the network is more likely to find adequate initial weights using random weight initialization(perhaps because the pattern is simpler, and could there are more possibilities of initial weights that would be suitable?</p>
  <p>And for example, if I use a much more complex dataset, which has a much more challenging pattern for the model. During random initial weight initialization, would it be much more difficult to find suitable initial weights so that the network can perform well?</p>
  <p><strong>In other words, the more complex a dataset is, the more difficult it will be to find adequate initial random weights?</strong></p>
  <p><strong>Detail: the neural network I'm using, to start the random weights, generates pseudo-random numbers (between 0 and 1), using the same seed(42) to guarantee reproducibility.</strong></p>
  
Tags: <neural-networks>
Title: In multilayer perceptron neural networks, the more complex a dataset is, the more difficult it will be to find adequate initial random weights?
UserDisplayName: will The J
ThreadText: # In multilayer perceptron neural networks, the more complex a dataset is, the more difficult it will be to find adequate initial random weights?
  
  (Asked by: username_0 on 2023-11-19)
  
  I know that neural networks don't need to have exactly the same weights, and that different networks trained on the same problem may result in different final weights, but produce comparable results. And i know that the initial weights initialized randomly are used to help the optimization procedures.
  
  In multilayer perceptron neural networks, during training, using a dataset with a simpler pattern such as Iris, the network is more likely to find adequate initial weights using random weight initialization(perhaps because the pattern is simpler, and could there are more possibilities of initial weights that would be suitable?
  
  And for example, if I use a much more complex dataset, which has a much more challenging pattern for the model. During random initial weight initialization, would it be much more difficult to find suitable initial weights so that the network can perform well?
  
  **In other words, the more complex a dataset is, the more difficult it username_0 be to find adequate initial random weights?**
  
  **Detail: the neural network I'm using, to start the random weights, generates pseudo-random numbers (between 0 and 1), using the same seed(42) to guarantee reproducibility.**
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-19. Score: 2 | Accepted answer
  
  *but produce the same result.* this is a fake news going around the ML community, they username_0 have comparable performances, but the predictions won't be the same.
  
  And no, initialization does not aim to get good initial performances, but to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on)
  
  So, to answer to your question: \_ the more complex a dataset is, the more difficult it username_0 be to find adequate initial random weights\_ no, it's pretty much impossible to find a random initial guess for neural network that works even for the simplest of the task, mainly due to stacked non linearities
  
  The only initialization that is based on good guesses is called pretraining, but it's far from being random
  
  > **Comments:**
  > **username_0**: Thanks for clarifying some points!  But I have a question: I know that the initial weights initialized randomly are to help the optimization procedures (so avoiding dead gradients, smooth locally the loss landscape and so on) as you said.  However, if this initialization is not good, would this hinder the training process?  
  > **username_1**: @username_0 yes, suppose to have normalized your feature in 0-1 (so they are all positive), and you initialize your weights with negative numbers, then $relu(Wx+b)$ username_0 always be 0, thus no learning username_0 happen at all. So yes, initialization is "an art", because it can most likely hinder your training (mainly the initial part), however now we have good guesses for most of the known activation functions  
  > **username_0**: Thanks for ReLu's example!. For example, the ReLu would produce output 0 if the value is negative, but what if for example it was the Sigmoid, would no learning occur either?  
  > **username_1**: @username_0 you can still have problems, for example initializing with big values your parameters $W,b$, doing $\sigma(Wx+b)$ username_0 lead you to the tails of the sigmoid function, which has pretty much 0 gradient, so the learning username_0 be very very slow  
  
  ---
  
  