Answers.Body: <p>Unless you use full batch gradient descent method for your BP algo which is very rare in practice mainly due to its inefficiency and slow convergence, the usual <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method" rel="nofollow noreferrer">stochastic</a> or mini-batch or momentum-based gradient descent methods don't produce exactly same results.</p>
  <blockquote>
  <p>As the algorithm sweeps through the training set, it performs the above update for each training sample. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles. Typical implementations may use an adaptive learning rate so that the algorithm converges.</p>
  </blockquote>
  <p>Obviously in above case the <em>order</em> of all your training samples to be fitted in any epoch matters due to the data shuffle along with an adaptive learning rate and the nonzero threshold even in all the cases of actual convergence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T23:09:52.983Z
Answers.LastActivityDate: 2023-11-13T23:09:52.983Z
Answers.UserDisplayName: cinch
Body: <p><strong>Reference:</strong>
  <a href="https://home.agh.edu.pl/%7Evlsi/AI/backp_t_en/backprop.html" rel="nofollow noreferrer">https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html</a></p>
  <p>If I trained a multilayer percetron neural network manually, following exactly the backpropagation steps described in the article, with fixed initial weights (that is, without starting randomly), and then repeating this training again several more times just like I did the first time, ignoring the randomness and unforeseen events of the computation(because as I would be following everything manually, there would be no random initialization of the weights, no rounding problems, nor any variation in the process), if I do this when training the network, and I always use the same fixed initial weights( so that it always starts from the same starting point), using the same network structure (same number of layers, neurons, exactly the same network), with exactly the same parameters (such as number of epochs, learning rate, etc.), with the same dataset, with this network being trained with the dataset samples in the same order, <strong>following meticulously, very carefully, mathematically will I always get the same results? this is true ?</strong></p>
  <p><strong>I are without considering of whether the predictions are correct or not. My specific doubt, whether mathematically the results will be the same in the circumstances described.</strong></p>
  
Tags: <neural-networks><backpropagation>
Title: If I manually trained a multilayer percetron neural network, always following exactly the same steps meticulously, would I always get the same result?
UserDisplayName: will The J
ThreadText: # If I manually trained a multilayer percetron neural network, always following exactly the same steps meticulously, would I always get the same result?
  
  (Asked by: username_0 on 2023-11-13)
  
  **Reference:**
  [https://home.agh.edu.pl/~vlsi/AI/backp\_t\_en/backprop.html](https://home.agh.edu.pl/%7Evlsi/AI/backp_t_en/backprop.html)
  
  If I trained a multilayer percetron neural network manually, following exactly the backpropagation steps described in the article, with fixed initial weights (that is, without starting randomly), and then repeating this training again several more times just like I did the first time, ignoring the randomness and unforeseen events of the computation(because as I would be following everything manually, there would be no random initialization of the weights, no rounding problems, nor any variation in the process), if I do this when training the network, and I always use the same fixed initial weights( so that it always starts from the same starting point), using the same network structure (same number of layers, neurons, exactly the same network), with exactly the same parameters (such as number of epochs, learning rate, etc.), with the same dataset, with this network being trained with the dataset samples in the same order, **following meticulously, very carefully, mathematically will I always get the same results? this is true ?**
  
  **I are without considering of whether the predictions are correct or not. My specific doubt, whether mathematically the results will be the same in the circumstances described.**
  
  **Tags:** neural-networks, backpropagation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-13. Score: 1
  
  Unless you use full batch gradient descent method for your BP algo which is very rare in practice mainly due to its inefficiency and slow convergence, the usual [stochastic](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method) or mini-batch or momentum-based gradient descent methods don't produce exactly same results.
  
  > As the algorithm sweeps through the training set, it performs the above update for each training sample. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles. Typical implementations may use an adaptive learning rate so that the algorithm converges.
  
  Obviously in above case the *order* of all your training samples to be fitted in any epoch matters due to the data shuffle along with an adaptive learning rate and the nonzero threshold even in all the cases of actual convergence.
  
  
  ---
  
  