Answers.Body: <p>Regarding your first code snippet, there is no weight storing or continuation of training <em>between</em> the different CV folds whatsoever: each model is trained anew with the respective training data of each fold and validated on the validation data. Notice that this is exactly the idea behind cross validation - models trained on different folds are completely independent and they do not &quot;communicate&quot; in any manner. Should the models of the different sub-folds &quot;communicated&quot; and shared information between them, like, say, each model start training from the point the previous one finished, we would not talk about CV anymore (notice that such an approach could be valid if you purpose is something different than CV, but is is not a valid CV procedure).</p>
  <p>The situation is slightly different in your 2nd code snippet; after it finishes, you could go on, either by running more epochs on the same data, or even training with <em>different</em> data. All you would need is to simply add <code>model.fit</code> statements sequentially:</p>
  <pre><code># your 2nd snippet already run
  # you can either continue with, say, 50 more epochs:
  model.fit(X_train, y_train, epochs=50, batch_size=32, verbose=0)
  
  # or continue training the model with different data, say X_train2 &amp; y_train2:
  model.fit(X_train2, y_train2, epochs=150, batch_size=32, verbose=0)
  
  # of course you can do both, sequentially
  </code></pre>
  <p>What is the difference between your two code examples? In the first one, your model is <em>redefined</em> each time the loop executes; this defines a new model each time with the weights re-initialised, so no continuation of training. And, as already said, this is exactly the correct approach for cross validation.</p>
  <p>But in the 2nd case, you have a single model, which you can just keep training sequentially (either with the same or even with different data, as shown above).</p>
  <p>There are even cases where you would want to continue training the model outside of the current script; in such cases, you would save the model, and load it again later (or in another script altogether) in order to continue training it. This will work OK as long as you do not proceed to redefine/reinitalise the model. You can have a look at the documentation on <a href="https://www.tensorflow.org/guide/keras/save_and_serialize" rel="nofollow noreferrer">Save and load Keras models</a> for more.</p>
  <blockquote>
  <p>I am not able to find explanations about if these initializers take into account information from the previous trainings.</p>
  </blockquote>
  <p>They most certainly <strong>don't</strong>; if there is already existing information from previous runs (like in the cases I describe above), we do not even talk about weight initialization, i.e. there is no <em>need</em> for it - we already have the &quot;initial&quot; weights from the previous run. <em>Random</em> weight initialization in the sense discussed in the Stack Overflow thread you have linked above makes sense only for <em>new</em> models, like the ones produced in each run of the loop in your first code snippet above.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-22T15:41:08.127Z
Answers.Text: Thank you for the answer ! So I didn't understand CV. Thus its purpose is to have a better and more reliable estimate of the accuracy of the model and not also to train del model on a larger dataset? You just "enlarge" the dataset for the testing and not for training in practice ? While, instead, using the second strategy adding ```model.fit```, you enlarge also the dataset for training and the NN both trains on more examples and tests on more examples...
Answers.UserDisplayName: HelpNeederStudent
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-22T17:07:29.307Z
Answers.Text: @Manuela More or less, yes and yes. There are at least [two different reasons](https://stackoverflow.com/a/48087663/4685471) to use CV. The 1st is essentially what you describe (get an estimate of the final model performance); the 2nd is for hyperparameter tuning, when it is combined with a grid search. And as is, your 2nd code snippet has nothing to do with CV.
Answers.UserDisplayName: desertnaut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-22T13:15:01.020Z
Answers.LastActivityDate: 2022-01-22T14:06:26.513Z
Answers.LastEditDate: 2022-01-22T14:06:26.513Z
Answers.UserDisplayName: desertnaut
Body: <p>I am trying to understand how weights are initialized in a Neural Network using Keras deep learning framework and what happens if I train a Neural Network and then I want to train it again: are the weights of the previous training stored in some way and are the weights of the same Neural Network in the new training initialized based on the latest training ?</p>
  <p>By &quot;one training&quot; I mean all the updates that the weights have received after all the epochs that are set.</p>
  <p>To tell all the story: so far I have made my analysis assuming that with each new training (leaving the neural network architecture unchanged) the weights were initialized without taking into account the previous training. The concern come out when I wanted to use <em>K-fold cross-validation</em> procedure for which the dataset is split into K parts and K-1 parts are used for training while the remaining is left for testing. Iteratively this method uses all the K parts both for training and for testing. It is advised to do this in case of few data because in this way you can train the model K times each time with a different train/test split and you can produce significantly better estimates.</p>
  <p>This can be achieved with the following code (I am dealing with natural language processing and so the code is based on those kinds of tasks):</p>
  <pre><code>
  #Whatever is needed to prepare data to be fed to our Neural Network
  
  ....
  
  #Build keras model and fit
  
  classes=2 # 3 or 4
  
  embedding_dim = 10
  
  X_train, X_test, y_train, y_test = train_test_split(padded_sequences, Y, test_size=.2)
  
  # Define per-fold score containers 
  acc_per_fold = []
  loss_per_fold = []
  
  # Merge inputs and targets
  inputs = np.concatenate((X_train, X_test), axis=0)
  targets = np.concatenate((y_train, y_test), axis=0)
  
  num_folds=3
  
  # Define the K-fold Cross Validator
  kfold = KFold(n_splits=num_folds, shuffle=True)
  
  # K-fold Cross Validation model evaluation
  fold_no = 1
  for train, test in kfold.split(inputs, targets):
      model = Sequential()#1.Define a model
      model.add(Embedding(len(tokenizer.word_index)+1, embedding_dim, input_length=max_length, name='embeddings')) 
      model.add(LSTM(64))
      model.add(Dense(1,activation='sigmoid')) 
      #Dense(classes,activation='softmax') for 4 classes
      print(model.summary())
  
      model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  #sparse_categorical_crossentropy for 4 classes
      
      # Generate a print
      print('------------------------------------------------------------------------')
      print(f'Training for fold {fold_no} ...')
      
      model.fit(X_train, y_train,  batch_size=128,  validation_data=(X_test, y_test),verbose=2, epochs=15) 
  
      # Generate generalization metrics
      scores = model.evaluate(X_test,y_test,verbose=0)
      print(f'Score for fold {fold_no}: {model.metrics_names[0]} of {scores[0]}; {model.metrics_names[1]} of {scores[1]*100}%')
      acc_per_fold.append(scores[1] * 100)
      loss_per_fold.append(scores[0])
      
      # Increase fold number
      fold_no = fold_no + 1
  
  </code></pre>
  <p>After the first 15 epochs in which the Neural Network is trained on the first fold, the Neural Network is then trained on the second and again after 15 epochs it is trained on the third.</p>
  <p>So, as said before, we do this because <em>we can train the model K times each time with a different train/test split and you can produce significantly better estimates</em>. So this assumes that after the first 15 epochs for the training on the first set of data, the weights are just updated towards better estimates in the second run of 15 epochs for the training on the second set of data and so on ?</p>
  <p>In other words: if training K times is better than training once (in case of few data and under the limit of no-overfitting), does this mean that in some way the information of the previous training is stored and used for the second training and so on in order to be optimized more and more ?</p>
  <p>In case of my specific example, I am interested in the weights of the embedding layer to have dense vector representation of my sequence of symbols. So I want to understand this for all the weights of a Neural Network in general but particularly on the embedding layers weights (I think that it is the same story for all the weights of all layers of the NN).</p>
  <p>Anyway, besides the cross-validation procedure, if some kind of information is stored and kept from one training to the other, does it mean that each time I run my Neural Network training, this is influenced by the previous training ? For example, if I train this simple NN:</p>
  <pre><code># define model
  model = Sequential()
  model.add(Dense(10, activation='relu'))
  model.add(Dense(8, activation='relu'))
  model.add(Dense(1, activation='sigmoid'))
  # compile the model
  model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
  # fit the model
  model.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0)
  </code></pre>
  <p>First on a set of data and then on an other, in the second training with the new set of data, the weights are initialized on the basis of the previous training ?</p>
  <p>On one side, this seems to me to have no sense and not to be possible, but on the other if it is not so, I do not find the meaning of using cross-validation technique (at least in the way it is done using Keras in the code I attached).</p>
  <p>I tried to search what is the default initializer in Keras and I found this thread in Stack Overflow <a href="https://stackoverflow.com/questions/54011173/what-is-the-default-weight-initializer-in-keras">What is the default weight initializer in Keras?</a>. I can find all the ways to initialize them but I am not able to find explanations about if these initializers take into account information from the previous trainings.</p>
  <p>For example, as suggested in the answer to Stack Overflow question,  for most of the layers, such as Dense, convolution and RNN layers, the default kernel initializer is '<code>glorot_uniform</code>' and the documentation says that:</p>
  <blockquote>
  <p>Draws samples from a uniform distribution within [-limit, limit], where limit = sqrt(6 / (fan_in + fan_out)) (fan_in is the number of input units in the weight tensor and fan_out is the number of output units).</p>
  </blockquote>
  <p>But is this uniform distribution conditioned by the weights of the previous trainings ? Or by default this is not taken into account and there is some parameter to set to consider the previous  ones ?</p>
  <p>I think I miss something of important but I hope to have been clear and I thank you in advice.</p>
  <p>Note: I took the code and emphasized sentence about cross-validation from this blog post: <a href="https://www.machinecurve.com/index.php/2020/02/18/how-to-use-k-fold-cross-validation-with-keras/" rel="nofollow noreferrer">How to use K-fold Cross Validation with TensorFlow 2 and Keras?</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-22T13:50:54.090Z
Comments.Text: This question seems to be about how 1-2 libraries implement certain functionalities, i.e. k-fold cross-validation and weight initialization. Questions about libraries and how they implement things are off-topic here and they are better suited for Stack Overflow. We focus on the theoretical aspects of AI. Your question could be made on-topic if you don't focus on Keras or sklearn, but ask a general question about k-fold cross-validation and how it's usually done. That seems to be the crux of your post anyway. You can leave the code if you think it's helpful to answer the theoretical question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-22T13:57:35.763Z
Comments.Text: By the way, welcome to our site! If you have some time, it might be a good idea to read https://ai.stackexchange.com/help/on-topic.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-22T14:12:31.040Z
Comments.Text: @nbro I think that the question falls under "*Implementation questions in the context of understanding the theoretical topics are on-topic*", as described in our [on-topic](https://ai.stackexchange.com/help/on-topic) page. OP seems to have some *methodology* questions regarding the relation of CV and weight initialization, and they use Keras just to *demonstrate* them. FWIW, I can only testify that this question would be closed as off-topic in Stack Overflow (it is not about *programming*).
Comments.UserDisplayName: desertnaut
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-22T14:20:26.757Z
Comments.Text: *Maybe* this would be more appropriate for Data Science SE, but with so many partially overlapping SE sites, no wonder new users feel disoriented regarding which one is more appropriate for their question(s). Still, I don't think it is off-topic here (otherwise of course I would not have answered it).
Comments.UserDisplayName: desertnaut
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-22T14:42:52.850Z
Comments.Text: @desertnaut Yes, I agree that this question could fall into the context of "understanding the theoretical topics", unless the OP wants to understand the details of the software libraries, so I think we should leave it open (for now). As a rule of thumb, though, I usually say that, if you can ask your question without any code, then it's more like your question is on-topic here. I also agree that the overlap between our sites can be a mess. The sites should collaborate more in order for them to focus more on specific types of questions.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-22T15:42:18.620Z
Comments.Text: Thank you for the welcome @nbro. Since the Keras tag was available, I thought that I could do this question with code. And since my question is about specificities of neural networks (or better also CV in context of neural network ), I chose AI SE to ask it.
Comments.UserDisplayName: HelpNeederStudent
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-22T21:31:02.027Z
Comments.Text: @Manuela Ok. Just keep in mind that **programming** questions are **off-topic** here. We focus on the theoretical/conceptual problems.
Comments.UserDisplayName: nbro
Tags: <neural-networks><training><keras><cross-validation><weights-initialization>
Title: Weights initialization once the Neural Network is trained
UserDisplayName: HelpNeederStudent
ThreadText: # Weights initialization once the Neural Network is trained
  
  (Asked by: username_0 on 2022-01-22)
  
  I am trying to understand how weights are initialized in a Neural Network using Keras deep learning framework and what happens if I train a Neural Network and then I want to train it again: are the weights of the previous training stored in some way and are the weights of the same Neural Network in the new training initialized based on the latest training ?
  By "one training" I mean all the updates that the weights have received after all the epochs that are set.
  To tell all the story: so far I have made my analysis assuming that with each new training (leaving the neural network architecture unchanged) the weights were initialized without taking into account the previous training. The concern come out when I wanted to use K-fold cross-validation procedure for which the dataset is split into K parts and K-1 parts are used for training while the remaining is left for testing. Iteratively this method uses all the K parts both for training and for testing. It is advised to do this in case of few data because in this way you can train the model K times each time with a different train/test split and you can produce significantly better estimates.
  This can be achieved with the following code (I am dealing with natural language processing and so the code is based on those kinds of tasks):
  
  #Whatever is needed to prepare data to be fed to our Neural Network
  
  ....
  
  #Build keras model and fit
  
  classes=2 # 3 or 4
  
  embedding_dim = 10
  
  X_train, X_test, y_train, y_test = train_test_split(padded_sequences, Y, test_size=.2)
  
  # Define per-fold score containers 
  acc_per_fold = []
  loss_per_fold = []
  
  # Merge inputs and targets
  inputs = np.concatenate((X_train, X_test), axis=0)
  targets = np.concatenate((y_train, y_test), axis=0)
  
  num_folds=3
  
  # Define the K-fold Cross Validator
  kfold = KFold(n_splits=num_folds, shuffle=True)
  
  # K-fold Cross Validation model evaluation
  fold_no = 1
  for train, test in kfold.split(inputs, targets):
      model = Sequential()#1.Define a model
      model.add(Embedding(len(tokenizer.word_index)+1, embedding_dim, input_length=max_length, name='embeddings')) 
      model.add(LSTM(64))
      model.add(Dense(1,activation='sigmoid')) 
      #Dense(classes,activation='softmax') for 4 classes
      print(model.summary())
  
      model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  #sparse_categorical_crossentropy for 4 classes
      
      # Generate a print
      print('------------------------------------------------------------------------')
      print(f'Training for fold {fold_no} ...')
      
      model.fit(X_train, y_train,  batch_size=128,  validation_data=(X_test, y_test),verbose=2, epochs=15) 
  
      # Generate generalization metrics
      scores = model.evaluate(X_test,y_test,verbose=0)
      print(f'Score for fold {fold_no}: {model.metrics_names[0]} of {scores[0]}; {model.metrics_names[1]} of {scores[1]*100}%')
      acc_per_fold.append(scores[1] * 100)
      loss_per_fold.append(scores[0])
      
      # Increase fold number
      fold_no = fold_no + 1
  
  
  After the first 15 epochs in which the Neural Network is trained on the first fold, the Neural Network is then trained on the second and again after 15 epochs it is trained on the third.
  So, as said before, we do this because we can train the model K times each time with a different train/test split and you can produce significantly better estimates. So this assumes that after the first 15 epochs for the training on the first set of data, the weights are just updated towards better estimates in the second run of 15 epochs for the training on the second set of data and so on ?
  In other words: if training K times is better than training once (in case of few data and under the limit of no-overfitting), does this mean that in some way the information of the previous training is stored and used for the second training and so on in order to be optimized more and more ?
  In case of my specific example, I am interested in the weights of the embedding layer to have dense vector representation of my sequence of symbols. So I want to understand this for all the weights of a Neural Network in general but particularly on the embedding layers weights (I think that it is the same story for all the weights of all layers of the NN).
  Anyway, besides the cross-validation procedure, if some kind of information is stored and kept from one training to the other, does it mean that each time I run my Neural Network training, this is influenced by the previous training ? For example, if I train this simple NN:
  # define model
  model = Sequential()
  model.add(Dense(10, activation='relu'))
  model.add(Dense(8, activation='relu'))
  model.add(Dense(1, activation='sigmoid'))
  # compile the model
  model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
  # fit the model
  model.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0)
  
  First on a set of data and then on an other, in the second training with the new set of data, the weights are initialized on the basis of the previous training ?
  On one side, this seems to me to have no sense and not to be possible, but on the other if it is not so, I do not find the meaning of using cross-validation technique (at least in the way it is done using Keras in the code I attached).
  I tried to search what is the default initializer in Keras and I found this thread in Stack Overflow What is the default weight initializer in Keras?. I can find all the ways to initialize them but I am not able to find explanations about if these initializers take into account information from the previous trainings.
  For example, as suggested in the answer to Stack Overflow question,  for most of the layers, such as Dense, convolution and RNN layers, the default kernel initializer is 'glorot_uniform' and the documentation says that:
  
  Draws samples from a uniform distribution within [-limit, limit], where limit = sqrt(6 / (fan_in + fan_out)) (fan_in is the number of input units in the weight tensor and fan_out is the number of output units).
  
  But is this uniform distribution conditioned by the weights of the previous trainings ? Or by default this is not taken into account and there is some parameter to set to consider the previous  ones ?
  I think I miss something of important but I hope to have been clear and I thank you in advice.
  Note: I took the code and emphasized sentence about cross-validation from this blog post: How to use K-fold Cross Validation with TensorFlow 2 and Keras?
  
  **Tags:** neural-networks, training, keras, cross-validation, weights-initialization
  > **Comments:**
  > **username_1**: This question seems to be about how 1-2 libraries implement certain functionalities, i.e. k-fold cross-validation and weight initialization. Questions about libraries and how they implement things are off-topic here and they are better suited for Stack Overflow. We focus on the theoretical aspects of AI. Your question could be made on-topic if you don't focus on Keras or sklearn, but ask a general question about k-fold cross-validation and how it's usually done. That seems to be the crux of your post anyway. You can leave the code if you think it's helpful to answer the theoretical question.  
  > **username_1**: By the way, welcome to our site! If you have some time, it might be a good idea to read https://ai.stackexchange.com/help/on-topic.  
  > **username_2**: @nbro I think that the question falls under "*Implementation questions in the context of understanding the theoretical topics are on-topic*", as described in our [on-topic](https://ai.stackexchange.com/help/on-topic) page. OP seems to have some *methodology* questions regarding the relation of CV and weight initialization, and they use Keras just to *demonstrate* them. FWIW, I can only testify that this question would be closed as off-topic in Stack Overflow (it is not about *programming*).  
  > **username_2**: *Maybe* this would be more appropriate for Data Science SE, but with so many partially overlapping SE sites, no wonder new users feel disoriented regarding which one is more appropriate for their question(s). Still, I don't think it is off-topic here (otherwise of course I would not have answered it).  
  > **username_1**: @desertnaut Yes, I agree that this question could fall into the context of "understanding the theoretical topics", unless the OP wants to understand the details of the software libraries, so I think we should leave it open (for now). As a rule of thumb, though, I usually say that, if you can ask your question without any code, then it's more like your question is on-topic here. I also agree that the overlap between our sites can be a mess. The sites should collaborate more in order for them to focus more on specific types of questions.  
  > **username_0**: Thank you for the welcome @nbro. Since the Keras tag was available, I thought that I could do this question with code. And since my question is about specificities of neural networks (or better also CV in context of neural network ), I chose AI SE to ask it.  
  > **username_1**: @Manuela Ok. Just keep in mind that **programming** questions are **off-topic** here. We focus on the theoretical/conceptual problems.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-22. Score: 3 | Accepted answer
  
  Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of each fold and validated on the validation data. Notice that this is exactly the idea behind cross validation - models trained on different folds are completely independent and they do not "communicate" in any manner. Should the models of the different sub-folds "communicated" and shared information between them, like, say, each model start training from the point the previous one finished, we would not talk about CV anymore (notice that such an approach could be valid if you purpose is something different than CV, but is is not a valid CV procedure).
  The situation is slightly different in your 2nd code snippet; after it finishes, you could go on, either by running more epochs on the same data, or even training with different data. All you would need is to simply add model.fit statements sequentially:
  # your 2nd snippet already run
  # you can either continue with, say, 50 more epochs:
  model.fit(X_train, y_train, epochs=50, batch_size=32, verbose=0)
  
  # or continue training the model with different data, say X_train2 & y_train2:
  model.fit(X_train2, y_train2, epochs=150, batch_size=32, verbose=0)
  
  # of course you can do both, sequentially
  
  What is the difference between your two code examples? In the first one, your model is redefined each time the loop executes; this defines a new model each time with the weights re-initialised, so no continuation of training. And, as already said, this is exactly the correct approach for cross validation.
  But in the 2nd case, you have a single model, which you can just keep training sequentially (either with the same or even with different data, as shown above).
  There are even cases where you would want to continue training the model outside of the current script; in such cases, you would save the model, and load it again later (or in another script altogether) in order to continue training it. This will work OK as long as you do not proceed to redefine/reinitalise the model. You can have a look at the documentation on Save and load Keras models for more.
  
  I am not able to find explanations about if these initializers take into account information from the previous trainings.
  
  They most certainly don't; if there is already existing information from previous runs (like in the cases I describe above), we do not even talk about weight initialization, i.e. there is no need for it - we already have the "initial" weights from the previous run. Random weight initialization in the sense discussed in the Stack Overflow thread you have linked above makes sense only for new models, like the ones produced in each run of the loop in your first code snippet above.
  
  > **Comments:**
  > **username_0**: Thank you for the answer ! So I didn't understand CV. Thus its purpose is to have a better and more reliable estimate of the accuracy of the model and not also to train del model on a larger dataset? You just "enlarge" the dataset for the testing and not for training in practice ? While, instead, using the second strategy adding ```model.fit```, you enlarge also the dataset for training and the NN both trains on more examples and tests on more examples...  
  > **username_2**: @Manuela More or less, yes and yes. There are at least [two different reasons](https://stackoverflow.com/a/48087663/4685471) to use CV. The 1st is essentially what you describe (get an estimate of the final model performance); the 2nd is for hyperparameter tuning, when it is combined with a grid search. And as is, your 2nd code snippet has nothing to do with CV.  
  
  ---
  
  