Answers.Body: <p>I think there is no simple way to transfer knowledge changes between different models.</p>
  <p>If you take your initial model and create a new version of it which you use to learn some other task (like &quot;Walk to a specific location&quot;), then the values copied from the first (original) model change in the second model.
  From that moment on, training the former model on another task will have different effects on its weights than continuing the training of the second model, whose parameter have been changed already.</p>
  <p>Consider, for example, that you had changed the mass of the robot and trained the initial model on that new task already. Then, if you took all the re-trained parameters from the first model and implanted them into the second model (trained on walking to a certain location), then you would essentially overwrite the additional knowledge the second model had gained already during its initial transfer-learning-process (not even taking into consideration any additional parameters appended to the list of parameters in the second model).</p>
  <p>So, you will have to re-train all three models (the original one and the two transfer-learning models) if you change the mass of the robot.</p>
  <p><strong>Edit:</strong></p>
  <p>There might be an option to apply the same knowledge changes to another model architecture if you refrain from pure transfer learning. This can be achieved with a more modularized model architecture.</p>
  <p>Consider that you train your first model on walking straight head. Let's call this model <span class="math-container">$m_{walk}$</span>.</p>
  <p>Then you intend to recycle <span class="math-container">$m_{walk}$</span> for another task, like walking straight ahead to a given location. Such a model architecture could be realized in two ways:</p>
  <ol>
  <li>You apply real transfer learning, retraining a copy of <span class="math-container">$m_{walk}$</span> to walk straight ahead until it reaches a certain location</li>
  <li>You take <span class="math-container">$m_{walk}$</span>, don't change it, but add a second model (let's call it <span class="math-container">$m_{navigator}$</span>), which is trained on predicting <span class="math-container">$go$</span> vs. <span class="math-container">$stop$</span>.</li>
  </ol>
  <p>In the second case, your overall model architecture (let's call it <span class="math-container">$m_{go\_to}$</span>) consists of both one model used for walking (i.e. <span class="math-container">$m_{walk}$</span>) and one model architecture which is used for predicting <span class="math-container">$go$</span> vs. <span class="math-container">$stop$</span> (i.e. <span class="math-container">$m_{navigator}$</span>).
  The idea then is that the robot executes the actions suggested by <span class="math-container">$m_{walk}$</span> until <span class="math-container">$m_{navigator}$</span> suggests stopping, upon which prediction the suggestions by <span class="math-container">$m_{walk}$</span> will be ignored.</p>
  <p>Then, whenever you retrain your model <span class="math-container">$m_{walk}$</span> (e.g. because the mass of some robot changes), you can simply apply the changes to <span class="math-container">$m_{go\_to}$</span> by replacing <span class="math-container">$m_{walk}$</span> by a new version, leaving the rest of <span class="math-container">$m_{go\_to}$</span> intact.</p>
  <p>If you generalize <span class="math-container">$m_{walk}$</span> to not only walk straight ahead, but also to take turns etc. and you generalize <span class="math-container">$m_{navigator}$</span> to predict going <span class="math-container">$left$</span>, <span class="math-container">$right$</span>, <span class="math-container">$straight\ ahead$</span>, <span class="math-container">$back$</span>, or <span class="math-container">$stop$</span> (being predicted when a certain destination has been reached), you can generalize <span class="math-container">$m_{go\_to}$</span> to walk where-ever you want it to go.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T23:14:46.163Z
Answers.LastActivityDate: 2020-07-10T09:49:10.957Z
Answers.LastEditDate: 2020-07-10T09:49:10.957Z
Answers.UserDisplayName: Daniel B.
Body: <p>I've trained a robot to walk in a straight line for as long as it can (using TD3), and now I'm using that pre-trained model for two new models with separate purposes: 1. Walk to a specific point and halt (adding target position to the NN inputs); 2. Walk straight at a specified velocity (adding a target velocity to NN inputs).</p>
  <p>Now let's say I retrain the original model again to walk properly after changing, say, the mass of the robot. How can I approach &quot;forwarding&quot; this update to the two transfer-learned models? The purpose of this is to minimize re-training time for all future models transfer-learned from the original.</p>
  <p>(What strikes me as particularly challenging is the fact that the input layer of the transfer-learned models have additional features, so this may re-wire the majority of the NN, making a &quot;forwarded update&quot; completely incompatible...)</p>
  
Tags: <reinforcement-learning><transfer-learning>
Title: How to "forward" updated NN model to a transferred model?
UserDisplayName: pinkie pAI
ThreadText: # How to "forward" updated NN model to a transferred model?
  
  (Asked by: username_0 on 2020-07-09)
  
  I've trained a robot to walk in a straight line for as long as it can (using TD3), and now I'm using that pre-trained model for two new models with separate purposes: 1. Walk to a specific point and halt (adding target position to the NN inputs); 2. Walk straight at a specified velocity (adding a target velocity to NN inputs).
  
  Now let's say I retrain the original model again to walk properly after changing, say, the mass of the robot. How can I approach "forwarding" this update to the two transfer-learned models? The purpose of this is to minimize re-training time for all future models transfer-learned from the original.
  
  (What strikes me as particularly challenging is the fact that the input layer of the transfer-learned models have additional features, so this may re-wire the majority of the NN, making a "forwarded update" completely incompatible...)
  
  **Tags:** reinforcement-learning, transfer-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-09. Score: 1 | Accepted answer
  
  I think there is no simple way to transfer knowledge changes between different models.
  
  If you take your initial model and create a new version of it which you use to learn some other task (like "Walk to a specific location"), then the values copied from the first (original) model change in the second model.
  From that moment on, training the former model on another task will have different effects on its weights than continuing the training of the second model, whose parameter have been changed already.
  
  Consider, for example, that you had changed the mass of the robot and trained the initial model on that new task already. Then, if you took all the re-trained parameters from the first model and implanted them into the second model (trained on walking to a certain location), then you would essentially overwrite the additional knowledge the second model had gained already during its initial transfer-learning-process (not even taking into consideration any additional parameters appended to the list of parameters in the second model).
  
  So, you will have to re-train all three models (the original one and the two transfer-learning models) if you change the mass of the robot.
  
  **Edit:**
  
  There might be an option to apply the same knowledge changes to another model architecture if you refrain from pure transfer learning. This can be achieved with a more modularized model architecture.
  
  Consider that you train your first model on walking straight head. Let's call this model $m\_{walk}$.
  
  Then you intend to recycle $m\_{walk}$ for another task, like walking straight ahead to a given location. Such a model architecture could be realized in two ways:
  
  1. You apply real transfer learning, retraining a copy of $m\_{walk}$ to walk straight ahead until it reaches a certain location
  2. You take $m\_{walk}$, don't change it, but add a second model (let's call it $m\_{navigator}$), which is trained on predicting $go$ vs. $stop$.
  
  In the second case, your overall model architecture (let's call it $m\_{go\\_to}$) consists of both one model used for walking (i.e. $m\_{walk}$) and one model architecture which is used for predicting $go$ vs. $stop$ (i.e. $m\_{navigator}$).
  The idea then is that the robot executes the actions suggested by $m\_{walk}$ until $m\_{navigator}$ suggests stopping, upon which prediction the suggestions by $m\_{walk}$ will be ignored.
  
  Then, whenever you retrain your model $m\_{walk}$ (e.g. because the mass of some robot changes), you can simply apply the changes to $m\_{go\\_to}$ by replacing $m\_{walk}$ by a new version, leaving the rest of $m\_{go\\_to}$ intact.
  
  If you generalize $m\_{walk}$ to not only walk straight ahead, but also to take turns etc. and you generalize $m\_{navigator}$ to predict going $left$, $right$, $straight\ ahead$, $back$, or $stop$ (being predicted when a certain destination has been reached), you can generalize $m\_{go\\_to}$ to walk where-ever you want it to go.
  
  
  ---
  
  