Answers.Body: <p>Transfer learning is a field where you apply knowledge from a source onto a target. This is a vague notion and there is an abundance of literature pertaining to it. Given your question I will work under the assumption that you are referring weight/architecture sharing between model (in other words training a model on one dataset and using it as a featurizer for another dataset)  </p>
  
  <p>Now any learning system without lossless memory will have remnants of catastrophic forgetting. So let's think about how we would implement this transfer and what effects can be derived from this.</p>
  
  <ol>
  <li>One implementation involves transferring a component and only training additional layers.</li>
  <li>Another is retraining the entire system but at a lower learning rate?  </li>
  </ol>
  
  <p>In setting 1, we can make the claim catastrophic forgetting is minimized by the fact that there is an unbiased featurizer that cant <em>forget</em> based on a sampling regime, though this does not mean additional layers which are still being trained can still faulter in this error mode.  </p>
  
  <p>In setting 2, we can make the claim catasrophic forgetting can be reduced compared to a normal end-to-end no-transfer training because the unbiased featurizers difference can be analytically bounded by its initial transferred featurization (complexity class is based on both the function and the number of steps -- so longer you train, the more likely it can <em>forget</em>)  </p>
  
  <p>These reasons are talking about mitigating and not erasing the concept of catastrophic forgetting, that is because as I mentioned above <em>any learning system without lossless memory will have remnants of catastrophic forgetting</em>, so making the generalized claim about transfer learning may not always fit the bill.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-24T15:18:47.850Z
Answers.LastActivityDate: 2019-08-24T15:18:47.850Z
Answers.UserDisplayName: mshlis
Body: <p>How can transfer learning be used to mitigate <a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="nofollow noreferrer">catastrophic forgetting</a>. Could someone elaborate on this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-24T11:07:51.803Z
Comments.Text: I think this claim is false 
  
  Specifically I do not see how it could make sense to define transfer learning, which is a feature, can be defined as a case for catastrophic forgetting which, as it also the name suggests, is an issue 
  
  Could you please elaborate more on why this claim makes sense or do you have reference to share?
Comments.UserDisplayName: Nicola Bernini
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-24T11:44:49.673Z
Comments.Text: Well, I am not *defining* transfer learning. I am also not claiming anything. My thoughts' basis is: Transfer learning is aimed at using pre-trained network weights to apply to some new task by only updating the previously learned weights for a comparatively very small size of input data. On the contrary, catastrophic forgetting is the tendency of an artificial neural network to completely and abruptly forget previously learned information upon learning new information. I will update the question to make it sensible enough.
Comments.UserDisplayName: naive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-01T15:56:58.730Z
Comments.Text: See [this answer](https://ai.stackexchange.com/a/16143/2444).
Comments.UserDisplayName: nbro
Tags: <machine-learning><transfer-learning><catastrophic-forgetting>
Title: How is transfer learning used to mitigate catastrophic forgetting in neural networks?
UserDisplayName: naive
ThreadText: # How is transfer learning used to mitigate catastrophic forgetting in neural networks?
  
  (Asked by: username_0 on 2019-08-24)
  
  How can transfer learning be used to mitigate [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference). Could someone elaborate on this?
  
  **Tags:** machine-learning, transfer-learning, catastrophic-forgetting
  > **Comments:**
  > **username_1**: I think this claim is false   Specifically I do not see how it could make sense to define transfer learning, which is a feature, can be defined as a case for catastrophic forgetting which, as it also the name suggests, is an issue   Could you please elaborate more on why this claim makes sense or do you have reference to share?  
  > **username_0**: Well, I am not *defining* transfer learning. I am also not claiming anything. My thoughts' basis is: Transfer learning is aimed at using pre-trained network weights to apply to some new task by only updating the previously learned weights for a comparatively very small size of input data. On the contrary, catastrophic forgetting is the tendency of an artificial neural network to completely and abruptly forget previously learned information upon learning new information. I will update the question to make it sensible enough.  
  > **username_2**: See [this answer](https://ai.stackexchange.com/a/16143/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-08-24. Score: 1
  
  Transfer learning is a field where you apply knowledge from a source onto a target. This is a vague notion and there is an abundance of literature pertaining to it. Given your question I will work under the assumption that you are referring weight/architecture sharing between model (in other words training a model on one dataset and using it as a featurizer for another dataset)
  
  Now any learning system without lossless memory will have remnants of catastrophic forgetting. So let's think about how we would implement this transfer and what effects can be derived from this.
  
  1. One implementation involves transferring a component and only training additional layers.
  2. Another is retraining the entire system but at a lower learning rate?
  
  In setting 1, we can make the claim catastrophic forgetting is minimized by the fact that there is an unbiased featurizer that cant *forget* based on a sampling regime, though this does not mean additional layers which are still being trained can still faulter in this error mode.
  
  In setting 2, we can make the claim catasrophic forgetting can be reduced compared to a normal end-to-end no-transfer training because the unbiased featurizers difference can be analytically bounded by its initial transferred featurization (complexity class is based on both the function and the number of steps -- so longer you train, the more likely it can *forget*)
  
  These reasons are talking about mitigating and not erasing the concept of catastrophic forgetting, that is because as I mentioned above *any learning system without lossless memory will have remnants of catastrophic forgetting*, so making the generalized claim about transfer learning may not always fit the bill.
  
  
  ---
  
  