Answers.Body: <p>No, transfer learning cannot be applied &quot;between&quot; different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer learning but not the architecture.</p>
  <p>In my understanding, transfer learning is also only really effective in deep learning, but I could be wrong, considering that <a href="https://www.google.com/search?q=random+forest+transfer+learning" rel="nofollow noreferrer">this Google search</a> seems to yield some results.</p>
  <p>You might otherwise be thinking of knowledge distillation, which is a related but different concept, where an already trained network acts as a teacher and teaches another network (a student network) with possibly a different architecture (or a machine learning model not based on neural networks at all) the correct outputs for a bunch of input examples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-22T03:16:03.560Z
Answers.Text: "Transfer learning and machine learning are closely related. On one hand, the aim of transfer learning encompasses that of machine learning in that its key in- gredient is “generalization.” In other words, it explores how to develop general and robust machine learning models that can apply to not only the training data, but also unanticipated future data. Therefore, all machine learning models should have the ability to conduct transfer learning." Chapter **1.3 Relationship to Existing Machine Learning Paradigms** of *Transfer Learning* by Yang, Zhang, Dai, and Pan.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-21T20:06:58.720Z
Answers.LastActivityDate: 2020-09-21T20:06:58.720Z
Answers.UserDisplayName: HelloGoodbye
Answers.Body: <p>I think the answer is yes, this problem is known as &quot;Where To transfer&quot;, please refer to this paper <a href="https://arxiv.org/abs/1905.05901" rel="nofollow noreferrer">Learning What and Where to Transfer</a> to know what I am talking about, and correct me if I understand something wrongly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-18T20:53:54.790Z
Answers.LastActivityDate: 2021-04-18T20:53:54.790Z
Answers.UserDisplayName: Kais Hasan
Body: <p>There are many types of CNN architectures: LeNet, AlexNet, VGG, GoogLeNet, ResNet, etc. Can we apply transfer learning between any two different CNN architectures? For instance, can we apply transfer learning from AlexNet to GoogLeNet, etc.? Or even just from a &quot;conventional&quot; CNN to one of these other architectures, or the other way around? Is this possible in general?</p>
  <p>EDIT: My understanding is that <em>all</em> machine learning models have the ability to perform transfer learning. If this is true, then I guess the question is, as I said, whether we can transfer between two <em>different</em> CNN architectures – for instance, what was learned by a conventional CNN to a different CNN architecture.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-20T17:46:50.293Z
Comments.Text: Do you know what "transfer learning" usually refers to in machine learning?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-20T17:48:34.003Z
Comments.Text: @nbro I’m studying it now. Apparently there are four categories of “transfer learning”, each involving a transfer of some different aspect of the neural network. Is that correct?
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-20T17:49:25.890Z
Comments.Text: I only know one form of learning that people usually refer to as "transfer learning", but maybe there are others. Can you please link us to the article or resource you are reading that tells you that there are 4 different forms of transfer learning?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-20T17:53:02.490Z
Comments.Text: @nbro This isn’t the resource I am studying, but you can find the categories in this https://medium.com/georgian-impact-blog/transfer-learning-part-1-ed0c174ad6e7 article.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-20T17:59:13.497Z
Comments.Text: @nbro actually, I think that article lists 5 categories. Regardless, you’ll see what I mean.
Comments.UserDisplayName: The Pointer
Tags: <convolutional-neural-networks><transfer-learning>
Title: Can we apply transfer learning between any two different CNN architectures?
UserDisplayName: The Pointer
ThreadText: # Can we apply transfer learning between any two different CNN architectures?
  
  (Asked by: username_0 on 2020-09-20)
  
  There are many types of CNN architectures: LeNet, AlexNet, VGG, GoogLeNet, ResNet, etc. Can we apply transfer learning between any two different CNN architectures? For instance, can we apply transfer learning from AlexNet to GoogLeNet, etc.? Or even just from a "conventional" CNN to one of these other architectures, or the other way around? Is this possible in general?
  
  EDIT: My understanding is that *all* machine learning models have the ability to perform transfer learning. If this is true, then I guess the question is, as I said, whether we can transfer between two *different* CNN architectures – for instance, what was learned by a conventional CNN to a different CNN architecture.
  
  **Tags:** convolutional-neural-networks, transfer-learning
  > **Comments:**
  > **username_1**: Do you know what "transfer learning" usually refers to in machine learning?  
  > **username_0**: @nbro I’m studying it now. Apparently there are four categories of “transfer learning”, each involving a transfer of some different aspect of the neural network. Is that correct?  
  > **username_1**: I only know one form of learning that people usually refer to as "transfer learning", but maybe there are others. Can you please link us to the article or resource you are reading that tells you that there are 4 different forms of transfer learning?  
  > **username_0**: @nbro This isn’t the resource I am studying, but you can find the categories in this https://medium.com/georgian-impact-blog/transfer-learning-part-1-ed0c174ad6e7 article.  
  > **username_0**: @nbro actually, I think that article lists 5 categories. Regardless, you’ll see what I mean.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-09-21. Score: 1 | Accepted answer
  
  No, transfer learning cannot be applied "between" different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer learning but not the architecture.
  
  In my understanding, transfer learning is also only really effective in deep learning, but I could be wrong, considering that [this Google search](https://www.google.com/search?q=random+forest+transfer+learning) seems to yield some results.
  
  You might otherwise be thinking of knowledge distillation, which is a related but different concept, where an already trained network acts as a teacher and teaches another network (a student network) with possibly a different architecture (or a machine learning model not based on neural networks at all) the correct outputs for a bunch of input examples.
  
  > **Comments:**
  > **username_0**: "Transfer learning and machine learning are closely related. On one hand, the aim of transfer learning encompasses that of machine learning in that its key in- gredient is “generalization.” In other words, it explores how to develop general and robust machine learning models that can apply to not only the training data, but also unanticipated future data. Therefore, all machine learning models should have the ability to conduct transfer learning." Chapter **1.3 Relationship to Existing Machine Learning Paradigms** of *Transfer Learning* by Yang, Zhang, Dai, and Pan.  
  
  ---
  
  ### Answer by username_3 on 2021-04-18. Score: 1
  
  I think the answer is yes, this problem is known as "Where To transfer", please refer to this paper [Learning What and Where to Transfer](https://arxiv.org/abs/1905.05901) to know what I am talking about, and correct me if I understand something wrongly.
  
  
  ---
  
  