Answers.Body: <p>Well, I would say, that purpose of Bayesian inference is not transfer learning, but uncertainty estimation.</p>
  <p>In case you have good feature extractor in the beginning, you can adjust small number of parameters, like few last layers to achieve good quality in few epochs.</p>
  <p>However, this is about adjusting the means of distributions over each weight.</p>
  <p>Concerning the variance, I think transfer learning is inapplicable since the source and target distributions can be very different. For example, ImageNet is a broad and diverse dataset with many classes, and the target problem can involve only a few classes. Most probably, uncertainty estimate and the standard deviations of model weights on the ImageNet would be larger, than for the model, trained solely on the target task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T07:57:19.423Z
Answers.Text: I do not get the argument of whether transfer learning would be make more or less sense for BNNs compared to usual ANNs. Whether the weights are distribution functions or pure numbers; what's the difference here? Don't BNNs overfit less, so transfer learning would actually make more sense?
Answers.UserDisplayName: Mathy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T21:05:20.663Z
Answers.LastActivityDate: 2021-07-06T21:05:20.663Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>In transfer learning, we use big data from similar tasks to learn the parameters of a neural network, and then fine-tune the neural network on our own task that has little data available for it. Here, we can think of the transfer learning step as learning a (proper) prior, and then fine-tuning as learning the posterior.</p>
  <p>So, we can argue that Bayesian networks can also solve the problem of small data-set regimes. But, what are the directions that we can mix Bayesian neural networks with similar tasks to transfer learning, for example, few-shot learning?</p>
  <p>They make sense when they both take a role as a solution to the low data regime problems, but I can't think of a mix of them to tackle this issue.</p>
  <p>Is it possible, for example, to learn a BNN for which we have picked a good prior to learn the posterior with little data and use the weight distribution for learning our new task? Is there any benefit in this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-07T10:39:39.190Z
Comments.Text: I think you could interpret the way BNNs are formalised as a form of transfer learning, but this is an interpretation. So, you could train a BNN with some data $D$ to learn a posterior $p$ for the weights. Then you could retrain this BNN with new data $D'$ by starting from that posterior $p$ that you learned previously, i.e. we would start learning on the new task with the prior that corresponds to the previously learned posterior $p$. In this sense, we could call it _transfer learning_, but probably people would simply call it Bayesian inference, although the distributions have changed.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-07T10:39:58.760Z
Comments.Text: I am not aware of any research work that has attempted to tackle or interpret the transfer learning problem with BNNs, but it's also been a while since I worked with them.
Comments.UserDisplayName: nbro
Tags: <reference-request><transfer-learning><bayesian-deep-learning><bayesian-neural-networks><one-shot-learning>
Title: How could Bayesian neural networks be used for transfer learning?
UserDisplayName: samsambakster
ThreadText: # How could Bayesian neural networks be used for transfer learning?
  
  (Asked by: username_0 on 2021-07-06)
  
  In transfer learning, we use big data from similar tasks to learn the parameters of a neural network, and then fine-tune the neural network on our own task that has little data available for it. Here, we can think of the transfer learning step as learning a (proper) prior, and then fine-tuning as learning the posterior.
  
  So, we can argue that Bayesian networks can also solve the problem of small data-set regimes. But, what are the directions that we can mix Bayesian neural networks with similar tasks to transfer learning, for example, few-shot learning?
  
  They make sense when they both take a role as a solution to the low data regime problems, but I can't think of a mix of them to tackle this issue.
  
  Is it possible, for example, to learn a BNN for which we have picked a good prior to learn the posterior with little data and use the weight distribution for learning our new task? Is there any benefit in this?
  
  **Tags:** reference-request, transfer-learning, bayesian-deep-learning, bayesian-neural-networks, one-shot-learning
  
  > **Comments:**
  > **username_1**: I think you could interpret the way BNNs are formalised as a form of transfer learning, but this is an interpretation. So, you could train a BNN with some data $D$ to learn a posterior $p$ for the weights. Then you could retrain this BNN with new data $D'$ by starting from that posterior $p$ that you learned previously, i.e. we would start learning on the new task with the prior that corresponds to the previously learned posterior $p$. In this sense, we could call it _transfer learning_, but probably people would simply call it Bayesian inference, although the distributions have changed.  
  > **username_1**: I am not aware of any research work that has attempted to tackle or interpret the transfer learning problem with BNNs, but it's also been a while since I worked with them.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-07-06. Score: 0
  
  Well, I would say, that purpose of Bayesian inference is not transfer learning, but uncertainty estimation.
  
  In case you have good feature extractor in the beginning, you can adjust small number of parameters, like few last layers to achieve good quality in few epochs.
  
  However, this is about adjusting the means of distributions over each weight.
  
  Concerning the variance, I think transfer learning is inapplicable since the source and target distributions can be very different. For example, ImageNet is a broad and diverse dataset with many classes, and the target problem can involve only a few classes. Most probably, uncertainty estimate and the standard deviations of model weights on the ImageNet would be larger, than for the model, trained solely on the target task.
  
  > **Comments:**
  > **username_3**: I do not get the argument of whether transfer learning would be make more or less sense for BNNs compared to usual ANNs. Whether the weights are distribution functions or pure numbers; what's the difference here? Don't BNNs overfit less, so transfer learning would actually make more sense?  
  
  ---
  
  