Answers.Body: <p>Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call <code>optimizer1.step(), optimizer2.step()</code> with a single <code>backward()</code> call</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T11:55:14.460Z
Answers.LastActivityDate: 2021-01-14T11:55:14.460Z
Answers.UserDisplayName: Alex
Body: <p>I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a <code>lr</code> of 0.01 was found to be best working via hyperparameter tunings</p>
  <p>However, when I trained them as a separate task (binary classification), a <code>lr</code> of 0.001 for images and 0.01 for numerical data were best.
  As for an AUC metric, the combined network (0.818) is performing on average of image and numerical networks (0.799 and 0.821, respectively).
  Here I thought maybe the combined network's <code>lr</code> is too high for image part and should apply lower <code>lr</code> for that part. But, I don't know if it is possible.</p>
  <p>If anyone has any idea of what is what, let me know</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-14T16:43:23.477Z
Comments.Text: Is this a programming issue/question or a conceptual/theoretical one? You tagged your question with PyTorch, so maybe you're looking for a solution in PyTorch and not really for a conceptual answer. If yes, then this question is off-topic here, and should be asked on Stack Overflow, because it would be a programming issue, which is off-topic here. Please, let me know, because this post was voted to be closed as off-topic, probably because of that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-14T23:47:33.643Z
Comments.Text: it is a conceptual one, I tagged pytorch just in case someone has done something similar in pytorch
Comments.UserDisplayName: bit_scientist
Tags: <neural-networks><convolutional-neural-networks><pytorch><hyperparameter-optimization>
Title: Is it possible to train one part of the network with a particular learning rate and the other part with a different one?
UserDisplayName: bit_scientist
ThreadText: # Is it possible to train one part of the network with a particular learning rate and the other part with a different one?
  
  (Asked by: username_0 on 2021-01-14)
  
  I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a `lr` of 0.01 was found to be best working via hyperparameter tunings
  
  However, when I trained them as a separate task (binary classification), a `lr` of 0.001 for images and 0.01 for numerical data were best.
  As for an AUC metric, the combined network (0.818) is performing on average of image and numerical networks (0.799 and 0.821, respectively).
  Here I thought maybe the combined network's `lr` is too high for image part and should apply lower `lr` for that part. But, I don't know if it is possible.
  
  If anyone has any idea of what is what, let me know
  
  **Tags:** neural-networks, convolutional-neural-networks, pytorch, hyperparameter-optimization
  > **Comments:**
  > **username_1**: Is this a programming issue/question or a conceptual/theoretical one? You tagged your question with PyTorch, so maybe you're looking for a solution in PyTorch and not really for a conceptual answer. If yes, then this question is off-topic here, and should be asked on Stack Overflow, because it would be a programming issue, which is off-topic here. Please, let me know, because this post was voted to be closed as off-topic, probably because of that.  
  > **username_0**: it is a conceptual one, I tagged pytorch just in case someone has done something similar in pytorch  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-14. Score: 1
  
  Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call `optimizer1.step(), optimizer2.step()` with a single `backward()` call
  
  
  ---
  
  