Answers.Body: <p>My student model has half of the sizes of my teacher model. Hence, I would duplicate student_c to match teacher_c and do L2, kind of like the reverse version of avg pool option without losing information.</p>
  <p>I doubt the &quot;pair-wise L2&quot; idea to be reasonable. A student channel being close to one teacher channel while far away from another teacher channel is well needed, meaning that the loss should be zero. However, it can have a high loss due to the far teacher channel.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-28T02:50:47.077Z
Answers.LastActivityDate: 2024-03-29T05:52:33.667Z
Answers.LastEditDate: 2024-03-29T05:52:33.667Z
Answers.UserDisplayName: trueGeek
Body: <p>Assume a teacher model and a student model. Teacher is bigger than student in terms of depth and/or width, however, comes from the same &quot;family&quot;.</p>
  <p>In addition to the loss that involves the ground-truth labels, I also want to distill intermediate output feature maps. Assume an intermediate feature map of a teacher of shape <code>[teacher_c,h,w]</code> and a feature map of the student <code>[student_c,h,w]</code>, what's the best way to distill different number of channels given same feature map resolution?</p>
  <p>If channels were the same, we could use L2 or something else, but this is not the case here.</p>
  <p>One option can be to avg pool to <code>[h,w]</code>, then L2, but that loses a lot of info.</p>
  <p>Another option is pair-wise L2, normalized: L2 between pairs of teacher&lt;&gt;student channels (basically <code>teacher_c_i*student_c_j</code> MSE of feature-maps.</p>
  <p>Any other suggestions?</p>
  
Tags: <deep-learning><computer-vision>
Title: KD-loss between intermediate feature maps of different channels
UserDisplayName: Natan ZB
ThreadText: # KD-loss between intermediate feature maps of different channels
  
  (Asked by: username_0 on 2024-03-01)
  
  Assume a teacher model and a student model. Teacher is bigger than student in terms of depth and/or width, however, comes from the same "family".
  
  In addition to the loss that involves the ground-truth labels, I also want to distill intermediate output feature maps. Assume an intermediate feature map of a teacher of shape `[teacher_c,h,w]` and a feature map of the student `[student_c,h,w]`, what's the best way to distill different number of channels given same feature map resolution?
  
  If channels were the same, we could use L2 or something else, but this is not the case here.
  
  One option can be to avg pool to `[h,w]`, then L2, but that loses a lot of info.
  
  Another option is pair-wise L2, normalized: L2 between pairs of teacher<>student channels (basically `teacher_c_i*student_c_j` MSE of feature-maps.
  
  Any other suggestions?
  
  **Tags:** deep-learning, computer-vision
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-28. Score: 0
  
  My student model has half of the sizes of my teacher model. Hence, I would duplicate student\_c to match teacher\_c and do L2, kind of like the reverse version of avg pool option without losing information.
  
  I doubt the "pair-wise L2" idea to be reasonable. A student channel being close to one teacher channel while far away from another teacher channel is well needed, meaning that the loss should be zero. However, it can have a high loss due to the far teacher channel.
  
  
  ---
  
  