Answers.Body: <p>The logit vectors in the aforementioned paper are actually the outputs of the models and not the reference values in the dataset:</p>
  <blockquote>
  <p>Each model output is a set of logit values normalized via a softmax
  function, hereafter denoted as a logit vector</p>
  </blockquote>
  <p>The average here serves as a proxy for communication efficient comparison of &quot;teacher&quot; and &quot;student&quot; outputs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-03T11:00:28.343Z
Answers.LastActivityDate: 2023-10-03T11:00:28.343Z
Answers.UserDisplayName: user1360448
Body: <p>As per my understanding, typically in FL, there is a global server that interacts with various client devices. The global server and the client both possess a ML models. The client(s) update their models locally and then send the weights across to the server where it is averaged.</p>
  <p>The <a href="https://arxiv.org/pdf/1811.11479.pdf" rel="nofollow noreferrer">paper</a>, Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data, has the following paragraph - &quot;To rectify this, each device in FD stores per-label mean logit vectors, and periodically uploads these local-average logit vectors to a server. For each label, the uploaded local-average logit vectors from all devices are averaged, resulting in a global-average logit vector per label.&quot;</p>
  <p>I am really lost with what one can do with &quot;mean logit vectors&quot; of a label. According to me, that's like saying, a dataset consists of 2 labels with the first label coming up 40% of the time and the second 60%. How does this help with prediction? Perhaps my understanding is wrong here.</p>
  
Tags: <deep-learning><federated-learning>
Title: How does Knowledge Distillation help Federated Learning?
UserDisplayName: desert_ranger
ThreadText: # How does Knowledge Distillation help Federated Learning?
  
  (Asked by: username_0 on 2023-03-30)
  
  As per my understanding, typically in FL, there is a global server that interacts with various client devices. The global server and the client both possess a ML models. The client(s) update their models locally and then send the weights across to the server where it is averaged.
  
  The [paper](https://arxiv.org/pdf/1811.11479.pdf), Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data, has the following paragraph - "To rectify this, each device in FD stores per-label mean logit vectors, and periodically uploads these local-average logit vectors to a server. For each label, the uploaded local-average logit vectors from all devices are averaged, resulting in a global-average logit vector per label."
  
  I am really lost with what one can do with "mean logit vectors" of a label. According to me, that's like saying, a dataset consists of 2 labels with the first label coming up 40% of the time and the second 60%. How does this help with prediction? Perhaps my understanding is wrong here.
  
  **Tags:** deep-learning, federated-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-03. Score: 1
  
  The logit vectors in the aforementioned paper are actually the outputs of the models and not the reference values in the dataset:
  
  > Each model output is a set of logit values normalized via a softmax
  > function, hereafter denoted as a logit vector
  
  The average here serves as a proxy for communication efficient comparison of "teacher" and "student" outputs.
  
  
  ---
  
  