<p>The logit vectors in the aforementioned paper are actually the outputs of the models and not the reference values in the dataset:</p>
<blockquote>
<p>Each model output is a set of logit values normalized via a softmax
function, hereafter denoted as a logit vector</p>
</blockquote>
<p>The average here serves as a proxy for communication efficient comparison of &quot;teacher&quot; and &quot;student&quot; outputs.</p>

CC BY-SA 4.0
2023-10-03T11:00:28.343Z
2023-10-03T11:00:28.343Z
user1360448
<p>As per my understanding, typically in FL, there is a global server that interacts with various client devices. The global server and the client both possess a ML models. The client(s) update their models locally and then send the weights across to the server where it is averaged.</p>
<p>The <a href="https://arxiv.org/pdf/1811.11479.pdf" rel="nofollow noreferrer">paper</a>, Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data, has the following paragraph - &quot;To rectify this, each device in FD stores per-label mean logit vectors, and periodically uploads these local-average logit vectors to a server. For each label, the uploaded local-average logit vectors from all devices are averaged, resulting in a global-average logit vector per label.&quot;</p>
<p>I am really lost with what one can do with &quot;mean logit vectors&quot; of a label. According to me, that's like saying, a dataset consists of 2 labels with the first label coming up 40% of the time and the second 60%. How does this help with prediction? Perhaps my understanding is wrong here.</p>

<deep-learning><federated-learning>
How does Knowledge Distillation help Federated Learning?
desert_ranger
# How does Knowledge Distillation help Federated Learning?

(Asked by: username_0 on 2023-03-30)

As per my understanding, typically in FL, there is a global server that interacts with various client devices. The global server and the client both possess a ML models. The client(s) update their models locally and then send the weights across to the server where it is averaged.

The [paper](https://arxiv.org/pdf/1811.11479.pdf), Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data, has the following paragraph - "To rectify this, each device in FD stores per-label mean logit vectors, and periodically uploads these local-average logit vectors to a server. For each label, the uploaded local-average logit vectors from all devices are averaged, resulting in a global-average logit vector per label."

I am really lost with what one can do with "mean logit vectors" of a label. According to me, that's like saying, a dataset consists of 2 labels with the first label coming up 40% of the time and the second 60%. How does this help with prediction? Perhaps my understanding is wrong here.

**Tags:** deep-learning, federated-learning


---

## 1 answers

### Answer by username_1 on 2023-10-03. Score: 1

The logit vectors in the aforementioned paper are actually the outputs of the models and not the reference values in the dataset:

> Each model output is a set of logit values normalized via a softmax
> function, hereafter denoted as a logit vector

The average here serves as a proxy for communication efficient comparison of "teacher" and "student" outputs.


---

