On the Distribution of Penultimate Activations of Classification Networks
This paper considers probability distributions of penultimate activations in deep classi?cation networks. We ?rst identify a dual relation between the activations and the weights of the ?nal fully connected layer: learning the networks with the cross-entropy loss makes their (normalized) penultimate activations follow a von Mises-Fisher distribution for each class, which is parameterized by the weights of the ?nal fully-connected layer. Through this analysis, we derive a probability density function of penultimate activations per class. This generative model allows us to synthesize activations of classi?cation networks without feeding images forward through them. We also demonstrate through experiments that our generative model of penultimate activations can be applied to real-world applications such as knowledge distillation and class-conditional image generation.
This paper studies the distribution of the second-to-last layers of neural networks. The paper argues that, under certain assumptions, minimizing the standard cross-entropy loss of a classification task corresponds to minimizing the KL divergence between the distribution of the penultimate activations and a von Mises-Fisher (vMF) distribution. They assess their assumptions through experiments, and use this interpretation of the penultimate activations as vMF distributions on two tasks: knowledge distillation and class-conditional image generation. They argue that this interpretation outperforms other knowledge distillation methods under domain shift, and report a similar success for class conditional image generation. This is an important problem, especially since these penultimate activations have become more frequently used as object representations in downstream tasks. Their result could also help us understand better interpretations of neural networks. The writing is clear throughout, and it*s straightforward to read. The results are based on assumptions that I will try to list below: * normalizing the penultimate activations does not affect classification performance * the class-specific vectors in the final weight matrix have constant normalizations * minimizing cross-entropy loss is equivalent to minimizing the KL divergence between the penultimate activation distribution and a vMF distribution (i.e. the entropy term in equation 6 is negligible) In light of these assumptions, theoretical claims like *learning the networks with the cross-entropy loss makes their (normalized) penultimate activations follow a von Mises-Fisher distribution for each class* in the abstract are false; even if the normalization assumptions are true, this is an approximate result, due to 1) no guarantee that the KL-divergence will be zero and 2) the entropy term in Equation 6. Additionally, Equation 6 is only applied to the numerator of the objective in Equation 4, so the objectives aren*t equivalent. These results are thus necessarily based on approximations and some heuristics. This is not inherently a problem, but I would argue that it necessitates that experiments prove these assumptions to be useful. I find Table 1 to be convincing of the assumption that the weight and activation normalizations are negligible. Figure 1 is also a nice visualization for arguing that penultimate activations are approximately vMF. The MNIST experiment in Table 4 is a red flag, since the presented results do not match the results in the HVAE paper. The paper claims the experimental setup is identical to the HVAE paper, so it is concerning that the results do not match, especially since the results in the HVAE paper appear to outperform the proposed method. It is possible that the results in the original paper are not reproducible or that the experiments differ, but this should be stated in the paper, since the natural assumption would be that the baseline was not adequately reproduced. Small comment: In Table 2 it is not clear from the figure or the caption that CKD is the method the paper is proposing. It would also make sense to bold the best performing method since you do that for the other tables in the paper. I like the paper and think the results are interesting, but because they are based on heuristics and assumptions, the paper currently promises too much. This puts the burden on presenting experiments that are convincing of improvement, which is not currently the case with the HVAE experiment. I would be open to changing my score if these reviews are addressed.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
On the Distribution of Penultimate Activations of Classification Networks
This paper considers probability distributions of penultimate activations in deep classi?cation networks. We ?rst identify a dual relation between the activations and the weights of the ?nal fully connected layer: learning the networks with the cross-entropy loss makes their (normalized) penultimate activations follow a von Mises-Fisher distribution for each class, which is parameterized by the weights of the ?nal fully-connected layer. Through this analysis, we derive a probability density function of penultimate activations per class. This generative model allows us to synthesize activations of classi?cation networks without feeding images forward through them. We also demonstrate through experiments that our generative model of penultimate activations can be applied to real-world applications such as knowledge distillation and class-conditional image generation.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper studies the distribution of the second-to-last layers of neural networks. The paper argues that, under certain assumptions, minimizing the standard cross-entropy loss of a classification task corresponds to minimizing the KL divergence between the distribution of the penultimate activations and a von Mises-Fisher (vMF) distribution. They assess their assumptions through experiments, and use this interpretation of the penultimate activations as vMF distributions on two tasks: knowledge distillation and class-conditional image generation. They argue that this interpretation outperforms other knowledge distillation methods under domain shift, and report a similar success for class conditional image generation. This is an important problem, especially since these penultimate activations have become more frequently used as object representations in downstream tasks. Their result could also help us understand better interpretations of neural networks. The writing is clear throughout, and it*s straightforward to read. The results are based on assumptions that I will try to list below: * normalizing the penultimate activations does not affect classification performance * the class-specific vectors in the final weight matrix have constant normalizations * minimizing cross-entropy loss is equivalent to minimizing the KL divergence between the penultimate activation distribution and a vMF distribution (i.e. the entropy term in equation 6 is negligible) In light of these assumptions, theoretical claims like *learning the networks with the cross-entropy loss makes their (normalized) penultimate activations follow a von Mises-Fisher distribution for each class* in the abstract are false; even if the normalization assumptions are true, this is an approximate result, due to 1) no guarantee that the KL-divergence will be zero and 2) the entropy term in Equation 6. Additionally, Equation 6 is only applied to the numerator of the objective in Equation 4, so the objectives aren*t equivalent. These results are thus necessarily based on approximations and some heuristics. This is not inherently a problem, but I would argue that it necessitates that experiments prove these assumptions to be useful. I find Table 1 to be convincing of the assumption that the weight and activation normalizations are negligible. Figure 1 is also a nice visualization for arguing that penultimate activations are approximately vMF. The MNIST experiment in Table 4 is a red flag, since the presented results do not match the results in the HVAE paper. The paper claims the experimental setup is identical to the HVAE paper, so it is concerning that the results do not match, especially since the results in the HVAE paper appear to outperform the proposed method. It is possible that the results in the original paper are not reproducible or that the experiments differ, but this should be stated in the paper, since the natural assumption would be that the baseline was not adequately reproduced. Small comment: In Table 2 it is not clear from the figure or the caption that CKD is the method the paper is proposing. It would also make sense to bold the best performing method since you do that for the other tables in the paper. I like the paper and think the results are interesting, but because they are based on heuristics and assumptions, the paper currently promises too much. This puts the burden on presenting experiments that are convincing of improvement, which is not currently the case with the HVAE experiment. I would be open to changing my score if these reviews are addressed.
assistant
This paper studies the distribution of the second-to-last layers of neural networks. The paper argues that, under certain assumptions, minimizing the standard cross-entropy loss of a classification task corresponds to minimizing the KL divergence between the distribution of the penultimate activations and a von Mises-Fisher (vMF) distribution. They assess their assumptions through experiments, and use this interpretation of the penultimate activations as vMF distributions on two tasks: knowledge distillation and class-conditional image generation. They argue that this interpretation outperforms other knowledge distillation methods under domain shift, and report a similar success for class conditional image generation. This is an important problem, especially since these penultimate activations have become more frequently used as object representations in downstream tasks. Their result could also help us understand better interpretations of neural networks. The writing is clear throughout, and it*s straightforward to read. The results are based on assumptions that I will try to list below: * normalizing the penultimate activations does not affect classification performance * the class-specific vectors in the final weight matrix have constant normalizations * minimizing cross-entropy loss is equivalent to minimizing the KL divergence between the penultimate activation distribution and a vMF distribution (i.e. the entropy term in equation 6 is negligible) In light of these assumptions, theoretical claims like *learning the networks with the cross-entropy loss makes their (normalized) penultimate activations follow a von Mises-Fisher distribution for each class* in the abstract are false; even if the normalization assumptions are true, this is an approximate result, due to 1) no guarantee that the KL-divergence will be zero and 2) the entropy term in Equation 6. Additionally, Equation 6 is only applied to the numerator of the objective in Equation 4, so the objectives aren*t equivalent. These results are thus necessarily based on approximations and some heuristics. This is not inherently a problem, but I would argue that it necessitates that experiments prove these assumptions to be useful. I find Table 1 to be convincing of the assumption that the weight and activation normalizations are negligible. Figure 1 is also a nice visualization for arguing that penultimate activations are approximately vMF. The MNIST experiment in Table 4 is a red flag, since the presented results do not match the results in the HVAE paper. The paper claims the experimental setup is identical to the HVAE paper, so it is concerning that the results do not match, especially since the results in the HVAE paper appear to outperform the proposed method. It is possible that the results in the original paper are not reproducible or that the experiments differ, but this should be stated in the paper, since the natural assumption would be that the baseline was not adequately reproduced. Small comment: In Table 2 it is not clear from the figure or the caption that CKD is the method the paper is proposing. It would also make sense to bold the best performing method since you do that for the other tables in the paper. I like the paper and think the results are interesting, but because they are based on heuristics and assumptions, the paper currently promises too much. This puts the burden on presenting experiments that are convincing of improvement, which is not currently the case with the HVAE experiment. I would be open to changing my score if these reviews are addressed.