Whatever information a deep neural network has gleaned from past data is encoded in its weights. How this information affects the response of the network to future data is largely an open question. In fact, even how to define and measure information in a network entails some subtleties. We measure information in the weights of a deep neural network as the optimal trade-off between accuracy of the network and complexity of the weights relative to a prior. Depending on the prior, the definition reduces to known information measures such as Shannon Mutual Information and Fisher Information, but in general it affords added flexibility that enables us to relate it to generalization, via the PAC-Bayes bound, and to invariance. For the latter, we introduce a notion of effective information in the activations, which are deterministic functions of future inputs. We relate this to the Information in the Weights, and use this result to show that models of low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs. These relations hinge not only on the architecture of the model, but also on how it is trained.
Summary of the paper:
This is a theoretical paper that builds on top of Achille and Soatto (2018), Achille et al. (2019), McAllester (2013), and Berglund (2011),. The paper attempts to answer the relationship between the inductive bias of SGD, generalization of DNNs, and the invariance of learned representation from an information theoretical point of view. 
The paper mentioned many interesting links. In my opinion, the contributions are the following:
1. Invoking the theoretical result of Berglund (2011) to justify why Fisher information is relevant -- SGD tends to avoid local minima with high Fisher information. 
2. Relating the Fisher information and the stability of SGD to I(w; D).
3. Introducing the definition of effective information in the activation, and show that which is closely related to the Fisher information. 
About the rating:
This is basically a good paper but I have a few concerns: 
1. A large fraction of this paper are taken from Achille and Soatto (2018), Achille et al. (2019).  
2. In terms of impact, the paper is somehow incomplete -- it only demonstrates that the Fisher information is important, but the insights didn't lead to any substantial improvement over the current deep learning framework. 
Detailed comments:
1. In my opinion, defining "information in the weights for the task D" by KL(Q||P) is inaccurate. 
The weights themselves are information, which form a representation or a lossy compression of the data (which is also an information). 
According to the rate-distortion theory, what we care about is the amount of information the representation attains rather than "where" the information are. Therefore, we should talk about "rate" or "amount of information" or "mutual information" rather than "information" itself.
A missing reference regarding this point: 
Hu et al. "β-BNN: A Rate-Distortion Perspective on Bayesian Neural Networks." 2018,
which derives the information Lagrangian directly from rate-distortion theory. 
2. More discussions on Xu and Raginsky (2017) is expected, since it proposes to use I(w; D) as a generalization bound. It seems, in terms of generalization, minimizing I(w; D) is a sufficient condition while minimizing Fisher is a necessary condition.  
3. There are in fact 4 key aspects: sufficiency, minimality, invariance and generalization. It would be great to have a theorem to summarize the relationships between them.  
4. Could you elaborate on the footnote 3?