In this work we offer an information-theoretic framework for representation learning that connects with a wide class of existing objectives in machine learning. We develop a formal correspondence between this work and thermodynamics and discuss its implications.
This paper attempts to establish a notion of thermodynamics for machine learning. Let me give an attempt at summary. First, an objective function is established based on demanding that the multi-information of two graphical models be small. The first graphical model is supposed to represent the actual dependence of variables and parameters used to learn a latent description of the training data, and the model demands that the latents entirely explain the correlation of the data, with the parameters marginalized out. Then, a variational approximation is made to four subsets of terms in this objective function, defining four "thermodynamic"  functionals. Minimizing the sum of these functionals puts a variational upper bound on the objective. Next, the sum is related to an unconstrained Lagrange multiplier problem making use of the facts (1) that such an objective will likely have many different realizations of the thermodynamic functionals for specific value of the bound and (2) that on the optimal surface the value of one of the functional can be parametrized in terms of the three others. If we pick the entropy functional to be parameterized in terms of the others, we find ourself precisely in the where the solution to the optimization is a Boltzmann distribution; the coefficients of the Lagrange multipliers will then take on thermodynamic interpretations in of temperature, generalized chemical potentials, etc. At this point, the machinery of thermodynamics can be brought to bear, including a first law, Maxwell relations (equality of mixed partial derivatives), etc.
I think the line of thinking in this paper is very much worth pursuing, but I think this paper requires significant improvement and modifications before it can be published. Part of the problem is that the paper is both very formal and not very clear. It's hard to understand why the authors are establishing this analogy, where they are going with it, what's its use will be, etc. Thermodynamics was developed to explain the results of experiments and is often explained by working out examples analytically on model systems. This paper doesn't really have either such a motivation or such examples, and I think as a result I think it suffers.
I also think the "Tale of Two Worlds" laid out in Section 2 requires more explanation. In particular, I think more can be said about why Q is the the "world we want" and why minimizing the difference between these worlds is the right way to create an objective. (I have no real problem with the objective once it is derived.) Since this paper is really about establishing this formal relationship, and the starting point is supposed to be the motivating factor, I think this needs to be made much clearer.
The I(Z_i, X_i, Theta) - I(X_i, Z_i) terms could have been combined into a conditional mutual information. (I see this is discussed in Appendix A.) This leads to a different set of variational bounds and a different thermodynamics. Why do we prefer one way over the other? At the level of the thermodynamics, what would be the relationship between these different ways of thinking? Since it's hard to see why I want to bother with doing this thermodynamics (a problem which could be assuaged with worked examples or more direct and clear experiments), it's hard to know how to think about this sort of freedom in the analogy. (I also don't understand why the world Q graphical model is different in Appendix A when we combined terms this way, since the world Q lead to the objective, which is independent of how we variationally bound it.) I think ultimately the problem can be traced to the individual terms in the objective (7) not being positive definitive, giving us the freedom to make different bounds by arranging the pieces to get different combinations of positive definite terms. How am I supposed to think about this freedom?
In conclusion, I would really like to see analogies like this worked out and be used to better understand machine learning methods. But for this program to be successful, I think a very compelling case needs to be made for it. Therefore, I think that this paper needs to be significantly rewritten before it can be published.