We achieve bias-variance decomposition for Boltzmann machines using an information geometric formulation. Our decomposition leads to an interesting phenomenon that the variance does not necessarily increase when more parameters are included in Boltzmann machines, while the bias always decreases. Our result gives a theoretical evidence of the generalization ability of deep learning architectures because it provides the possibility of increasing the representation power with avoiding the variance inflation.
Summary of the paper:
The paper derives a lower bound on the expected  squared KL-divergence between a true distribution and the sample based maximum likelihood estimate (MLE) of that distribution modelled by an Boltzmann machine (BM) based on methods from information geometry. This  KL-divergence is first split into the squared KL-divergence between the true distribution and MLE of that distribution,  and the expected squared KL-divergence between the MLE of the true distribution and the sample based MLE (in a similar spirit to splitting the excess error into approximation and estimation error in statistical learning theory). The letter is than lower bounded (leading to a lower bound on the overall KL-divergence) by a term  which does not necessarily increase if the number of model parameters is increased. 
Pros:
- Using insights from information geometry  opens up a very interesting and (to my knowledge) new approach for analysing the generalisation ability of ML models.
- I am not an expert on information geometry and I did not find the time to follow all the steps of the proof in detail, but the analysis seems to be correct.
Cons:
- The fact that the lower bound does not necessary increase with a growing number of parameters does not guarantee that the same holds true for the KL-divergence (in this sense an upper bound would be more informative). Therefore, it is not clear how much of insights the theoretical analysis gives for practitioners (it could be nice to analyse the tightness of the bound for toy models).
- Another drawback reading the practical impact is, that the theorem bounds the expected  squared KL-divergence between a true distribution and the sample based MLE, while training minimises the divergence between the empirical distribution and the model distribution ( i.e. the sample based MLE in the optimal case),  and the theorem does not show the dependency on the letter. 
I found some parts difficulty to understand and clarity could be improved  e.g. by
- explaining why minimising KL(\hat P, P_B) is equivalent to minimising the KL-divergence between the empirical distribution and the Gibbs distribution \Phi.
- explaining in which sense the formula on page 4 is equivalent to “the learning equation of Boltzmann machines”.
- explaining what is the MLE of the true distribution (I assume the closest distribution in the set of distributions that can be modelled by the BM).
Minor comments:
- page 1: and DBMs….(Hinton et al., 2006) : The paper describes deep belief networks (DBNs) not DBMs 
- \theta is used to describe the function in eq. (2) as well as the BM parameters in Section 2.2 
- page 5: “nodes H is” -> “nodes H are” 
REVISION:
Thanks to the reviewers for replying to my comments and making the changes. I think they improved the paper. On the other hand the other reviewers raised valid questions, that led to my decision to not change the overall rating of the paper.