The problem of distributed representation learning is one in which multiple sources of information X1, . . . , XK are processed separately so as to extract useful information about some statistically correlated ground truth Y . We investigate this problem from informationtheoretic grounds. For both discrete memoryless (DM) and memoryless vector Gaussian models, we establish fundamental limits of learning in terms of optimal tradeoffs between relevance and complexity. We also develop a variational bound on the optimal tradeoff that generalizes the evidence lower bound (ELBO) to the distributed setting. Furthermore, we provide a variational inference type algorithm that allows to compute this bound and in which the mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Experimental results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms which we develop in this paper.
This paper studies a distributed representation problem where multiple features X_1,...,X_K are processed (or encoded) separately to estimate (or decode) some quantity of interest Y.
The log loss is considered throughout, which amounts to measuring the mutual information between Y and \hat Y, defined as the "accuracy" of the estimation method. The average rate (measured in number of bits per sample) of the encoded feature is defined as the "complexity" of the representation method.
The author derived the fundamental trade-off between the accuracy and the complexity for any representation-estimation (or encoding-decoding) method.
The author also derived a variational representation of the optimal accuracy-complexity region, which also expresses the optimal encoder and decoder map as the solution of the optimization problem.
Finally, the author considered the case where the joint distribution of P_{X_1,,,,X_K,Y} is unknown, and encoder and decoder are parameterized by neural networks, parameters of which are tuned using data.
I incline to reject the paper, for the following reasons.
1. The accuracy-complexity trade-off studied in the paper is more of a rate-distortion type of information-theoretic problem, where the joint distribution P_{X_1,,,,X_K,Y} is assumed to be known. Its connection to the learning problem, where the joint distribution P_{X_1,,,,X_K,Y} is unknown, is unclear. Even if the precise accuracy-complexity region is obtained, it says little about the sample complexity needed by a learning algorithm to achieve this region.
2. Deriving the optimal encoder-decoder mapping from the variational representation of the accuracy-complexity region also requires the joint distribution, which violates the basic assumption of the learning problem.
3. The author did consider the case where the joint distribution is unknown, and the encoder-decoder pair is learned from data. However, this learning problem is somewhat artificial: each encoder only encodes one of the features, but in order to encode optimally, it has to know the entire joint distribution, hence need to access all the features during training. This discrepancy of seeing different components of the data set during training and inference is not well-motivated.
The author mentioned "multi-view learning" at the beginning of the paper. It would be good if the author can elaborate more on this problem in Sec 4 of Experiment Results, and discuss with more detail on how the proposed method solves this problem and how it is different from the existing results, both in terms of the algorithm and the performance.
=================================================
Feedback to authors' reply
I got a better understanding on how the proposed learning algorithms works after reading the authors' reply.
I guess the idea for the case where the joint distribution is unknown is that, for encoding, different nodes uses its own training data (without accessing other nodes' data) to optimize the encoder separately; while for decoding, the master node trains the decoder uses data available to all nodes to estimate the joint distribution.
In this way, the encoders and the decoder jointly optimizes a variational lower bound of the optimal rate region.
If this is the case, I think the proposed method may have some value in practice. 
But now the question is how good the variational lower bound is compared to the optimal region, and how well can this variational lower bound be approximated by neural networks and how efficient can the training be done. Without theoretical analysis on these questions, one may only use experiments to assess the performance. From Table 2, it looks like the improvement of the proposed method on the existing method is quite marginal.
In summary, I would like to thank the authors' valuable reply. I encourage the authors to study the gap between the variational lower bound and the optimal region, and maybe do more experiments to find a good use case of the proposed method.