Likelihood-based Permutation Invariant Loss Function for Probability Distributions
We propose a permutation-invariant loss function designed for the neural networks reconstructing a set of elements without considering the order within its vector representation. Unlike popular approaches for encoding and decoding a set, our work does not rely on a carefully engineered network topology nor by any additional sequential algorithm. The proposed method, Set Cross Entropy, has a natural information-theoretic interpretation and is related to the metrics defined for sets. We evaluate the proposed approach in two object reconstruction tasks and a rule learning task.
This paper proposes an objective function for sets autoencoders such that the loss is permutation invariant with respect to the order of reconstructed inputs. I think that the problem of autoencoding sets is important and designing custom loss functions is a good way to approach it. Thus, I quite like the idea of SCE  from that point of view. However, I find the experiments not convincing for me to accept the paper. 
While reading Section 3, I found it hard to keep in mind that x and y are discrete probability distributions and the notation like P(x=y) is not making things easier. Actually, I’ve never seen cross entropy written with P(x=y). Though is my personal opinion and I don’t have a suggestion on how to improve the explanations in Eq. 1-8. However, I’m glad there is an example at the end of Section 3.
I have some comments on the Experiments section. 
* Puzzles:
(1) Figure 1 could have been prettier. 
(2) The phrase “The purpose of this experiment is to reproduce the results from (Zaheer et al., 2017)” makes little sense to me.  In Deep Sets, there are many experiments and it’s not clear which experiment is meant here.
(3) Table 1 gives test error statistics for 10 runs. What is changed in every run? Does the test set stay the same in every run or is a kind of a cross-validation? Or is it just a different random seed for the initial weights? I could not find an explanation in the text, so there is no way I can interpret the results.
* Blocksworld: the reconstructions are nice, but the numbers in Table 2 are difficult to interpret. 
For example, I cannot estimate how important the difference of 10 points in SH scores is.
* Rule learning ILP tasks: I don’t know enough about learning logic rules tasks to comment on those experiments, but Table 3 seems overwhelming and the concept of 10 runs is still unclear.
--- General comment on the experiments ---
I think an important goal of any autoencoder is to learn a representation that can be useful in other tasks. There is even an example in the paper: “set representation of the environment is crucial in the robotic systems”. Thus, the experiments I would like to see are about evaluating the quality of a representation from an SCE-trained autoencoder compared to other training methods.  Without those experiments, I cannot estimate how valuable the SCE loss function is.