BRUNO: A Deep Recurrent Model for Exchangeable Data
We present a novel model architecture which leverages deep learning tools to perform exact Bayesian inference on sets of high dimensional, complex observations. Our model is provably exchangeable, meaning that the joint distribution over observations is invariant under permutation: this property lies at the heart of Bayesian inference. The model does not require variational approximations to train, and new samples can be generated conditional on previous samples, with cost linear in the size of the conditioning set. The advantages of our architecture are demonstrated on learning tasks that require generalisation from short observed sequences while modelling sequence variability, such as conditional image generation, few-shot learning, and anomaly detection.
[After author rebuttal]  My thanks to the authors for their careful rebuttal. I found it convincing & I've modified my review to a weak accept. Original review below.  [Original review]  This paper proposes a model for exchangeable data based on sampling latent variables from an exchangeable student T process, followed by passing the variables through an easily-invertible (NVP) transformation. The paper proposes an efficiently-computable decomposition of the proposed distribution, and provides experimental evidence of its utility.  This paper is reasonably well-written and clearly explained. The intro and relevant work sections are particularly well done. Some comments for improvements in the rest of the paper: -- section 3.3: the generative model is described backwards, i.e. x first (with x exchangeable), then a mapping from z to x (z exch), then the distribution on z. I would think it would be easier to understand if the distribution on z is described, then how z maps to observed data x, then show x is exchangeable -- from l.173-183, it's clear that the paper derives the quantities needed to express p(z_{n+1} | z_{1:n}), but it's not clear why that distribution is needed or useful -- section 3.4 makes it clearer that the efficient expression of p( *_n+1 | *_{1:n} ) is used for training, where the goal is to maximize the log marginal p(x_1, ... x_n) (decomposed as the sequence of conditionals).  -- I think the right way to fix this confusion is to move section 3.4 earlier in 3.3, then expand on it. That is, state around l.168 that given the specification of the model, the major task is to train it. Then explicitly write the training problem of maximizing the log marginal, then decompose it, then show that the student T process yields a nice efficient formula for the decomposition, then argue this can be optimized (via e.g. autograd). Finally, mention that the efficient conditional formula can be used for prediction once training is done. It would also be very helpful to write down the full model including the NVP transformation and how it affects training. -- A similar clarity issue occurs in the experiments. It's generally unclear how data is being used for training & testing/prediction. Is the idea that you take a bunch of sequences of length N and optimize the sum of log marginals for each (  so maximize sum_{j=1}^J sum_{n=1}^{N-1} log p(x_{j,n+1} | x_{j,1:n}) where each (x_{j,1}, ... , x{j, N}) is an observed sequence )?  -- I'm not sure what I'm supposed to get from figure 2. Aside from being too small (given the intricate characters), it is missing a lot of explanation. What is an "input sequence"? Were these samples generated on a model trained on the whole dataset or a single sequence? Is this showing samples of p(x_{N+1}, ... x_{2N} | x_{1:N}) where x_{1:N} are shown in the top row? Is this all simulated from the model? I really am not sure what I'm seeing here. -- I'm also not convinced that I see "lower variability in appearance" for images in the right col -- the experiment in 4.2 is a bit clearer, but it's still missing details on how the model was trained. Is this the same model from 4.1? -- the extra "discriminative training" is not clear and not well-described.  - a technical question: Is there a known de Finetti representation for the exchangeable student T (similar to that for the gaussian at l.37)? If one were to explicitly incorporate this top hierarchical layer into the model, would that make inference more efficient (given that the z's would then be conditionally independent)?  Given the missing/unclear details in section 3 and the experiments, I've given this a weak reject. It's very borderline though; I'd certainly be open to switching to an accept given a convincing rebuttal.  Detailed comments: - l.2 - the paper asserts it performs exact bayesian inference. I would hesitate to make this claim, given that the paper marginalizes out the latent parameter - l.12-16: exchangeable sequences are not a model for unordered sets, but rather for ordered sequences where the probability of generating a particular sequence is invariant to reordering. Sequences of unordered sets (and derivative combinatorial structures) need not be exchangeable, although sometimes are. cf Pitman, "exchangeable and partially exchangeable random partitions" and Broderick, Pitman, & Jordan "feature allocations, probability functions, and paintboxes"  - Figure 1 should appear later in the paper, perhaps at the beginning of section 3.3.  - mathcal script font shouldn't be used for TP / GP. Just use regular text for these initialisms - l.105: "nontrivial differences in the task" - better to just be clear here about exactly why they're different and what benefit TP gives over GP - l.130: In particular it's the marginalization over the inverse Wishart covariance matrix - l.134: NVP is undefined - l.150-153: is the uniform noise discussion really necessary? Why not just ignore the discreteness and treat the quantized data as real-valued in practice? - l.163+: generally I'd just use two subscripts rather than superscript - too easily confused with exponentiation (especially in statements like K_{ii}^d = v^d)