BAYESIAN PREDICTION OF FUTURE STREET SCENES USING SYNTHETIC LIKELIHOODS
For autonomous agents to successfully operate in the real world, the ability to anticipate future scene states is a key competence. In real-world scenarios, future states become increasingly uncertain and multi-modal, particularly on long time horizons. Dropout based Bayesian inference provides a computationally tractable, theoretically well grounded approach to learn likely hypotheses/models to deal with uncertain futures and make predictions that correspond well to observationsare well calibrated. However, it turns out that such approaches fall short to capture complex real-world scenes, even falling behind in accuracy when compared to the plain deterministic approaches. This is because the used log-likelihood estimate discourages diversity. In this work, we propose a novel Bayesian formulation for anticipating future scene states which leverages synthetic likelihoods that encourage the learning of diverse models to accurately capture the multi-modal nature of future scene states. We show that our approach achieves accurate state-of-the-art predictions and calibrated probabilities through extensive experiments for scene anticipation on Cityscapes dataset. Moreover, we show that our approach generalizes across diverse tasks such as digit generation and precipitation forecasting.
The work proposes a Bayesian neural network model that is a hybrid between autoencoders and GANs, although it is not presented like that. Specifically, the paper starts from a Bayesian Neural Network model, as presented in Gal and Ghahramani, 2016 and makes two modifications.
First, it proposes to define one Bernoulli variational distribution per weight kernel, instead  of per patch (in the original work there was one Bernoulli distribution per patch kernel). As the paper claims, this reduces the complexity to be exponential to the number of weights, instead of the number of patches, which leads to a much smaller number of possible models. Also, because of this modification the same variational distributions are shared between locations, being closer to the convolutional nature of the model.
The second modification is the introduction of synthetic likelihoods. Specifically, in the original network the variational distributions are designed such that the KL-divergence of the true posterior p(ω|X, y) and the approximate posterior q(ω) is minimiezd. This leads to the optimizer encouraging the final model to be close to the mean, thus resulting in less diversity. By re-formulating the KL-divergence, the final objective can be written such that it depends on the likelihood ratio between generated/"fake" samples and "true" data samples. This ratio can then be approximated by a GAN-like discriminator. As the optimizer now is forced to care about the ratio instead of individual samples, the model is more diverse.
Both modifications present some interesting ideas. Specifically, the number of variational parameters is reduced, thus the final models could be much better scaleable. Also, using synthetic likelihoods in a Bayesian context is novel, to the best of my knowledge, and does seem to be somewhat empirically justified. 
The negative points of the paper are the following.
- The precise novelty of the first modification is not clearly explained. Indeed, the number of possible models with the proposed approach is reduced. However, what is the degree to which it is reduced. With some rough calculations, for an input image of resolution 224x224, with a kernel size of 3x3 and stride 1, there should be about 90x90 patches. That is roughly a complexity of O(N^2) ~ 8K (N is the number of patches). Consider the proposed variational distributions with 512 outputting channels, this amount to 3x3x512 ~ 4.5K. So, is the advantage mostly when the spatial resolution of the image is very high? What about intermediate layers, where the resolution is typically smaller?
- Although seemingly ok, the experimental validation has some unclarities.
  + First, it is not clear whether it is fair in the MNIST experiment to report results only from the best sampled model, especially considering that the difference from the CVAE baseline is only 0.5%. The standard deviation should also be reported.
  + In Table 2 it is not clear what is compared against what. There are three different variants of the proposed model. The WD-SL does exactly on par with the Bayes-Standard (although for some reason the boldface font is used only for the proposed method. The improvement appears to come from the synthetic likelihoods. Then, there is another "fine-tuned" variant for which only a single time step is reported, namely +0.54 sec. Why not report numbers for all three future time steps? Then, the fine-tuned version (WD-SL-ft) is clearly better than the best baselines of Luc et al., however, the segmentation networks are also quite different (about 7% difference in mIoU), so it is not clear if the improvement really comes from the synhetic likelihoods or from the better segmentation network. In short, the only configuration that appears to be convincing as is is for the 0.06 sec. I would ask the authors to fill in the blank X spots and repeat fair experiments with the baseline.
- Generally, although the paper is ok written, there are several unclarities.
  + Z_K in eq. (4) is not defined, although I guess it's the matrix of the z^{i, j}_{k, k'}
  + In eq (6) is the z x σ a matrix or a scalar operation? Is z a matrix or a scalar?
  + The whole section 3.4 is confusing and it feels as if it is there to fill up space. There is a rather intricate architecture, but it is not clear where it is used. In the first experment a simple fully connected network is used. In the second experiment a ResNet is used. So, where the section 3.4 model used?
  + In the first experiment a fully connected network is used, although the first novelty is about convolutions. I suppose the convolutions are not used here? If not, is that a fair experiment to outline the contributions of the method?
  + It is not clear why considering the mean of the best 5% predictions helps with evaluating the predicted uncertainty? I understand that this follows by the citation, but still an explanation is needed.
All in all, there are some interesting ideas, however, clarifications are required before considering acceptance.