We present fast implementations of linear interpolation operators for piecewise linear functions and multi-dimensional look-up tables. We use a compiler-based solution using MLIR to accelerate this family of workloads. On two-layer and deep lattice networks, we show these strategies run 5 − 10× faster on a standard CPU compared to a C++ interpreter implementation that uses prior techniques, producing runtimes that are 1000s of times faster than TensorFlow 2.0 for single evaluations.

This paper proposes several low-level code optimizations aimed at speeding up evaluation time for linearly interpolated look-up tables, a method often used in situations where fast evaluation times are required. The paper is very practically oriented in the sense that it does not aim for improving asymptotic speed-ups, but rather real speed-ups expressible in terms of CPU cycles that are achievable by exploiting compiler optimizations such as branch prediction and loop unrolling. The focus is on unbatched computations which can arise in many real-time situations. The proposed implementation techniques are as follows:
- A method for fast index mapping, which first transforms inputs using a monotonic function such as log_2 or 2^x, and the applying a branch-free linear search implementation on a uniformly-spaced auxiliary LUT.
- A memory-efficient bit-packing technique to store both integer and floating point representations together.
- Speed-up for multilinear interpolation using latency hiding
- Branch-free implementation of sorting permutation, needed for simplex interpolation
The proposed implementation is then evaluated on 4 different benchmarks, with considerable speed gains over interpreter-based baselines, while batching and single-vs-double precision do not have major impact on speed.
While the paper is relevant and interesting, and the proposed techniques are reasonable and probably result of a considerable amount of work, more effort is needed to improve clarity and preciseness of the explanations, and (most importantly) the experimental evaluation. Detailed strengths and weaknesses are outlined below.
Strengths
- The paper is well-motivated and relevant to the ML community
- Low-level speed optimizations are needed but overlooked in the community
- Reasonable choice of experimental conditions (focus on unbatched CPU evaluation, testing on a selection of 4 different tasks)
- Proposed techniques are sensible
Weaknesses (roughly in order of decreasing significance)
- Gains over Tensorflow performance is advertised in the intro, but only mentioned anecdotally in the experiments. Also, the Tensorflow implementation should be briefly explained to make clear where these gains come from.
- The experiments put much focus on speed performance over different batch sizes, but this is (1) not the focus of the paper (unbatched CPU operation is the focus), and (2) is little informative because the introduced methods (which do not benefit much from batching) are not compared against Tensorflow (which does benefit from batching).
- No ablation study is presented. The method description mentions informally how much speed-up is to be expected from different parts of the proposed method, but do not clarify how much this contributes to overall speed-gains.
- The description in 4.1 is rather hard to follow, even though the ideas behind it are relatively simple. An illustrative figure might be of help for readers.
- The method description in 4.1 lacks formal preciseness. For example, in 4.1. alpha, P, supp, are not defined (or in some cases introduced much after they are used first), and the “+4” in the beginning of page 5 appears out of nowhere.
- The proposed bit-packing is not well motivated. It promises to save half of the memory at a minor loss in precision (at least for the double-precision case), but it is unclear how much of a bottleneck this memory consumption is in the first place. In addition, it remains unclear to me why this is relevant particularly in the shared index case.
- While the topic is relevant for this conference, readers are likely not familiar with some of the concepts used in the paper, and a bit more explanation is needed. An example for this is “branch prediction”, which is a key concept; readers unfamiliar with this compiler concept will likely not understand the paper, and a brief explanation is needed. Another example is “loop-carry dependency”, a term that could be explained in a short footnote. A third example is FPGA, which is mentioned in the very last sentence without further explanation/justification.
- The introduction could be a bit more concrete on describing tasks where PWLs are relevant


Existing neural architecture search (NAS) methods explore a limited featuretransformation-only search space, ignoring other advanced feature operations such as feature self-calibration by attention and dynamic convolutions. This disables the NAS algorithms to discover more advanced network architectures. We address this limitation by additionally exploiting feature self-calibration operations, resulting in a heterogeneous search space. To solve the challenges of operation heterogeneity and significantly larger search space, we formulate a neural operator search (NOS) method. NOS presents a novel heterogeneous residual block for integrating the heterogeneous operations in a unified structure, and an attention guided search strategy for facilitating the search process over a vast space. Extensive experiments show that NOS can search novel cell architectures with highly competitive performance on the CIFAR and ImageNet benchmarks.

Summary:
Often in (neural architecture search) NAS papers on vision datasets, only feature transform operations (e.g. convolution, pooling, etc) are considered while operations like attention and dynamic convolution are left out. These operations have shown increasing performance improvements across many domains and datasets. This paper attempts to incorporate these operations into the search space of NAS.
Specifically they design a new cell (residual block in a resnet backbone) which contains these operations (traditionally used feature transforms, attention, and dynamic convolutions). This new cell contains two-tiers. The first tier separately computes the three kinds of operations and combines the results via elementwise summation to form an 'intermediate calibrated tensor'. This is then fed to the second tier where again the three kinds of operations are performed on them and they net output is again formed via elementwise summation.
For the search algorithm the authors use the bilevel optimization of DARTS. The one difference to aid in the search is that an 'attention-guided' transfer mechanism is used where a teacher network trained on the larger dataset (like ImageNet) is used to align the intermediate activations of the proxy network found during search, by adding a alignment loss function.
Results of search with this new search space on cifar10 and transfer to cifar100, imagenet are presented. It is found via manual ablation in resnet backbone that dilated dynamic convolutions dont help and are dropped from the search space.
The numerical results are near state-of-the-art although as is pervasive in the NAS field actual fair comparisons between methods are hard to get due to differences in search space, hardware space, stochasticity in training etc. But the authors do a best-attempt and have included random search and best and average performances so that is good.
Comments:
- The paper is generally well-written and easy to understand (thanks!)
- Experiments are generally thorough.
- The main novelty is the design of the cell such that attention and dynamic convolution operations can be incorporated. I was hoping that the results in terms of error would be much better due to the more expressive search space but they are not at the moment.
- I am curious how the attention guided alignment loss has been ablated. How do the numbers look without alignment loss keeping gpu training time constant? Basically I am trying to figure out how much is the contribution of the new search space vs. the addition of attention guided search. Do we have a negative result (an important one though) that attention and dynamic convolutions don't really help?


In this paper, we tackle the problem of detecting samples that are not drawn from the training distribution, i.e., out-of-distribution (OOD) samples, in classification. Many previous studies have attempted to solve this problem by regarding samples with low classification confidence as OOD examples using deep neural networks (DNNs). However, on difficult datasets or models with low classification ability, these methods incorrectly regard in-distribution samples close to the decision boundary as OOD samples. This problem arises because their approaches use only the features close to the output layer and disregard the uncertainty of the features. Therefore, we propose a method that extracts the uncertainties of features in each layer of DNNs using a reparameterization trick and combines them. In experiments, our method outperforms the existing methods by a large margin, achieving state-of-the-art detection performance on several datasets and classification models. For example, our method increases the AUROC score of prior work (83.8%) to 99.8% in DenseNet on the CIFAR-100 and Tiny-ImageNet datasets.

The paper considers the problem of out-of-distribution (OOD) sample detection while solving a classification task. The authors tackle the problem of OOD detection with exploiting uncertainty while passing a test sample through the neural network. They treat outputs of (some) layers in a NN as random Gaussian-distributed variables and measure uncertainty as variance of these Gaussians. Then when uncertainty is high, OOD is detected.
The overall idea behind the paper could be interesting, but its realisation in the current form is questionable.
The paper seems totally misusing the reparametrisation trick and stochastic outputs of layers in NNs. Eq. (2) is not the objective of variational inference that seems to be required for stochastic outputs and the reparametrisation trick as presented before the equation. The objective misses the KL-divergence term! Without it what would stop a neural net to set sigmas to 0 and forget about the stochasticity altogether? Not to mention that the current objective is not mathematically justified.
If there is no mix and error in eq. (2) and the networks were trained using this loss (and based on provided code they were using this loss), my wild guess of explaining why this may give best results in the experiments is that the models were trained for surprisingly small number of epochs. Therefore, a hypothesis would be that this small number of epochs did not allow the networks to switch sigmas to 0.
Until the authors can clarify and justify the objective, I will vote for rejection only based on this ground.
However, there are other issues in the paper as well. First of all, its clarity. It seems that the paper requires a lot of polishing.  The first paragraph of this review is based on my assumptions from the paper since I am not completely sure I understand it correctly. More about the clarity issues below
For strong evaluation, comparison with Malinin & Gales (2018) work seems to be important since it was the only work also using uncertainties for OOD detection in related work. Also related work section does not look like an exhaustive overview.
Some of the detailed comments:
1.	“In other words, in-distribution samples possess more features that convolutional filters react to than OOD samples” – first of all, this sentence is not easy to parse. Secondly, it is unclear, why this should be true. If OOD samples are still natural images, they would contain edges just the same as in-distribution samples. That is the power of deep learning enabling transfer learning, that low-level features are the transferable across different data and tasks. Therefore, the claim that “Therefore, the uncertainties of the features will be larger when the inputs are in-distribution samples” requires more elaboration and arguments
2.	The arguments of the next paragraph regarding uncertainty of deeper layers should be larger for OOD samples are not very convincing either.  It is either requires a definition what the authors mean here as uncertainty, or it is not necessarily true that absence of fixed regions for embeddings leads to higher uncertainty.
3.	3rd and 4th paragraphs in Introduction have too many repetitions of phrases between each other. Compare, e.g. the first sentences of the paragraphs or the last sentences.
4.	“One cause of the abovementioned problem is that their approaches” and similarly the next paragraph: “their approaches” stylistically sound wrong. It is appropriate in the previous paragraph since there is a link to “previous studies”. It seems that “these approaches” or “the existing approaches” would be a better choice for this and the next paragraph.
5.	“Each uncertainty is easily estimated after training the discriminative model by computing the mean and the variance of their features using a reparameterization trick” – conventional discriminative models do not estimate mean and variances of the features. The issue of estimating uncertainty is addressed by several special methods such as Bayesian variational methods used in the referred papers. Therefore, in order to use a reparametrisation trick one need to firstly choose a special class of models, which is not obvious from the text.
6.	“Moreover, UFEL is robust to hyperparameters such as the number of in-distribution classes and the validation dataset.” – the size of the validation dataset? In any case neither size of the validation dataset nor the validation dataset itself are not hyperparameters (should not be hyperparameters for out-of-distribution detection). The number of classes can hardly be called a hyperparameter also.
7.	“depends on the difference in the Dirichlet distribution of the categorical parameter <…> In our work, the distribution of the logit of the categorical parameters” – what is/are this/these categorical parameter(s)?
8.	“Further, they estimate the parameter of the Dirichlet distribution using a DNN and train the model with in-distribution and OOD datasets” – this sentence may mislead to impression that the proposed method does not need OOD dataset for training, which does not seem to be the case, since \lambda and \theta are trained based on OOD samples
9.	“because they will not be relevant to the classification accuracy” – who are they?
10.	“and \epsilon is the Gaussian noise” –> the standard Gaussian noise
11.	“where z^0 = x” – it seems this should be placed somewhere earlier when z^l is introduced since z^0 is not used in eq.(2) after which this text is placed
12.	It is unclear how \lambda^l and CNN \theta are learnt
13.	It is unclear how the values of features d(x) are used to detect OOD samples
14.	“comparison methods, and models” – not clear what models mean here
15.	Missing references to datasets in the main text. At least reference to Appendix A.2 is required
16.	“We used 5,000 validation images split from each training dataset and chose the parameter that can obtain” – which parameter?
17.	“All the hyperparameters of ODIN” – a reader does not know yet that ODIN is used for comparison
18.	“which consists of 100 OOD images from the test dataset and 1,000 images from the in-distribution validation set” – it is a bit confusing to call OOD dataset as a test dataset in this context
19.	“We tuned the parameters of the CNN in Equation 4 using 50 validation training images taken from the 100 validation images. The best parameters were chosen by validating the performance using the rest of 50 validation images.” – this is confusing. What parameters do the authors talk about in the second sentence if not the parameters of the CNN?
20.	“We used TNR at 95% TPR, AUROC, AUPR, and accuracy (ACC),” – Some elaboration is required, at least the reference to Appendix A.1. What is the changing threshold for AUROC and AUPR? Why AUPR-In and AUPR-Out are considered and only a single AUROC is considered. What is the positive class for AUROC?
23.	“For LeNet5, we increased the number of channels of the original LeNet5 to improve accuracy” – do the authors mean that they allowed RGB images as input rather than greyscale? If yes, this explicit explanation would be preferable
24.	“We inserted the reparameterization trick” – not the best word choice. Reparametrisation trick is a computational/implementation trick/method and it is hard to say that it can be inserted into a network. I believe what the authors mean is that they inserted mean/std outputs instead of point outputs. Conceptually, this means that the output of the corresponding layers is considered to be stochastic rather than deterministic.
Also, it is unclear when the authors say they insert it to the softmax layer. According to Section 3 the softmax layer is never considered to output means and stds.
25.	The numbers of epochs for training NNs are very small for LeNet and WideResNet in the experiments. Did the models manage to converge during this short training?
Minor:
1.	“These data were also used” -> “this data”


In this paper, we present a general framework for distilling expectations with respect to the Bayesian posterior distribution of a deep neural network, significantly extending prior work on a method known as "Bayesian Dark Knowledge." Our generalized framework applies to the case of classification models and takes as input the architecture of a "teacher" network, a general posterior expectation of interest, and the architecture of a "student" network. The distillation method performs an online compression of the selected posterior expectation using iteratively generated Monte Carlo samples from the parameter posterior of the teacher model. We further consider the problem of optimizing the student model architecture with respect to an accuracy-speed-storage trade-off. We present experimental results investigating multiple data sets, distillation targets, teacher model architectures, and approaches to searching for student model architectures. We establish the key result that distilling into a student model with an architecture that matches the teacher, as is done in Bayesian Dark Knowledge, can lead to sub-optimal performance. Lastly, we show that student architecture search methods can identify student models with significantly improved performance.

Contributions:
The paper considers the distillation of a Bayesian neural network as presented in [Balan et al. 2015]
The main contribution of the paper is the extension of [Balan et al. 2015] to apply to general posterior expectations instead of being restricted to predictions.
A second contribution of the paper is the finding that restricting the architecture of the student network to coincide with the teacher can lead to suboptimal performance and this can be mitigated by expanding the student's architecture using architecture search.
Originality/Significance:
I want to discuss the result regarding the generalization of the posterior expectation (section 3.1). To my knowledge this is novel, however, I am failing to see the importance of this result. The paper mentions two cases as motivations: to calculate the entropy and the variance of the marginal. The  problem with these two examples is that it is unclear why these are important and they are not used in the experiments anywhere. Their use should be motivated and the performance of the distillation should be properly evaluated in the experiments.
The result regarding architecture search is interesting, but it should be expanded and explained more to be a main contribution.
Clarity:
The paper generally understandable, and well written, but it could be better organized. It should expand on the motivation and the architecture search, since these are key components of the paper. The figures are not legible. Even fully zoomed in, they are difficult to read.
Overall assessment:
The paper has some interesting ideas, but it lacks motivation and significant results.
_______________________________________________________________________
Response to the rebuttal:
Thank you for the detailed reply.
> A primary motivation for generalising posterior expectations is to help quantify model uncertainty. Indeed, the expectation of the predictive entropy is an important quantity that is distinct from the entropy of the posterior predictive distribution. For instance, the difference between these two quantities is exactly the BALD score used in active learning [1]. ...
BALD would be problematic to use with this framework due to computational costs. The main benefit of distillation is the reduced computational cost at inference time. But training itself is still expensive. In BALD, the bulk of the computational cost is fitting the model after each new observation. Distillation does not provide a speedup here.
If the method indeed works well in an active learning setting, it would be interesting to see experiments showcasing this result.
> - The result regarding architecture search is interesting, but it should be expanded and explained more to be a main contribution.
I was hoping for more discussion/guidance on finding the right architecture or perhaps an algorithm that efficiently optimises the architecture. But I understand that this is more of a future work so I am not holding this against the paper.
I realise that my initial assessment was rather short so I decided to increase my rating and lower my confidence score. I think the paper is borderline, but I am slightly leaning towards rejection due to the insufficient motivation.


Graph Neural Networks (GNNs) for prediction tasks like node classification or edge prediction have received increasing attention in recent machine learning from graphically structured data. However, a large quantity of labeled graphs is difficult to obtain, which significantly limits the true success of GNNs. Although active learning has been widely studied for addressing label-sparse issues with other data types like text, images, etc., how to make it effective over graphs is an open question for research. In this paper, we present an investigation on active learning with GNNs for node classification tasks. Specifically, we propose a new method, which uses node feature propagation followed by K-Medoids clustering of the nodes for instance selection in active learning. With a theoretical bound analysis we justify the design choice of our approach. In our experiments on four benchmark datasets, the proposed method outperforms other representative baseline methods consistently and significantly.

The authors propose an interesting method to actively select samples using the embeddings learned from GNNs. The proposed method combines graph embeddings and clustering to intelligently select new node samples. Theoretical analysis is provided to support the effectiveness and experimental results shows that this method can outperform many other active learning methods.
This paper can be improved on the following aspects:
1.	The proposed method conducts clustering using node embeddings. Although these embeddings have encoded graph structure to some extent, I would suggest explicitly incorporating the graph structure in clustering or at least comparing to a baseline on that. The proposed method conducts embedding learning and clustering in two consecutive but separate steps. It would be interesting to see that the clustering can also leverage the graph information.
2.	It would be better to provide more details about network settings (some hyperparams have already been given in the paper), and more analysis would be helpful. For example, how the number of clusters affects the performance?
3.	Is it possible to create a scenario where there are more labeled data from one cluster but less data from another cluster? In this case, should we still take equal amount of samples from different clusters?


The detection of out of distribution samples for image classification has been widely researched. Safety critical applications, such as autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution. This paper adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detecting out of distribution pixels, which can localise the unusual objects. It further experimentally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and DeeplabV3+ architectures, as well as proposing a new metric for the task. The evaluation shows that the performance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts.

# Summary
This paper puts forward a study over out-of-distribution (OOD) detection for semantic segmentation. OOD detection is an active area research which has recently dealt mostly with image level classification. For semantic segmentation the same conclusions might not apply since decisions must be taken for each pixel individually. To this effect the authors propose here to study this task over a set of architectures (PSPNet, DeepLabV3+), outlier datasets (SUN, Indian Driving Dataset, synthetic images) and multiple methods for OOD detection from recent works on image classification.
A major difficulty in OOD works is the definition of a relevant OOD dataset and evaluation setup, and the authors propose here a novel setup for this task by adjusting the SUN and IDD datasets as OOD for Cityscapes.
The experimental part is thorough with multiple evaluation metrics and some qualitative examples and discussion.
# Rating
Although the paper studies a meaningful and interesting problem, I would reject the paper for the following reasons (more detailed arguments some lines below):
1) I find that the proposed dataset setups for OOD detection are not adequate for semantic segmentation and thus alter the reliability of the results.
2) Other than the evaluation (which has some faults), there is no proposed method addressing this task and its challenges.
3) The proposed MaxIoU metric proposed is not discussed and compared in more detail against the usual ones for this task.
# Strong points
- This work is dealing with a highly interesting and challenging problem. Indeed there has been few studies in this area and defining proper techniques and evaluation setups is challenging.
- The authors consider a wide array of methods for evaluation and test them across two architectures and multiple real and synthetic datasets.
# Weak points
- There are two real datasets considered for OOD evaluation, but I consider there are some flaws in their utilization for OOD detection. First, the SUN dataset is quite different from Cityscapes with a significant domain gap (at least in the visual appearance and distributions of the classes) between the two datasets. Upsampling the SUN images to 5x their size in order to make them compatible to Cityscapes should increase the artefacts even further making it easier to spot the gap between SUN and Cityscapes. This means that there is a risk of having the OOD detector acting merely as domain classifier spotting whenever the domain is different from the one use for training.
This argument applies to IDD, although to a smaller extent as the datasets are both automotive, but again there are strong visual differences between samples of the same class across the two datasets.
The authors argue that they somehow take this argument into consideration (Figure 2) and select only the non-ID classes to perform evaluation. However in both networks the scene information is mixed into the representation (via pyramid pooling in PSPNet  respectively via a trous convolutions with different dilations in DeepLabV3+). So again, instead of an OOD detector we can end up doing domain classification.
It would be useful to see how is the classification performance changing for ID classes, e.g. how is a model trained on Cityscapes scoring for cars and other ID objects in IDD comparing to a model that is trained on IDD for the same classes. A big difference between these scores would correspond to a significant domain gap and in correlation with the OOD performance we might be able to take some more conclusions on the matter.
Ahmed and Courville[i] propose an interesting discussion on this type of problems and propose focusing on semantic anomaly detection, i.e. detecting different classes from the same dataset, to make sure the setting has practical interest. They propose a very basic technique to detect OOD in previous classification setups showing the limitations of previous OOD methods. I encourage the authors to check the arguments stated in that work.
- In my opinion, the Fishyscapes work from Blum et al. is unfairly dismissed here by considering only a part of the benchmark for which animals from COCO and internet are inserted over Cityscapes images. The authors argue that this lack of realism of the inserted images make this dataset insufficient for OOD detection. However in the first version of their paper, Blum et al. propose a mix of Foggy Driving, Foggy Zurich, WildDash and Mapillary as dataset for ODD detection, which is similar with the setup proposed here. Furthermore, the latest version of Fishyscapes includes the Lost & Found dataset (mentioned in Fig. 1 here) which is recorded in similar conditions with Cityscapes with the addition of a few small outlier objects used as OOD. This is a relevant dataset and work and I would adjust the critics brought to their work here. That paper has the same objectives and endeavors as the current submission.
- Although there are some discussions and experiments on multiple techniques there is not technical contribution mitigating all the limitations of previous OOD methods on classification and the challenges of OOD detection in semantic segmentation. This would have had greatly helped the paper.
- I find that the MaxIoU metric considered here is not sufficiently discussed and analysed to show its utility and the additional perspective it brings when evaluating along with the usual metrics.
## Other less important weak points
- The choice of dataset in Figure 1 , i.e. Lost & Found, can be misleading. This dataset is not further mentioned and evaluated in the rest of the paper. The image could be replaced with a qualitative results from the rest of the evaluation.
# Suggestions for improving the paper:
1) The current evaluation setting could have some flaws. I would propose some sanity checks and look at the classification performances over other ID classes, as suggested in the previous section
2) Evaluate on a setting similar to Fishyscapes Lost and Found, in which the dataset does not change much, but there are some novel objects.
3) Include a trivial OOD baseline in the spirit of [i] to show the utility of the proposed datasets for this task by being robust to such baselines.
4) Consider extending the breadth of OOD methods with Deep Prior Networks[ii] which have been shown to perform well on Fishyscapes for OOD detection.
5) Add a qualitative example with OOD detection on Perlin noise images
# References
[i] F. Ahmed and A. Courville, Detecting semantic anomalies, arxiv 2019 https://arxiv.org/abs/1908.04388
[ii] A. Malinin and M. Gales, Predictive uncertainty estimation via prior networks, NeurIPS 2018


In this work, we address the task of natural image generation guided by a conditioning input. We introduce a new architecture called conditional invertible neural network (cINN). It combines the purely generative INN model with an unconstrained feed-forward network, which efficiently preprocesses the conditioning input into useful features. All parameters of a cINN are jointly optimized with a stable, maximum likelihood-based training procedure. Even though INNs and other normalizing flow models have received very little attention in the literature in contrast to GANs, we find that cINNs can achieve comparable quality, with some remarkable properties absent in cGANs, e.g. apparent immunity to mode collapse. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.

This paper proposes conditional Invertible Neural Networks (cINN), which introduces conditioning to conventional flow-based generative models. Conditioning is injected into the model via a conditional affine coupling block, which concatenates conditioning with the input to the scaling and shifting sub-networks in the coupling block. Other small modification are proposed to improve training stability at higher learning rates, including soft clamping of scaling coefficients, and Haar wavelet downsampling, which is proposed to replace the squeeze operation (pixel shuffle) that is often used in flow-based models. The invertibility of the cINN allows for style transfer by projecting images into the latent space, and then reconstructing the image with a different conditioning. The performance of the cINN is evaluated empirically on the task of colorization, where it is shown to outperform other techniques in terms of nearness to the true colours, as well as sample diversity.
Overall, I would tend to vote for accepting this paper. The base method for integrating conditioning into the flow is simple and intuitive, and additional modifications which allow for stable training at higher learning rates, such as soft clamping and Haar wavelet downsampling, appear to be very effective. Conditional models often lend themselves to a wide variety of useful applications, so I think this work could be of interest to many.
My primary concerns with this paper are related to the comparison of image colorization methods. Specifically:
1) I would like to see a comparison to Probabilistic Image Colorization (PIC) [1], which was mentioned in the related work section but not included in the comparison of colorization models. PIC has been shown to outperform VAE-MDN in terms of diversity, so it would be good to include it. Code is available online with pretrained ImageNet models (https://github.com/ameroyer/PIC), so it should not be difficult to add.
2) Pix2pix is known to have very bad sample diversity. A more useful comparison would be to evaluate one of the newer variants of Pix2pix that emphasizes sample diversity, such as BicycleGAN [2] or MSGAN [3]. Code is also available for each of these models (https://github.com/junyanz/BicycleGAN, https://github.com/HelenMao/MSGAN), although you would need to train models from scratch.
3) Pixel-wise metrics such as MSE are bad at measuring perceptual similarity. While these pixel-wise metrics are still useful for comparison to prior work, better metrics are available for evaluating image colorization. I would recommend the use of Learned Perceptual Image Patch Similarity (LPIPS) [4] in place of pixel-wise distance measures for evaluating image similarity and diversity.
Things to improve the paper that did not impact the score:
5) I was somewhat disappointed by how little attention was spent on the Haar wavelet downsampling method. It seems like a very neat idea, but it is only briefly explored in the ablation study. It would be nice to include a more in-depth study of how it compares to the conventional pixel shuffle downsampling, perhaps in terms of stability with learning rates, and final model performance.
6) There is some potentially related work on conditional adversarial generative flows [5] that could be added to the literature review if deemed relevant enough.
References:
[1] Amelie Royer, Alexander Kolesnikov, and Christoph H. Lampert. Probabilistic image colorization. In British Machine Vision Conference (BMVC), 2017.
[2] Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.
[3] Mao, Qi, et al. "Mode seeking generative adversarial networks for diverse image synthesis." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[4] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[5] Liu, Rui, et al. "Conditional Adversarial Generative Flow for Controllable Image Synthesis." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
### Post-Rebuttal Comments ###
Thank you for including the PIC results. I did not realize that they would take so long to run, but I think it important to include comparison with the current SOTA methods so that we have some reference. It is also nice to see the addition of the LPIPS metric. My overall opinion of the paper is not much changed from before, so I will retain the same score.


In this paper, we study the problem of constrained robust (min-max) optimization in a black-box setting, where the desired optimizer cannot access the gradients of the objective function but may query its values. We present a principled optimization framework, integrating a zeroth-order (ZO) gradient estimator with an alternating projected stochastic gradient descent-ascent method, where the former only requires a small number of function queries and the later needs just one-step descent/ascent update. We show that the proposed framework, referred to as ZO-Min-Max, has a sub-linear convergence rate under mild conditions and scales gracefully with problem size. From an application side, we explore a promising connection between black-box min-max optimization and black-box evasion and poisoning attacks in adversarial machine learning (ML). Our empirical evaluations on these use cases demonstrate the effectiveness of our approach and its scalability to dimensions that prohibit using recent black-box solvers.

The paper presents an algorithm for performing min-max optimisation without gradients and analyses its convergence. The algorithm is evaluated for the min-max problems that arise in the context of adversarial attacks. The presented algorithm is a natural application of a zeroth-order gradient estimator and the authors also prove that the algorithm has a sublinear convergence rate (in a specific sense).
Considering that the algorithm merely applies the zeroth-order gradient estimator to min-max problems, the algorithm itself only makes up a somewhat novel contribution. However, to the best of my knowledge, it has not been used in this context before and personally I find the algorithm quite appealing. In fact, due to its simplicity it is essentially something that anyone could implement from scratch.
Perhaps a more important contribution is that the authors provide a fairly extensive convergence analysis, which is an important tool in analysing the algorithm and its properties. Unfortunately, it is not trivial to understand the presented convergence results and their practical implications (if any). For instance, equation (10), which is arguably one of the key equations in the paper, contains variables zeta, nu and P1, all of which depend on a number of other variables in a fairly complicated manner. The expression in (10) also contains terms that do not depend on T and it is not obvious how large these terms might be in practice (in the event that the assumptions are at least approximately true in a local region). Even though I am somewhat sceptical to the practical relevance of this convergence analysis, I recognise that it is an interesting and fascinating achievement that the authors have managed to provide a convergence analysis of an algorithm which is based on black-box min-max optimisation.


Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unseen corruptions and perturbations encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers. We propose AUGMIX, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unseen corruptions. AUGMIX significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.

This paper proposes a method called AugMix, which is intended to improve model robustness to data distribution shift. AugMix appears fairly simple to implement. Several new images are created by augmenting an original image through chains of sequentially applied transformations (the "Aug" part of AugMix), then the augmented images are combined together, along with the original image, via a weighted sum (the "Mix" part of AugMix). Additionally, a Jensen-Shannon Divergence consistency loss is applied during training to encourage the model to make similar predictions for all augmented variations of a single image. This technique is shown to achieve state-of-the-art performance on standard robustness benchmarks without loss of clean test accuracy, and is also shown to improve calibration of model confidence estimates.
Overall, I would tend to vote for accepting this paper. The method is simple yet effective, the paper is very well written and easy to follow, and experiments are extensive and, for the most part, convincing.
Questions:
1) The one main concern I have with the training procedure is the amount of time the models were trained for. It is known that model trained with aggressive data augmentation schemes often require much longer training than normal in order to fully benefit from the stronger augmentations. For example, AutoAugment trains ImageNet models for 270 epochs [1], while CutMix trains for 300 epochs [2]. However, the ImageNet experiments in this paper claim to follow the procedure outlined in [3], which only trains for 90 epochs. This is reflected in the clean test accuracy, where AutoAugment only appears to provide a 0.3% gain over standard training, while we might expect 1.3% improvement (according to [2]). The AllConvNet and WideResNet in CIFAR-10 and CIFAR-100 experiments were also trained for only 100 epochs each, where 200 is more conventional. Again this shows in the reported numbers: on WideResNet for CIFAR-10, Mixup only has a 0.3% gain were as we might expect 1% improvement instead [4], and AutoAugment has 0.4% improvement, were as we might expect 1.3% gain if trained longer [1]. My question then is, how much does training time affect results? Do AugMix, and other techniques such as Mixup, CutMix, and AutoAugment, achieve better robustness when models are trained for longer, or do they become more brittle as training time is extended?
2) For the Jensen-Shannon divergence consistency, how much worse does it perform when using JS(Porig;Paugmix1) versus JS(Porig;Paugmix1;Paugmix2)? What might cause this behaviour?
3) Patch Gaussian is changed to Patch Uniform to avoid overlap with corruptions in ImageNet-C. How does Patch Uniform compare to Patch Gaussian in terms of performance for non-Gaussian noise corruptions?
4) How does AugMix perform as an augmentation technique in terms of clean test accuracy compared to other SOTA techniques? Is there a trade-off between clean test accuracy and robustness, or does AugMix improve performance in both domains? Can AugMix be combined with other augmentation techniques or does this destroy robustness properties?
Things to improve the paper that did not impact the score:
5) It would be nice if the best result in each column could be bolded in Tables 2-4.
References:
[1] Cubuk, Ekin D., Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. "Autoaugment: Learning augmentation policies from data." CVPR (2019).
[2] Yun, Sangdoo, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. "Cutmix: Regularization strategy to train strong classifiers with localizable features." ICCV (2019).
[3] Goyal, Priya, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. "Accurate, large minibatch sgd: Training imagenet in 1 hour." arXiv preprint arXiv:1706.02677 (2017).
[4] Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. "mixup: Beyond empirical risk minimization." ICLR (2018).


Deep autoregressive models are one of the most powerful models that exist today which achieve state-of-the-art bits per dim. However, they lie at a strict disadvantage when it comes to controlled sample generation compared to latent variable models. Latent variable models such as VAEs and normalizing flows allow meaningful semantic manipulations in latent space, which autoregressive models do not have. In this paper, we propose using Fisher scores as a method to extract embeddings from an autoregressive model to use for interpolation and show that our method provides more meaningful sample manipulation compared to alternate embeddings such as network activations.

Motivated by the observation that powerful deep autoregressive models such as PixelCNNs lack the ability to produce semantically meaningful latent embeddings and generate visually appealing interpolated images by latent representation manipulations, this paper proposes using Fisher scores projected to a reasonably low-dimensional space as latent embeddings for image manipulations. A decoder based on a CNN, a Conditional RealNVP, or a Conditional Pyramid PixelCNN is used to decode high-dimensional images from these projected Fisher score.  Experiments with different autoregressive and decoder architectures are conducted on MNIST and CelebA datasets are conducted.
Pros:
This paper is well-written overall and the method is clearly presented.
Cons:
1) It is well-known that the latent activations of deep autoregressive models don’t contain much semantically meaningful information. It is very obvious that either a CNN decoder, a conditional RealNVP decoder, or a conditional Pyramid PixelCNN decoder conditioned on projected Fisher scores will produce better images because the Fisher scores simply contain much more information about the images than the latent activations. When the $\alpha$ is small, the learned decoder will function similarly to the original pixelCNN, therefore, latent activations produce smaller FID scores than projected Fisher scores for small $\alpha$’s. These results are not surprising. Detailed explanations should be added here.
2) The comparisons to baselines are unfair. As mentioned in 1), it’s obvious that Fisher scores contain more information than latent activations for deep autoregressive models and are better suited for manipulations. Fair comparisons should be performed against other latent variable models such as flow models and VAEs with more interesting tasks, which will make the paper much stronger.
3) In Figure 3, how is the reconstruction error calculated? It’s squared error per pixel per image?
4) On pp. 8, for semantic manipulations, some quantitative evaluations will strengthen this part.
In summary, this paper proposes a novel method based on projected Fisher scores for performing semantically meaningful image manipulations under the framework of deep autoregressive models. However, the experiments are not well-designed and the results are unconvincing. I like the idea proposed in the paper and strongly encourage the authors to seriously address the raised questions regarding experiments and comparisons.
------------------
After Rebuttal:
I took back what I said. It's not that obvious that the "latent activations of deep autoregressive models don’t contain much semantically meaningful information". But the latent activations are indeed a weak baseline considering that PixelCNN is so powerful a generator. If the autoregressive generator is powerful enough, the latent activations can theoretically encode nothing.  I have spent a lot of time reviewing this paper and related papers, the technical explanation about the hidden activation calculation of PixelCNN  used in this paper is unclear and lacking (please use equations not just words).
Related paper:  The PixelVAE paper ( https://openreview.net/pdf?id=BJKYvt5lg ) explains that PixelCNN doesn't learn a good hidden representation for downstream tasks
Another paper combining VAE and PixelCNN also mentions this point:
ECML 2018: http://www.ecmlpkdd2018.org/wp-content/uploads/2018/09/455.pdf
Please also check the related arguments about PixcelCNN (and the "Unconditional Decoder" results) in Variational Lossy Autoencoder (https://arxiv.org/pdf/1611.02731.pdf )
As I mentioned in the response to the authors' rebuttal, training a separate powerful conditional generative model from some useful condition information (Fisher scores) is feasible to capture the global information in the condition, which is obvious to me. This separate powerful decoder has nothing to do with PixelCNN, which is the major reason that I vote reject.


Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.

This paper presents a small multiple choice reading comprehension dataset drawn from LSAT and GMAT exams.  I like the idea of using these standardized tests as benchmarks for machine reading, and I think this will be a valuable resource for the community.
The two major concerns I have with this paper are with its presentation and with the quality of the baselines.  These concerns leave me at a weak accept, instead of a strong accept, which I would be if these issues were fixed.
Presentation:
The main contribution here is in collecting a new dataset drawn from the LSAT and GMAT.  (I assume that is all that is used, though the text actually says "such as".  Or is this from practice exams and not the actual exams?)  The job of this paper is to convince me, a researcher focusing on reading comprehension, that I should use this dataset as a target of my research.  Out of 8 pages, however, at most 2 are devoted to actually describing this contribution and why it's a good target for reading comprehension.  Much of the interesting description of the phenomena in the dataset is relegated to the appendix, which I did not really look at because it far exceeds the page limit of an ICLR submission.  I have a lot of questions about this dataset that could have been answered in the main text had more of the paper been given to actually describing the data.  Such as:
- What are some actual examples of the different kinds of reasoning in table 2?
- How did you decide the proportions of questions listed in table 2?  Was that manual?
- Where did the questions actually come from?  Real exams?  Practice exams?  Which ones?
- What is the reasoning process that a person would actually use to answer some of these questions?  Preferably answered for several of the question types that you listed.  You have 8 pages; there's a lot of space that could be given to this.
- What kinds of things do you see in the distractors that make this hard?
I'm recommending a weak accept for this paper, but only because of my background knowledge about the LSAT and GMAT, and my prior understanding of reading comprehension and why this would be a really interesting dataset.  I think the paper itself could do a *much* better job convincing a reader about the dataset than what is done here.
What should be cut to make room for this?
I think far too much space is given to the discussion of easy vs. hard (~2.5 pages).  Given that the best performance is at ~54% on the full test set, there isn't a lot of need for this.  The argument about four flips of a 25% coin giving a 0.39% chance of guessing right every time is only true if the coin flips are independently drawn; training the same pretrained model on the same data with a different random seed is clearly not independent, so this argument rings hollow.  It's not really clear what to conclude from the easy vs. hard split, other than the models that you used to create it expectedly do well on the easy split and hard on the test split.  You'd want to have separate models that create the split than what you're evaluating them with, but even then, you're training them on the same data, so it's still not really clear what to conclude.  It's sufficient in a case like this to just evaluate as a baseline a model that shouldn't have access to enough information to solve the task, and measure its performance on the test set.  That would have taken a few sentences to describe, instead of 2.5 pages.
You could also recover a lot of space by evaluating fewer baselines.  Describing six baseline models and including all of their performance numbers takes up a lot of space, and we really don't need to know what all of those numbers are.  One or two, along with question-only and option-only baselines, would be plenty.  If you really want to include all of them, that's information that should go into the appendix, instead of something that's core to your paper's contribution, like describing the dataset that you're releasing.
Quality of baselines:
With a dataset this small, you definitely want to pretrain BERT / RoBERTa on a larger QA dataset before training on ReClor.  I would guess that performance would be significantly higher if you used a RACE-trained model as a starting point, instead of raw BERT / RoBERTa.  And if it doesn't end up helping, then you've made a stronger argument about how your dataset is different from what has come before, and a good target for future research.  What I would recommend for table 6:
- chance
- option-only
- question-only
- raw XLNet or RoBERTa
- XLNet or RoBERTa pretrained on RACE, or SQuAD, or similar
And I would pick either XLNet or RoBERTa based on which one did the best after pre-training.  That way, related to my first point, you can remove the description of all other models, and table 4, and free up a bunch of space for more interesting things.
As an aside, I think it's good that the dataset is relatively small, as it gives these over-parameterized models less opportunity to overfit, and it makes us come up with other ways of obtaining supervision.  But you've got to be sure to use the best baselines available, and with small datasets, that means more pretraining on other things.
EDIT Nov. 19, 2019: The authors' revision has satisfied my concerns.  I think that this is a good paper and it should be accepted.


Monte Carlo Tree Search (MCTS) algorithms have achieved great success on many challenging benchmarks (e.g., Computer Go). However, they generally require a large number of rollouts, making their applications costly. Furthermore, it is also extremely challenging to parallelize MCTS due to its inherent sequential nature: each rollout heavily relies on the statistics (e.g., node visitation counts) estimated from previous simulations to achieve an effective exploration-exploitation tradeoff. In spite of these difficulties, we develop an algorithm, WU-UCT, to effectively parallelize MCTS, which achieves linear speedup and exhibits negligible performance loss with an increasing number of workers. The key idea in WU-UCT is a set of statistics that we introduce to track the number of on-going yet incomplete simulation queries (named as unobserved samples). These statistics are used to modify the UCT tree policy in the selection steps in a principled manner to retain effective exploration-exploitation tradeoff when we parallelize the most time-consuming expansion and simulation steps. Experiments on a proprietary benchmark and the Atari Game benchmark demonstrate the linear speedup and the superior performance of WU-UCT comparing to existing techniques.

This paper introduces a new algorithm for parallelizing Monte-Carlo Tree Search (MCTS). Specifically, when expanding a new node in the search tree, the algorithm updates the parent nodes’ statistics of the visit counts but not their values; it is only when the expansion and simulation steps are complete that the values are updated as well. This has the effect of shrinking the UCT exploration term, and making other workers less likely to explore that part of the tree even before the simulation is complete. This algorithm is evaluated in two domains, a mobile game called “Joy City” as well as on Atari. The proposed algorithm results in large speedups compared to serial MCTS with seemingly little impact in performance, and also results in higher scores on Atari than existing parallelization methods.
Scaling up algorithms like MCTS is an important aspect of machine learning research. While significant effort has been made by the RL community to scale up distributed model-free algorithms, less effort has been made for model-based algorithms, so it is exciting to see that emphasis here. Overall I thought the main ideas in paper were clear, the proposed method for how to effectively parallelize MCTS was compelling, and the experimental results were impressive. Thus, I tend to lean towards accept. However, there were three aspects of the paper that I thought could be improved. (1) It was unclear to me how much the parallelization method differs from previous approaches (called “TreeP” in the paper) which adjust both the visit counts and the value estimate. (2) The paper is missing experiments showing the decrease in performance compared to a serial version of the algorithm. (3) The paper did not always provide enough detail and in some cases used confusing terminology. If these three things can be addressed then I would be willing to increase my score.
Note that while I am quite familiar with MCTS, I am less familiar with methods for parallelizing it, though based on a cursory Google Scholar search it seems that the paper is thorough in discussing related approaches.
1. When performing TreeP, does the traversed node also get an increased visit count (in addition to the loss which is added to the value estimate)? In particular, [1] and [2] adjust both the visit counts and the values, which makes them quite similar to the present method (which just adjusts visit counts). It’s not clear from the appendix whether TreeP means that just the values are adjusted, or both the values and nodes. If it is the former, then I would like to see experiments done where TreeP adjusts the visit counts as well, to be more consistent with prior work. (Relatedly, I thought the baselines could be described in significantly more detail than they currently are—-pseudocode would in the appendix would be great!)
2. I appreciate the discussion in Section 4 of how much one would expect the proposed parallelization method to suffer compared to perfect parallelization. However, this argument would be much more convincing if there were experiments to back it up: I want to know empirically how much worse the parallel version of MCTS does in comparison to the serial version of MCTS, controlling for the same number of simulations.
3. While the main ideas in the paper were clear, I thought certain descriptions/terminology were confusing and that some details were missing. Here are some specifics that I would like to see addressed, roughly in order of importance:
- I strongly recommend that the authors choose a different name for their algorithm than P-UCT, which is almost identical (and pronounced the same) as PUCT, which is a frequently used MCTS exploration strategy that incorporates prior knowledge (see e.g. [1] and [2]). P-UCT is also not that descriptive, given that there are other existing algorithms for parallelizing MCTS.
- Generally speaking, it was not clear to me for all the experiments whether they were run for a fixed amount of wallclock time or a fixed number of simulations, and what the fixed values were in either of those cases. The fact that these details were missing made it somewhat more difficult for me to evaluate the experiments. I would appreciate if this could be clarified in the main text for all the experiments.
- The “master-slave” phrasing is a bit jarring due to the association with slavery. I’d recommend using a more inclusive set of terms like “master-worker” or “manager-worker” instead (this shouldn’t be too much to change, since “worker” is actually used in several places throughout the paper already).
- Figure 7c-d: What are game steps? Is this the number of steps taken to pass the level? Why not indicate pass rate instead, which seems to be the main quantity of interest?
- Page 9: are these p-values adjusted for multiple comparisons? If not, please perform this adjustment and update the results in the text. Either way, please also report in the text what adjustment method is used.

- Figure 7: 3D bar charts tend to be hard to interpret (and in some cases can be visually misleading). I’d recommend turning these into heatmaps with a visually uniform colormap instead.
- Page 1, bottom: the first time I read through the paper I did not know what a “user pass-rate” was (until I got to the experiments part of the paper which actually explained this term). I would recommend phrasing this in clearer way, such as “estimating the rate at which users pass levels of the mobile game…”
- One suggestion just to improve the readability of the paper for readers who are not as familiar with MCTS is to reduce the number of technical terms in the first paragraph of the introduction. Readers unfamiliar with MCTS may not know what the expansion/simulation/rollout steps are, or why it’s necessary to keep the correct statistics of the search tree. I would recommend explaining the problem with parallelizing MCTS without using these specific terms, until they are later introduced when MCTS is explained.

- Page 2: states correspond to nodes, so why introduce additional notation (n) to refer to nodes? It would be easier to follow if the same variable (s) was used for both.
Some additional comments:
- Section 5: I’m not sure it’s necessary to explain so much of the detail of the user-pass rate prediction system in the main text. It’s neat that comparing the results of different search budgets of MCTS allow predicting user behavior, but this seems to be a secondary point besides the main point of the paper (which is demonstrating that the proposed parallelization method is effective). I think the right part of Figure 5, as well as Table 1 and Figure 6, could probably go in the supplemental material. As someone with a background in cognitive modeling, I think these results are interesting, but that they are not the main focus of the paper. I was actually confused during my first read through as it was unclear to me initially why the focus had shifted from demonstrating that parallel MCTS works to
- The authors may be interested in [3], which also uses a form of tree search to model human decisions in a game.
- Page 9: the citation to [2] does not seem appropriate here since AlphaGo Zero did not use a pretrained search policy, I think [1] would be correct instead.
[1] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., ... & Dieleman, S. (2016). Mastering the game of Go with deep neural networks and tree search. nature, 529(7587), 484.
[2] Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., ... & Chen, Y. (2017). Mastering the game of go without human knowledge. Nature, 550(7676), 354.
[3] van Opheusden, B., Bnaya, Z., Galbiati, G., & Ma, W. J. (2016, June). Do people think like computers?. In International conference on computers and games (pp. 212-224). Springer, Cham.


Bayesian Neural Networks (BNNs) provides a mathematically grounded framework to quantify uncertainty. However BNNs are computationally inefficient, thus are generally not employed on complicated machine learning tasks. Deep Ensembles were introduced as a Bootstrap inspired frequentist approach to the community, as an alternative to BNN's. Ensembles of deterministic and stochastic networks are a good uncertainty estimator in various applications (Although, they are criticized for not being Bayesian). We show Ensembles of deterministic and stochastic Neural Networks can indeed be cast as an approximate Bayesian inference. Deep Ensembles have another weakness of having high space complexity, we provide an alternative to it by modifying the original Bayes by Backprop (BBB) algorithm to learn more general concrete mixture distributions over weights. We show our methods and its variants can give better uncertainty estimates at a significantly lower parametric overhead than Deep Ensembles. We validate our hypothesis through experiments like non-linear regression, predictive uncertainty estimation, detecting adversarial images and exploration-exploitation trade-off in reinforcement learning.

Summary: This paper proposes to use either relaxed mixture distributions or relaxed mixtures of matrix Gaussians as the approximate posterior for Bayesian neural networks. Naturally, taking the mixture variance to zero allows a stretched interpretation of ensembling to become Bayesian as well. Experiments are performed on small neural networks on regression tasks, as well as uncertainty estimation experiments on deep networks. Finally, a downstream task of bandit optimization (the mushroom experiment) is performed.
post-rebuttal: I'd like to sincerely thank the authors of the paper for their enlightening discussions about their work. I'm inclined to think that this paper has a future, and definitely encourage the authors to resubmit - taking into account the reviews here. In particular, it would be greatly helpful if they cleaned up some of the writing and explanation, as our discussion points to.
However, my stance on the codebase remains mostly unchanged - while a notebook ended up being released with their method, it was too close to the deadline and in a bit too rough of shape to really be able to poke through their implementation.
From a quick pass, their implementation seems to be correct - although rather than attempting to base their code on the old torch/lua implemenations, I'd still suggest using a pre-written version. However, without the completely trained numbers, I cannot in good conscience vote to accept the paper.
Tldr: I vote to reject this paper for several reasons. The interpretation of ensembling as variational inference is both flawed and well known. Mixture distributions have been previously proposed for variational inference. There are significant enough weaknesses in the empirical results that make me question the authors’ own implementations. I will increase my score if these issues are resolved.
Originality:
1)	It is well known (and immediately obvious) that one can interpret ensembles as a Bayesian method – with a mixture of delta functions around the independently trained models. Furthermore, the Bayesian bootstrap (Rubin, 1983) is a Bayesian interpretation of the bootstrap (identical under many conditions), while re-weighted ensembles are Bayesian models (Newton, 1995; Newton, 2018) that converge to the true posterior under many conditions – intriguingly, this version of re-weighting could potentially have good uncertainty quantification properties. For the specific case of stacking (which is closely related to ensembling), there are interpretations of stacking as Bayesian model combination (Yao et al, 2018, Clyde & Iversen, 2013). As a result, this view of ensembling as a Bayesian method is not novel.
2)	Additionally, the derivation of dropout as Bayesian inference (which relies on a similar interpretation on taking the variance parameter in the Gaussian to zero) is known to be flawed (Hron et al, 2018); this flaw is directly applicable to the interpretation of ensembling as variational inference as described in Section 3 of this paper. The issue is fundamental: when taking the variance parameter to zero, the supports of the variational distribution and the true distributions have a mismatch, causing the KL divergence to become infinite. As a result, one cannot minimize the objective (as it is infinite by design).
Clarity:
1)	The proposed methods are clearly explained in the setting under which one might be able to re-implement them, particularly in the description of the approximate posteriors.
post rebuttal: to my knowledge, this was never really addressed. In particular, "why do we want to interpet ensembling in a Bayesian manner?" is still an open question to me.
2)	Why is that we should expect (approximate) Bayesian procedures to be better calibrated than frequentist ones? It may be the case that if the integrals are accurately estimated (e.g. we have reached the true posterior and the model used for the data is in fact accurate), then we should expect Bayesian procedures to be well calibrated. However, this does not seem to be the case. A reference or argument for why (even in the model incorrect & approximate inference cases) in the introduction would greatly enhance the strength of the paper.
post rebuttal: I think that this issue was addressed in a reasonable manner through the rebuttal, but will require a bit of rewriting.
3)	From a quick pass through Appendix A.2, the derivation is quite unstructured and difficult to follow. It is tough to see exactly what is being meant by the probabilities in the first line.
Additionally, the phrasing of “We further lower bound…” seems to suggest that the bound being used is not the well known bound of Durrieu et al, 2012, but a looser bound. If so, what is the approximation accuracy of this bound in relation to the other bounds compared against in this setting?
4)	In sum, three different methods are proposed: concrete mixtures of Gaussians, concrete ensembles, and concrete mixtures of matrix variate Gaussians. Throughout the experiments, it seems like __one__ of these methods always wins on the task at hand. However, it does not always seem to be the same method. Is there a recommendation for practitioners as to which method to use – or is this simply task dependent? Specifically, is there an understanding as to why each method performs vastly differently on each task?
A priori, I would expect the concrete mixture of matrix variate Gaussians to always perform best because it does seem to be able to model multivariate posteriors the best. However, it seems to perform worse on several of the predictive uncertainty tasks.
5)	For the uncertainty experiments, it would be nice to have more qualitative numbers – for example the expected calibration error – rather than just the plots on in and out of distribution example. After all, not only do we want to be able to recognize out of distribution examples, but we want to be able to trust the probabilities that are output.
6)	In Figure 4, thank you for providing the adversarial attacks used to fool the networks – the step size parameter doesn’t give a great intuition, and the images are welcome and useful.
Significance:
post rebuttal: I think that this issue was satisfactorily addressed.
1)	Mixture distributions as the variational family have already been proposed – at least once – see Miller et al, 2017 for an example that additionally uses the reparameterization trick. There, updates to the posterior are performed via selectively adding mixture components when necessary. Given that they also run experiments on neural networks, it would be nice to see a comparison to that method.
Quality:
1)	I am very concerned by the performance of the ResNet20 models trained on CIFAR10 in the paper. The reported results for deep ensembles (an ensemble of _three_ independently trained models) in Appendix A are 85.23% accuracy. By comparison, the number in the original paper (He et al, 2015) is 91.25% for a _single_ model. Furthermore, the first link to a PyTorch repo from a Google search (https://github.com/akamaster/pytorch_resnet_cifar10) comes up with a re-implementation that gets 91.63% (again for a single model).
__This issue alone is enough for me to vote to reject the paper until the authors’ implementations are fixed and run with deep ensembled ResNets that get similar accuracy. I hope to see the implementation issues fixed in the rebuttal period. __
2)	Furthermore, in Appendix 2, I see a potential issue in using the KL divergence and would like the authors to clarify.
If the secondary lower bound is being used to approximate the KL divergence between the mixture distributions, it becomes very unclear if what is being optimized is in fact a __lower bound__ on the log probability. After all, the standard ELBO is written as:
	Log p(y) \geq ELBO := E_q(\log p(y | \theta)) – KL(q(\theta) || p(\theta))
Implying that another lower bound on the KL would be greater than the ELBO and thus not necessarily a lower bound on the log marginal likelihood.
3)	Bottom of page 7: “The posterior is used only for fully connected layers…” Why is this the case? Alternative Bayesian methods can be used on the full network.
Minor Comments:
-	Please attempt to use the math_commands.tex in the ICLR style file to write math. This makes the math standardized throughout.
-	Figure 1 is especially unclear and the legends and captions should be made considerably larger. Zooming in at 800% seems to be necessary to even be able to make out which method is which. Consider, placing all methods on a single plot and then coloring/dashing them separately. Additionally, please compare to HMC on this task (and an equivalent RBF GP) to show the uncertainties from the “gold standard” methods.
-	Please additionally make the figure legends larger for the rest of the figures.
-	What is the meaning of time in Figure 2? I assume that it means training time, but it is not especially clear from the caption. With that being said, I think that it is quite interesting that the marginals do seem to look multi-modal, even at the end of training.
-	Line before Eq 7: please use \citet if possible to cite Gupta & Nagar, rather than stating the clunky Gupta et al. \citep{Gupta & Nagar}.
-	Eq 2: please do not use \hspace{-…} to save space in equations, so that the equation does not overwrite lines before.
-	Page 2: “Deep Ensembles have lacked theoretical support …” In order to make this claim, you must first explain why one ought to be Bayesian in the first place. See above for the history of interpreting ensembling from a Bayesian perspective. There is no a priori reason why one would not just wish to have frequentist justifications of ensembling (and potentially even calibration).
-	Please do not capitalize Ensembling or Variational Inference throughout.
References:
Clyde & Iversen, Bayesian model averaging in the M-open framework. In Bayesian Theory and Applications, 2013. DOI:10.1093/acprof:oso/9780199695607.003.0024
He, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition, CVPR, 2016. https://arxiv.org/abs/1512.03385
Hron, Matthews & Ghahramani, Variational Bayesian dropout: pitfalls and fixes, ICML, 2018. https://arxiv.org/abs/1807.01969
Miller, Foti & Adams, Variational Boosting: Iteratively Refining Posterior Approximations, ICML, 2017. https://arxiv.org/abs/1611.06585
Newton & Raftery, Approximate Bayesian Inference with the Weighted Likelihood Bootstrap. JRSS:B, 1994. https://www.jstor.org/stable/2346025?seq=1#metadata_info_tab_contents
Newton, Polson & Xu, Weighted Bayesian Bootstrap for Scalable Bayes, 2018; https://arxiv.org/abs/1803.04559
Rubin, The Bayesian Bootstrap, 1981. Annals of Statistics. https://projecteuclid.org/euclid.aos/1176345338
Yao, Vehtari, Simpson, & Gelman, Using Stacking to Average Bayesian Predictive Distributions, Bayesian Analysis, 2018; https://projecteuclid.org/euclid.ba/1516093227


A common belief in the machine learning community is that using adaptive gradient methods hurts generalization. We re-examine this belief both theoretically and experimentally, in light of insights and trends from recent years. We revisit some previous oft-cited experiments and theoretical accounts in more depth, and provide a new set of experiments in larger-scale, state-of-the-art settings. We conclude that with proper tuning, the improved training performance of adaptive optimizers does not in general carry an overfitting penalty, especially in contemporary deep learning. Finally, we synthesize a "user's guide" to adaptive optimizers, including some proposed modifications to AdaGrad to mitigate some of its empirical shortcomings.

This paper revisited the a common belief that adaptive gradient methods hurts generalization performances. The authors re-examine this in more depth and provide a new set of experiments in larger-scale, state-of-the-art settings. The authors claimed that with proper tuning, the performance of adaptive optimizers can mitigate the gap  with non-adaptive methods.
- It is great to have someone revisit and challenge the conventional ideas in the community. However, I did not find much insightful information in this paper. As the author mentioned, there are many recent works focusing on further improving the empirical generalization performances of adaptive gradient methods. The main aspects mentioned in the paper, \epsilon tuning, learning rate warmup and decaying schedule, are not something new and many of which are mentioned or used in the recent advances. This makes the contribution of this paper look like combining all the tricks together. The authors might want to carefully comment the differences with the recent advances
[1] Liu, Liyuan, et al. "On the variance of the adaptive learning rate and beyond." arXiv preprint arXiv:1908.03265 (2019).
[2] Loshchilov, Ilya, and Frank Hutter. "Decoupled weight decay regularization." ICLR 2019.
[3] Zaheer, Manzil, et al. "Adaptive methods for nonconvex optimization." Advances in Neural Information Processing Systems. 2018.
[4] Chen, Jinghui, and Quanquan Gu. "Closing the generalization gap of adaptive gradient methods in training deep neural networks." arXiv preprint arXiv:1806.06763 (2018).
[5] Luo, Liangchen, et al. "Adaptive gradient methods with dynamic bound of learning rate." arXiv preprint arXiv:1902.09843 (2019).
- In terms of tuning \epsilon, the authors mentioned that default setting in Tensorflow is 0.1 for Adagrad which is too high. However, most papers regarding adaptive gradient method usually set \epsilon as 10^-8. Pytorch set default value as 10^-10. In fact, Yogi paper mentioned above gives some different conclusions. In their experiments, they found that setting \epsilon to be a bit larger like 10^-3 give better results compared with 10^-8. I wonder if the authors examine the reasons for different conclusions here?

- at the end of page 6, missing reference for histogram


Single Image Super Resolution (SISR) has significantly improved with Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), often achieving order of magnitude better pixelwise accuracies (distortions) and state-of-the-art perceptual accuracy. Due to the stochastic nature of GAN reconstruction and the ill-posed nature of the problem, perceptual accuracy tends to correlate inversely with pixelwise accuracy which is especially detrimental to SISR, where preservation of original content is an objective. GAN stochastics can be guided by intermediate loss functions such as the VGG featurewise loss, but these features are typically derived from biased pre-trained networks. Similarly, measurements of perceptual quality such as the human Mean Opinion Score (MOS) and no-reference measures have issues with pre-trained bias. The spatial relationships between pixel values can be measured without bias using the Grey Level Co-occurence Matrix (GLCM), which was found to match the cardinality and comparative value of the MOS while reducing subjectivity and automating the analytical process. In this work, the GLCM is also directly used as a loss function to guide the generation of perceptually accurate images based on spatial collocation of pixel values. We compare GLCM based loss against scenarios where (1) no intermediate guiding loss function, and (2) the VGG feature function are used. Experimental validation is carried on X-ray images of rock samples, characterised by significant number of high frequency texture features. We find GLCM-based loss to result in images with higher pixelwise accuracy and better perceptual scores.

The paper considers the problem of generating a high-resolution image from a low-resolution one. The paper introduces the Grey Level Co-occurrence Matrix Method (GLCM) for evaluating the performance of super-resolution techniques and as an auxiliary loss function for training neural networks to perform well for super-resolution. The GLCM was originally introduced in a 1973 paper and has been used in a few papers in the computer vision community. The paper trains and validates a super-resolution GAN (SRGAN)  and a super-resolution CNN (SRCNN) on the DeepRock-SR dataset. Specifically, for the SRGAN, the paper uses the EDSRGAN network trained on loss function particular to the paper: The loss function consists of the addition of the L1 pixel-wise loss plus the VGG19 perceptual objective plus the proposed GLCM loss.
The paper finds that SRCNN outperforms SRGAN in terms of the PSNR metric, but SRGAN performs better in terms of the spatial texture similarity.
Next, the paper shows that when trained with the mean L1-GMCM loss function, SRGAN performs best.
In summary, the paper proposes to use the GMCM loss for training and evaluation of the super-resolution methods. However, this metric is well known in the computer-vision community along with many others. Also, the idea to use this metric in training is only evaluated for one network (albeit a very sensible one) and only for one dataset (the DeepRock one). Since the novelty provided by the paper is small, I cannot recommend the acceptance of the paper.


Partial multi-label learning (PML), which tackles the problem of learning multilabel prediction models from instances with overcomplete noisy annotations, has recently started gaining attention from the research community. In this paper, we propose a novel adversarial learning model, PML-GAN, under a generalized encoder-decoder framework for partial multi-label learning. The PML-GAN model uses a disambiguation network to identify noisy labels and uses a multi-label prediction network to map the training instances to the disambiguated label vectors, while deploying a generative adversarial network as an inverse mapping from label vectors to data samples in the input feature space. The learning of the overall model corresponds to a minimax adversarial game, which enhances the correspondence of input features with the output labels. Extensive experiments are conducted on multiple datasets, while the proposed model demonstrates the state-of-the-art performance for partial multi-label learning.

This paper proposed a new method for partial multi-label learning, based on the idea of generative adversarial networks. The partial multi-label learning is the problem that one instance is associated with several ground truth labels simultaneously, but we are given a superset of the ground truth labels for training. To solve the problem, the paper proposes a learning model composing of four deep neural networks, two of them for prediction the ground truth labels (the prediction network, and the disambiguation network), one of them to generate instances based on the disambiguated labels, and one discriminative network to discriminated generated instance versus true instances. The four network are concatenated, and the paper proposed to optimize the sum of generation loss, the classification loss, and the adversarial loss. Theoretical results like the theories in the original GAN paper are also given. Finally, the paper does thorough empirical studies compared the proposed method with four other baselines and three variants on fourteen datasets with various settings.
The paper is the first paper to proposes a GAN-style algorithm for the partial multi-label learning problem. The empirical results also validate its superior performance compared to other baselines on the data sets used. On the other hand, the paper fails to give enough intuition on why the GAN can naturally be used to solve the partial multi-label learning problem, and why the proposed method can perform better than other baselines. There is also some inaccuracy in discussing related work. Overall, given the contribution to introduce GAN to partial multi-label learning problem and the thorough empirical studies, I think this paper can be weakly accepted.
Main arguments
The paper argues that existing methods may not be good on this problem is due to the fact that they generally learn a “confidence score” for every candidate labels, while the learned “confidence score” may not be accurate enough, and the errors may accumulate through these process. I agree with the paper on this aspect. However, in the paper, it also learns something similar to the “confidence” score in the disambiguation network \tilde D, so why the proposed method is better than previous baselines? I guess the reason is due to the adversarial training part in the latter steps. But the paper fails to give clear intuition why the adversarial training part can lead to better performance.
I am also wondering will the proposed model leads to a trivial solution. For example, one possible solution may be that both the prediction network F and the disambiguation network \tilde D get the same solution: the given partial label set. In this way, the classification error can always be zero, and we can train the generation network to give a ground truth instance x, given that the generation network G is strong enough. So I also guess the discrimination network is the key part for the success of the proposed method empirically. So I am curious if we only use the adversarial loss without training the prediction network F, what the results will be. In the Ablation Study Sec. 5.3, the paper does not compare the results optimizing only the adversarial loss. I believe the results of such a study can help us understand what the key part of the proposed method is.
In the related work part, the arguments on “weak label learning” cannot be used for partial multi-label learning is not accurate. In fact, “weak label learning” studies the problem when positive labels are missing, and partial multi-label learning studies the problem when negative labels are missing. So in general, the methods for “weak label learning” can be used for “partial multi-label learning” if we exchange the role of positive labels and negative labels in both problems. So why we need to study partial multi-label learning? I think the reason is that, generally, in multi-label learning, the number of negative labels will be much larger than the number of positive labels. So “partial multi-label” is actually more strong supervision information than “weak labels” because positive labels are more important. Such arguments and related empirical studies (when the number of negative labels is much larger than positive labels) are suggested to be added into the further version of the paper.
Although the paper has some deficiencies, it does a good job of introducing GAN into partial multi-label learning, and it is also the first paper to use the powerful deep neural networks into this problem, which may trigger some interesting studies given the booming of neural networks these days. And thorough empirical studies are done by comparing to not only baselines but also different loss parts of the proposal it owes. It is worth reading especially it may have an impact on further studies.
---------------------------------------
I am satisfied with the rebuttal and tend to increase my score. Although the paper is not perfect, it does a good job of introducing GAN into partial multi-label learning, and it is also the first paper (thus it is novel) to use the powerful deep neural networks into this problem, which may trigger some interesting studies given the booming of neural networks these days.
Actually, I think partial multi-label learning problem is difficult and it may not possible to have perfect solution nowadays. Partial multi-class learning may be easier since you know only one label is true, but partial multi-label learning is difficult since you do not know how many true labels there are. It makes no point to require a perfect solution for such a dirty problem.


We present a graph neural network assisted Monte Carlo Tree Search approach for the classical traveling salesman problem (TSP). We adopt a greedy algorithm framework to construct the optimal solution to TSP by adding the nodes successively. A graph neural network (GNN) is trained to capture the local and global graph structure and give the prior probability of selecting each vertex every step. The prior probability provides a heuristic for MCTS, and the MCTS output is an improved probability for selecting the successive vertex, as it is the feedback information by fusing the prior with the scouting procedure. Experimental results on TSP up to 100 nodes demonstrate that the proposed method obtains shorter tours than other learning-based methods.

EDIT: After the authors response and update of the contributions to indicate that the main contribution of the paper is the application of GNNs and MCTS to the TSP (rather than the original claims that that the model architecture and search approach were novel contributions), I increased my score from Weak Reject to Weak Accept. However, given that the paper is now more focused on solving the TSP, and I am not an expert on that specifically I had to reduce my experience assessment, as while I am more confident now that the paper is technically correct, it is harder for me to judge if the paper should be accepted in terms of empirical strength since I am not familiar with TSP baselines.
The authors propose an MCTS-based learned approach using Graph Neural Networks to solve the traveling salesman problem (TSP) agents.
The authors write the TSP as an MDP where the state consists of the nodes visited by the agent and the last node visited by the agent, the action consists of selecting the next node to visit, and the reward at each step is the negative cost of the travel between the last node and the next node.
The learned part of the model uses a “static-edge graph neural network” (SE-GNN). This network allows to access the full graph context, including edge features, to make node predictions. This is listed as the first paper contribution. At train time, this network is trained to predict the probability of each unvisited node to be next in the optimal path. This is trained via supervised learning using optimal paths precomputed with state of the art TSP solvers.
At test time, they use MCTS with a variant of PUCT, where the pre-trained SE-GNN is used as the prior policy, and there is a selection strategy during search that balances the prior probability, and the Q values estimated by MCTS, using max based updates (e.g. during back up new Q estimates replace old estimates if and only if the are larger than the previous ones). This is listed as the second paper contribution. Authors show that the approach beats other learned solvers in the TSP problem by a large margin in terms of optimality gap.
While I think the work is interesting, I am not sure that what the authors cite as main contributions of the paper are truly the main contributions. In my opinion the main contribution would be the state of the art performance at solving the TSP using learned methods. I cannot, however, recommend acceptance due to the following reasons.
With respect to the first claim “SE-GNN that has access to the full graph context and generates the prior probability of each vertex”, there are already many models that allow to condition on edge features, including InteractionNetworks, RelationNetworks and GraphNetworks. This paper has a good overview of this family of methods and most of them allow to access the full graph context too (https://arxiv.org/abs/1806.01261). Most of these models are very well known and are in principle more expressive than the one proposed in this paper, and allow generalization to different graph sizes, so the motivation for introducing a new model is not very clear, specially if these baselines are not compared.
With respect to the MCTS contribution at test time, it seems that the changes made to the algorithm compared to AlphaGo, are very specific to the TSP and there is not much discussion about which other sort of problems may benefit from the same modifications, so it is hard to evaluate its value as a standalone contribution independent from the TSP.
On the basis of state of the art performance at solving the TSP using learned methods:
* The model requires access to a dataset with optimal solutions to train it, and I doubt it can solve the problems faster than Gurobi in terms of wall time. For this result to be more interesting, the authors should be able to show that the model can generalize to larger problems (where the combinatorial complexity may start making approaches like Gurobi struggle). However it is not clear if the model can generalize to larger graphs.
* Beyond that I am not an expert on TSP specifically, and I don’t know the TSP literature, so I cannot give a strong recommendation.
There are some additional papers that may be relevant to this line of work:
* (MIP, NeurIPS 2019) Learning to branch in MIP problems using similar technique pretraining a GNN and use it to guide a solver at test time (no MCTS though) (https://arxiv.org/abs/1906.01629)
* (SAT, SAT Conference 2019) Learning to predict unsat cores (similar to the previous one but for SAT problems) (https://arxiv.org/abs/1903.04671)
* (Structural construction, ICML 2019) Building graphs by choosing actions over the edges of a graph solving the full RL problem end to end, and also integrating MCTS with learned prior both at train time and test time (together and independently) (http://proceedings.mlr.press/v97/bapst19a/bapst19a.pdf)
Some additional typos/ feedback:
* It would be good to have a pure MCTS baseline with not learned prior as an additional ablation (e.g. taking the SE-GNN prior out of the picture).
* In the “Selection Strategy” paragraph, the action is said to be picked as argmax(Q + U), where U is proportional to the prior for each action. However, Q is said to be initialized to infinite. This would mean that at the beginning of search all actions will be tied at infinite value, and my default assumption would be that in these conditions an action is chosen uniformly at random. I suspect what happens in this case is that the action with the highest prior is picked to break the tie at infinite, however if this is the case this should be indicated in the math.
* In the “Expansion Strategy” paragraph, the Q values are said to be initialized to infinite. However in the Back-Propagation strategy it is said they are updated using newQ = max(oldQ, value_rollout). If this was true the values would always remain infinite, I assume the max is not applied if the previous value was still infinite.
* In the “Play” paragraph: The action is said to be picked according to the biggest Q value at the root, I assume in cases where the planning budget is smaller than the number of nodes, and not all actions at the root are explored, the actions that have not been explored are masked out.
* Non-exhaustive list of typos: “Rondom” —> “Random”, “provides a heuristics” —> “provides a heuristic”, “strcuture2vec” — > “structure2vec”, weird line break at top of page 8.