STOCHASTIC OPTIMIZATION OF SORTING NETWORKS VIA CONTINUOUS RELAXATIONS
Sorting input objects is an important step in many machine learning pipelines. However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization. In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting operator from permutation matrices to the set of unimodal row-stochastic matrices, where every row sums to one and has a distinct arg max. This relaxation permits straight-through optimization of any computational graph involve a sorting operation. Further, we use this relaxation to enable gradient-based stochastic optimization over the combinatorially large space of permutations by deriving a reparameterized gradient estimator for the Plackett-Luce family of distributions over permutations. We demonstrate the usefulness of our framework on three tasks that require learning semantic orderings of high-dimensional objects, including a fully differentiable, parameterized extension of the k-nearest neighbors algorithm.

After responses: I now understand the paper, and I believe it is a good contribution.
================================================
At a high level, the paper considers how to sort a number of items without explicitly necessarily learning their actual meanings or values. Permutations are discrete combinatorial objects, so the paper proposes a method to perform the optimization via a continuous relaxation.
This is an important problem to sort items, arising in a variety of applications, particularly when the direct sorting can be more efficient than the two step approach of computing the values and then sorting.
I like both the theoretical parts and the experimental results. In the context of ICLR, the specific theoretical modules comprise some cute results (Theorem 4; use of past works in Lemma 2 and Proposition 5). possibly of independent interest. The connections to the (Gumbel distribution <--> Plackett Luce) results are also nicely used. This Gumbel<-->PL result is well known in the social choice community but perhaps not so much in the ML community, and it is always nice to see more connections drawn between techniques in different communities. The empirical evaluations show quite good results.
However, I had a hard time parsing the paper. The paper is written in a manner that may be accessible to readers who are familiar with this (or similar) line of research, but for someone like me who is not, I found it quite hard to understand the arguments (or lack of them) made in the paper connecting various modules. Here are some examples:
- Section 6.1 states "Each sequence contains n images, and each image corresponds to an integer label. Our goal is to learn to predict the permutation that sorts these labels". One interpretation of this statement suggests that each row of Fig 3a is a sequence, that each sequence contains n=4 images (e.g., 4 images corresponding to each digit in 2960), and the goal is to sort [2960] to [0269]. However, according to the response of authors to my earlier comment, the goal is to sort [2960,1270,9803] to [1270,2960,9803].
- I did not understand Section 2.2.
- I would appreciate a more detailed background on the concrete goal before going into the techniques of section 3 and 4.
- I am having a hard time in connecting the experiments in Section 6 with the theory described in earlier sections. And this is so even after my clarifying questions to the authors and their responses. For instance, the authors explained that the experiments in Section 6.1 have \theta as vacuous and that the function f represents the cross-entropy loss between permutation z and the true permutation matrix. Then where is this true permutation matrix captured as an argument of f in (6)? Is the optimisation/gradients in (7) over s or over the CNN parameters?


In this paper, we study the problem of training real binary weight networks (without layer-wise or filter-wise scaling factors) from scratch under the Bayesian deep learning perspective, meaning that the final objective is to approximate the posterior distribution of binary weights rather than reach a point estimation. The proposed method, named as SnapQuant, has two intriguing features: (1) The posterior distribution is parameterized as a policy network trained with a reinforcement learning scheme. During the training phase, we generate binary weights on-the-fly since what we actually maintain is the policy network, and all the binary weights are used in a burn-after-reading style. At the testing phase, we can sample binary weight instances for a given recognition architecture from the learnt policy network. (2) The policy network, which has a nested parameter structure consisting of layer-wise, filter-wise and kernel-wise parameter sharing designs, is applicable to any neural network architecture. Such a nested parameterization explicitly and hierarchically models the joint posterior distribution of binary weights. The performance of SnapQuant is evaluated with several visual recognition tasks including ImageNet. The code will be made publicly available.

Summary: The paper considers a variational inference strategy for learning neural networks with binary weights. In particular, the paper proposes using a structured recognition model to parameterise the variational distribution, which couples the weights in different layers/filters in a non-trivial way. The gradient of the expected likelihood term in the variational lower bound is estimated using the REINFORCE estimator. This paper adjusts this estimator to use the gradient of the log-likelihood wrt the samples. Experiments on several image classification tasks are provided.
evaluation:
pros:
- the idea of the proposed approach is interesting: using variational inference for binary weight neural networks. While recent work on VI for discrete variables only focused on discrete latent variable models, this work shows how VI can be used for binary neural networks.

cons:
- the writing, in my opinion, needs to be improved [see my comments below]. The VI presentation is cluttered and the justification of using the pseudo-reward for reinforce is not clear.
- the experimental results are mixed and it's not clear to me how to interpret them/compare to the baselines -- what is the goal here: computational efficiency, compression or accuracy?
Some specific questions/comments:
+ What is the input of the policy/recognition network? It's not clear from the paper whether this includes the inputs of the current batch or outputs or both? If so, how are variable batch sizes handled? What is the input to this network at test time? In contrast to generative models/VAEs, the weights here are global parameters and it's not clear to me these should be varied for different data batches.
+ related to the question above: how is prediction handled at test time? Say the parameters of the variational distribution over weights are generated using the recognition network, then 100 weights are sampled given these parameters which then give 100 predictions -- should these be then averaged out to get the final prediction? I'm not quite sure I understand why the paper chose to *pick the best one* out of 100 predictions and the justification/criterion for this procedure.
+ The writing is not very clear at places, and it does not help that the references being merged with the text. I'm also not sure about some of the technical jargons/terms used in the papers:
- reinforcement learning: is this really a reinforcement learning problem? If you tackle this problem from a pure variational perspective, reinforce is used to obtain the gradient of the expected log-likelihood wrt the variational parameters. But instead of using the log likelihood, a learning signal that depends on the gradient of the log-likelihood is used.
- concrete weights -- what are these? I assume they are just binary weights sampled from the variational approximation.
- middle of page 3: p(w|X, Y) = p_\theta(w): this is not precise as p_\theta(w) is only an approximation to the exact posterior, which then allows us to lower bound the log marginal likelihood. "common practice in modern variational approximation": This is the standard way of deriving the lower bound and has been used for many years.
+ the reinforce estimator tends to have high variances since it does not make use of the gradient of the function in the expectation. This paper adjusts the vanilla estimator with a learning signal that involves the gradient. Could you comment on the bias/variance trade-off of the resulting estimator? Much of recent literature on learning discrete variables, as far as I understand, propose ways to not to have to use the vanilla reinforce, for example Concrete, Relax or rebar, albeit the focus on latent variable models.
+ model selection and uncertainty measure: the paper mentions these potential advantages of the proposed approach over deterministic binarisation schemes, but does not fully explore and test these.


Exploration is a well known challenge in Reinforcement Learning. One principled way of overcoming this challenge is to find a hierarchical abstraction of the base problem and explore at these higher levels, rather than in the space of primitives. However, discovering a deep abstraction autonomously remains a largely unsolved problem, with practitioners typically hand-crafting these hierarchical control architectures. Recent work with multitask linear Markov decision processes, allows for the autonomous discovery of deep hierarchical abstractions, but operates exclusively in the offline setting. By extending this work, we develop an agent that is capable of incrementally growing a hierarchical representation, and using its experience to date to improve exploration.

This work builds on the ICML paper from Saxe et al (2017) in which the compositionality property of LMDPs was exploited to solve multi-task hierarchies. The paper extends this work by proposing a method that learns incrementally such hierarchies instead of pre-defining them by design. Some experimental results illustrate the method on two toy problems, a 1D corridor and a corridor of rooms.
The paper deals with an interesting and hard problem. Learning hierarchies while solving an MDP is a much harder problem than solving the flat MDP or solving the hierarchical MDP. The authors leverage the compositionality of optimal controls of the LMDP framework to learn incrementally the hierarchies. Surprisingly, the proposed method not only learns those hierarchies, but also is more effective in terms of exploration.
On the positive side, the main idea is very interesting and has a lot of potential. The authors combine many techniques under the powerful framework of LMDPs such as hierarchical RL, low-rank factorization, and count-based exploration. The authors do a good job describing their approach (at a higher level).
On the negative side, the paper looks a bit incremental, given the prior existing work. I also found the paper unclear in many aspects lacking some relevant technical details (see below). The narrative is sometimes superficial or focused mainly on intuitions and analogies. Overall, it is difficult to assess the significance of this work and the results give the impression of limited applicability, beyond the described toy problem.
1- First, in order to combine optimal controls, the LMDPs need to be solved for each different boundary state, i.e., do you require to solve as many LMDPs as possible states? If that is the case, I don't think it makes sens to talk about exploration/exploration trade-off, since you really need to visit all states a priori.
2- I cannot understand what is learned and what is required a priori. the authors state that if "the multi-scale structure of the domain is known a priori, the decomposition (...) explicitly specified". What does exactly that mean? If what the method does is an incremental version of the low-rank factorization proposed in Earle at al (2018), I think the presentation can be better described in those terms.
2- Regarding exploration/exploitation tradeoff. From section 3, it seems that authors focus on a particular spatial problem and define already some exploration choices. But this means that the choice about when a state is integrated in the current MDP is already done, so no real trade-off exists?
3- The narrative in Section 3.2 is not very rigorous. The authors just mention the computational problem to keep consistency between layers and then just argue that "in practice" using count-based exploration everything works. I think a more principle approach is necessary.
4- Experiments I: what do the authors means by "exploration"? Is it just Boltzmann exploration? I can think of an exploration strategy that would choose an unseen state with probability 1 and would bring the agent to the goal in one shot.
5- Experiments II: I like the benchmark but, how does the result depend on the structure of the problem? What happens if I the rooms have very different sizes?
6- I miss some references that are very relevant to this work:
- the ICAPS paper "Hierarchical Linearly-Solvable Markov Decision Problems" by Jonson et al. seems to be the first proposing a hierarchical embedding of LMDPs.
- other factorization techniques exist, e.g., "Incremental Stochastic Factorization for Online Reinforcement Learning", Barreto et al (AAAI' 16), to uncover an MDP structure.
There are also some minor grammar mistakes:
"passive dynamics then become" -> "passive dynamics then becomes"
"reward function r" -> "reward function R"
"Howver" -> "However"
"the room contains" -> "the room that contains"
"have simply add" -> "have simply added"
"spacial and temporal" -> "spatial and temporal"
...


DIALOGWAE: MULTIMODAL RESPONSE GENERATION WITH CONDITIONAL WASSERSTEIN AUTO-ENCODER
Variational autoencoders (VAEs) have shown a promise in data-driven conversation modeling. However, most VAE conversation models match the approximate posterior distribution over the latent variables to a simple prior such as standard normal distribution, thereby restricting the generated responses to a relatively simple (e.g., unimodal) scope. In this paper, we propose DialogWAE, a conditional Wasserstein autoencoder (WAE) specially designed for dialogue modeling. Unlike VAEs that impose a simple distribution over the latent variables, DialogWAE models the distribution of data by training a GAN within the latent variable space. Specifically, our model samples from the prior and posterior distributions over the latent variables by transforming context-dependent random noise using neural networks and minimizes the Wasserstein distance between the two distributions. We further develop a Gaussian mixture prior network to enrich the latent space. Experiments on two popular datasets show that DialogWAE outperforms the state-of-the-art approaches in generating more coherent, informative and diverse responses.

This paper uses Wasserstein GAN in conditional modeling of the dialog response generation. The main goal is to learn to use two network architecture to approximate the posterior distribution of the prior network. Instead of a KL divergence, like in VAE training, they use adversarial training and instead of using softmax output from the discriminator, they use Wasserstein distance. They also introduce a multi-modal distribution, GMM, while sampling from a the posterior during training, prior during the test time. The multi-modal sampling is based on gumbel-softmax over K possible G-distributions. They experiment on Daily Dialog and Switchborad datasets and show promising improvements on qualitative measures like BLEU and BOW embedding similarities, as well as qualitative measures including human evaluations comparing againsts substantial amount of baselines.
The paper presents a marriage of a few ideas together. First of, it uses the conditional structure presented in the ACL 2017 paper "Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders". It's great that they used that paper as their baseline. The extension is to use a GAN objective function (the discriminator) as critic and use Wasserstein GAN to to resolve the gradient vanishing issue and produce smooth gradients everywhere. In ACL 2017 paper they use KL divergence to make the posterior from the prior and rec-networks as close to each other so at test time the prior network can generate the samples similar to the true data features distribution. In this paper instead of KL, they use a Discriminator as in 'Adversarial AutoEncoders' paper. This paper extends AAE, instead uses the Wasserstein distance instead (1-Lipschitz function instead of softmax for the discriminator). The W-GAN has been shown to produce good results in text generation in this year's ICML 2018 with the paper 'Adversarially Regularized GAN' (AARE). The idea was to resolve VAE posterior collapse issue by using a discriminator as a regularizer instead of KL divergence with a stronger sampler from the output of the generator to map from noise sampler into the latent space. Interestingly, AARE paper is not cited in this work, which i think is an issue. I understand that paper was just for generation purpose not specific to the dialog modeling, but it makes the claims in the paper misleading such as: "Unlike VAE conversation models that impose a simple distribution over latent variables, DialogWAE models the data distribution by training a GAN within the latent variable space".
The part that i liked is the fact that they used multimodal gaussian distributions. I agree with the authors that using Gaussian for the approximating distribution only limits the sampling space and can weaken the models capability of variation. Although it is not proven for text, in image, the gaussian posteriors during training converge together into a single gaussian, causing blurry images. In this text this might correspond to dull responses in dialog. I would like the authors to comment on the interpretability of the components. Perhaps show a sample from each component (in the end the model decides which modal to choose before generation. Are these GMMs overlapping and how much ? Can you measure the difference between the means ?
I find the experiments extensive except the datasets are weaker.
I like the fact that they included human evaluations.


Recently, there is regained interest in large batch training of neural networks, both in theory and practice. New insights and methods allowed certain models to be trained using large batches with no adverse impact on performance. Most works focused on accelerating wall clock training time by modifying the learning rate schedule, without introducing accuracy degradation. We propose to use large batch training to boost accuracy and accelerate convergence by combining it with data augmentation. Our method, "batch augmentation", suggests using multiple instances of each sample at the same large batch. We show empirically that this simple yet effective method improves convergence and final generalization accuracy. We further suggest possible reasons for its success.

This paper tested a very simple idea: when we do large batch training, instead of sampling more training data for each minibatch, we use data augmentation techniques to generate training data from a small minibatch. The authors claim the proposed method has better generalization performance.
I think it is an interesting idea, but the current draft does not provide sufficient support.
1. The proposed method is very simple. In this case, I would expect the authors provide more intuitive explanations. It looks to me the better generalization comes from more complicated data augmentation, not from the proposed large batch training.
2. It is unclear to me what is the benefit of the proposed method. Even provided more computing resources, the proposed method is not faster than small batch training. The improvement on test errors does not look significant. If given more computing resources, and under same timing constraint, we have many other methods to improve performance. For example, a simple thing to do is t0 separately train networks with standard setting and then ensemble trained networks. Or apply distributed knowledge distillation like in (Anil 2018
Large scale distributed neural network training through online distillation)
3. The experiments are not strong. The largest batch considered is 64*32, which is relatively small. In figure 1 (b), the results of M=4,8,16,32 are very similar, and it looks unstable. It is unclear what is the default batchsize for Imagenet. In Table 1, the proposed method tuned M as a hyperparameter. The baselines are fairly weak, the authors did not compare with any other method. I would expect at least the following baselines:
i)  use normal large batch training and complicated data augmentation, train the model for same number of epochs
ii) use normal large batch training and complicated data augmentation, train the model for same number of iterations
ii) use normal large batch training and complicated data augmentation, scale the learning rate up as in Goyal et al. 2017
4. For theorem 1, it is hard to say how much the theoretical analysis based on linear approximation near global minimizer would help understand the behavior of SGD. I fail to understand the the authors’ augmentation. Following the author’s logic, normal large batch training decrease the variability of <H>_k and \lambda_max, which converges to ‘’flat’’ minima. It contradicts with the authors’ other explanation.
5. In section 4.2, I fail to understand why the proposed method can affect the norm of gradient.
6. Related works:
Smith et al. 2018 Don't Decay the Learning Rate, Increase the Batch Size.
=============== after rebuttal ====================
I appreciate the authors' response, but I do not think the rebuttal addressed my concerns. I will keep my score and argue for the rejection of this paper.
My main concern is that the benefit of this method is unclear. The main baseline  that has been compared is the standard small-batch training. However, the proposed method use a N times larger batch and same number of iterations, and hence N times more computation resources. Moreover, the proposed method also use N times more augmented samples. Like the authors said, they did not propose new data augmentation method, and their contribution is how to combine data augmentation with large-batch training. However, I am not convinced by the experiments that the good performance is from the proposed method, not from the N times more augmented samples. I have suggested the authors to compare with stronger baselines to demonstrate the benefits. However, the authors quote a previous paper that use different data augmentation and (potentially) other experimental settings.
The proposed method looks unstable. Moreover, instead of showing the consistent benefits of large batch, the authors tune the batchsize as a hyperparameter for different experiments.
Regarding the theoretical part, I still do not follow the authors' explanation. I think it could at least be improved for clarity.


GENERATIVE QUESTION ANSWERING: LEARNING TO ANSWER THE WHOLE QUESTION
Discriminative question answering models can overfit to superficial biases in datasets, because their loss function saturates when any clue makes the answer likely. We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it. Our question answering (QA) model is implemented by learning a prior over answers, and a conditional language model to generate the question given the answerallowing scalable and interpretable many-hop reasoning as the question is generated word-by-word. Our model achieves competitive performance with comparable discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work. The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving state-of-the-art results on ADVERSARIALSQUAD.

This paper proposes a generative approach to textual QA on SQUAD and visual QA on CLEVR dataset, where, a joint distribution over the question and answer space, given the context (image or Wikipedia paragraphs) is learned (p(q,a|c)). During inference the answer is selected by argmax p(q,a|c) that is equal to p(a|c,q) if the question is given. Authors propose an architecture shown in Fig. 3 of the paper, where generation of each question word is condition on the corresponding answer, context and all the previous words generated in the question so far. The results compared to discriminative models are worse on SQUAD and CLEVR. Nevertheless, authors show that given the nature of the model that captures more complex relationships, the proposed model performs better than other models on a subset of SQUAD that they have created based on answer type (number/date/people), and also on adversarial SQUAD.
Comments / questions:
The paper is well written, except for a few parts mentioned below, all the equations / components are explained clearly. The motivation of the paper is clearly stated as using generative modelling in (V)QA to overcome biases in these systems, e.g., answering questions by just using word matching and ignoring the context (context=image or Wikipedia paragraph). I have the following questions / comments about the paper which addressing them by authors will help to better understand/evaluate the paper:
1.	In page 3 on the top of section 2.3, can authors provide a more clear explanation of the additional 32-dimensional embedding added to each word representation? Also in Table 2, please add an ablation how much gain are you getting from this?
2.	In the same page (page 3), section 2.4, paragraph 2, put the equation in a separate line and number it + clearly explain how you have calculated s^{endpoints} and s{length}.
3.	In page 4 section 2.5.2 paragraph 2, the way the bias term is calculated and the incentive behind it is not clear. Can authors elaborate on this?
4.	In page 6 section 3.2 the first paragraph authors claim that their model is performing multihop reasoning on CLEVR, while there is no explicit component in their model to perform multiple rounds of reasoning. Can authors clarify their statement?
5.	In section 3.3 the third paragraph, where authors explain the question agnostic baselines, can they clarify what they mean by “the first answer of the correct type”?
6.	In Table 5 and section 3.4 the second paragraph, authors are stating that “… The improvement may be due to the model’s attempt to explain all question words, some of which may be unlikely under the distractor”. It is very important that the authors do a complete ablation study similar to that of Table 2 to clarify how much gain is achieved using each component of generative model.
7.	In page 8 under related works:
a.	In paragraph 2 where authors state “Duan et al. (2017) and Tang et al. (2017) train answering and generation models with separate parameters, but add a regularisation term that encourages the models to be consistent. They focus on answer sentence selection, so performance cannot easily be compared with our work.”. I do not agree that the performance can not be compared, it is easily comparable by labeling a sentence containing the answer interval as the answer sentence. Can authors provide comparison of their work with that of Duan et al. (2017) and Tang et al. (2017)?
b.	In the same paragraph as 7.a, the authors have briefly mentioned “Echihabi & Marcu (2003) describe an earlier method for answering questions in terms of the distribution of questions given answers.” Can they provide a more clear explanation of this work and its relation to / difference with their work?
////////////
I would like to thank authors for providing detailed answers to my questions. After reading their feedback, I am now willing to change my score to accept.


GENERALIZABLE ADVERSARIAL TRAINING VIA SPEC- TRAL NORMALIZATION
Deep neural networks (DNNs) have set benchmarks on a wide array of supervised learning tasks. Trained DNNs, however, often lack robustness to minor adversarial perturbations to the input, which undermines their true practicality. Recent works have increased the robustness of DNNs by fitting networks using adversarially-perturbed training samples, but the improved performance can still be far below the performance seen in non-adversarial settings. A significant portion of this gap can be attributed to the decrease in generalization performance due to adversarial training. In this work, we extend the notion of margin loss to adversarial settings and bound the generalization error for DNNs trained under several well-known gradient-based attack schemes, motivating an effective regularization scheme based on spectral normalization of the DNN's weight matrices. We also provide a computationally-efficient method for normalizing the spectral norm of convolutional layers with arbitrary stride and padding schemes in deep convolutional networks. We evaluate the power of spectral normalization extensively on combinations of datasets, network architectures, and adversarial training schemes.

The paper first provides a generalization bounds for adversarial training, showing that the error bound depends on Lipschitz constant. This motivates the use of spectral regularization (similar to Miyato et al 2018) in adversarial training. Using spectral regularization to improve robustness is not new, but it's interesting to combine spectral regularization and adversarial training. Experimental results show significant improvement over vanilla adversarial training.
The paper is nicely written and the experimental results are quite strong and comprehensive. I really like the paper but I have two questions about the results:
1. The numbers reported in Figure 5 do not match with the performance of adversarial training in previous paper. In PGM L_inf adversarial training/attack (column 3 of Figure 5), the prediction accuracy is roughly 50% under 0.1 infinity norm perturbation. However, previous papers (e.g., "Obfuscated Gradients Give a False Sense of Security") reported 55% accuracy under 0.031 infinity norm perturbation. I wonder why the numbers are so different.
Maybe it's because of different scales? Previous works usually scale each pixel to [0,1] or [-1,1], maybe the authors use the [0, 255] scale? But 0.1/255 will be much smaller than 0.031.
Another factor might be the model structure. If Alexnet has much lower accuracy, it's probably worthwhile to conduct experiments on the same structure with previous works (Madry et al and Athalye et al) to make the conclusion more clear.
2. What's the training time of the proposed method compared with vanilla adversarial training?
3. The idea of using SN to improve robustness has been introduced in the following paper:
"Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks"
(but this paper did not combine it with adv training).


Deep neural networks (DNN) are widely used in many applications. However, their deployment on edge devices has been difficult because they are resource hungry. Binary neural networks (BNN) help to alleviate the prohibitive resource requirements of DNN, where both activations and weights are limited to 1-bit. We propose an improved binary training method (BNN+), by introducing a regularization function that encourages training weights around binary values. In addition to this, to enhance model performance we add trainable scaling factors to our regularization functions. Furthermore, we use an improved approximation of the derivative of the sign activation function in the backward computation. These additions are based on linear operations that are easily implementable into the binary training framework. We show experimental results on CIFAR-10 obtaining an accuracy of 86.5%, on AlexNet and 91.3% with VGG network. On ImageNet, our method also outperforms the traditional BNN method and XNOR-net, using AlexNet by a margin of 4% and 2% top-1 accuracy respectively.

Summary:
This paper presents three small improvements for training binarized neural networks: (1) a modified straight-through estimator, (2) a novel regularizer to push weights to +/- 1, and (3) the use of scaling factors for the binarized weights. Using the methods presented, the validation accuracies on ImageNet and CIFAR-10 are improved by just under 2 percentage points.
  Pros:
    - Decent improvement in the performance of the binarized network in the end
    - The presented regularizers make sense and seem effective. The modified straight-through estimator seems reasonable as well, although the authors do not compare to recent work with a similar adjustment.
  Cons:
    - The paper is poorly written and confusing. It reads as if it was written in one pass with no editing or re-writing to clarify contributions or key points, or ensure consistency.
    - While the final numbers are acceptable, the experiments themselves could be stronger and could be presented more effectively.
   - The novelty of the scale factors is questionable.
Questions and comments:
1. How exactly is the SS_\beta activation used? It is entirely unclear from the paper, which contradicts itself in multiple ways. Is SS_\beta used in the forward pass at all for either the weight or activation binarization? Or is only its derivative used in the backward pass? If the latter, then you are not replacing the activation anywhere but are simply using a different straight-through estimator in place of the saturated straight-through estimator (e.g., see [1]).
   (a) At the beginning of Section 3.3, you say that you modify the training procedure by replacing the sign binarization with the SS_\beta activation. This sounds like it is referring to the activation function at each layer; however, the pseudocode says that you are using sign() as the per-layer activation.
   (b) Further, Figure 4 shows that you are using the SS_\beta function to do weight binarization. However, again, the pseudocode shows that you are using sign() to do the weight binarization.
2. In [1], the authors used a similar type of straight-through estimator (essentially, the gradient of tanh instead of hard_tanh) and found that to be quite effective. You should compare to their method. Also, it's possible that SS_\beta reduces to tanh for some choice of \beta -- is this true?
3. The use of scale factors seems to greatly increase the number of parameters in the network and thus greatly decrease the compression benefits gained by using binarization, i.e., you require essentially #scale_factors =  a constant factor times the number of actual parameters in the network (since you have a scale factor for each convolutional filter and for each column of each fully-connected layer). As a result of this, what is the actual compression multiplier that your network achieves relative to the original network?
4. For the scale factor, how does yours differ from that used in Rastegari et al. (2016)? It seems the same but you claim that it is a novel contribution of your work. Please clarify.
5. Why did learning \beta not work? What was the behavior? What values of \beta did learning settle on?
6. I assume that for each layer output y_i = f(W_i x_i), the regularizer is applied as R(y_i) while at the same time y_i is passed to the next layer -- is this correct? The figures do not clearly show this and should be changed to more clearly show how the regularizer is computed and used, particularly in relation to the activation.
7. In the pseudocode:
   (a) What does "mostly bitwise operations" mean? Are some floating point?
   (b) Is this the shift-based batchnorm of Hubara et al. (2016)?
8. For Table 1:
   (a) I assume these are accuracies? The caption should say.
   (b) Why are there no comparisons to the performance of other methods on this dataset?
   (c) Any thoughts as to why your method performs better than the full-precision method on this dataset for VGG?
8. For Table 2:
   (a) Does Table 2 show accuracies on ImageNet? You need to make this clear in the caption.
   (b) What type of behavior do the runs that do not converge show? This seems like a learning rate problem that is easily fixable. Are there no hyperparameter values that allow it to converge?
   (c) What behavior do you see when you use SS_1 or SS_2, i.e., \beta = 1 or \beta = 2? Since lower \beta values seem better.
   (d) The regularization seems to be the most useful contribution -- do you agree?
   (e) Why did you not do any ablations for the scale factor? Please include these as well.
9. For Table 3, did you compute the numbers for the other approaches or did you use the numbers from their papers? Each approach has its own pros and cons. Please be clear.
10. Are there any plots of validation accuracy versus epoch/time for the different algorithms in order to ensure that the reported numbers were not simply cherry-picked from the run? I assume that you simply used the weights from the end of the 50th epoch -- correct?
11. Is there evidence for your introductory claims that 'quantizing weights ... make neural networks harder to train due to a large number of sign fluctuations' and 'maintaining a global structure to minimize a common cost function is important' ? If so, you should cite this evidence. If not, you should make it clear that these are hypotheses.
12. Why are there not more details about the particular architectures used? These should be included in the appendices to aid those who would like to rerun your experiments. In general, please include more experiment details in the body or appendices.
Detailed comments:
- R(l) is not defined in Figure 1 and thus is confusing. Also, its replacement of 'Error' from the original figure source makes the figure much more confusing and less clear.
- Typos:
   - 'accustomed' (p.1)
   - 'the speed by quantizing the activation layers' doesn't make sense (p.1)
   - 'obtaining' (p.4)
   - 'asymmetric' doesn't make sense because these are actually symmetric functions across the y-axis (p.4)
   - 'primary difference is that this regularization ...' --> 'primary difference is that their regularization ...' (p.4)
   - 'the scales with 75th percentile of the absolute value ... ' is very confusing and unclear (p.7)
   - 'the loss metric used was the cross-entropy loss, the order of R_1.' I do not know what you're trying to say here (p.8)
- Citations: Fix the capitalization issues, typos, and formatting inconsistencies.
[1]  Friesen and Domingos. Deep Learning as a Mixed Convex-Combinatorial Optimization Problem. ICLR 2018.
-------------------
After reading the author response, I do not think the paper does a sufficient job of evaluating the contributions or comparing to existing work. The authors should run ablation experiments, compare to existing work such as [1], and evaluate on additional datasets. These were easy tasks that could have been done during the review period but were not.
If I wanted to build on top of this paper to train higher accuracy binary networks, I would have to perform all of these tasks myself to determine which contributions to employ and which are unnecessary. As such, the paper is currently not ready for publication.


This paper introduces an approach to learn representations based on the Lorentzian distance in hyperbolic geometry. Hyperbolic geometry is especially suited to hierarchically-structured datasets, which are prevalent in the real world. Current hyperbolic representation learning methods compare examples with the Poincaré distance metric. They formulate the problem as minimizing the distance of each node in a hierarchy with its descendants while maximizing its distance with other nodes. This formulation produces node representations close to the centroid of their descendants. We exploit the fact that the centroid w.r.t the squared Lorentzian distance can be written in closed-form. We show that the Euclidean norm of such a centroid decreases as the curvature of the hyperbolic space decreases. This property makes it appropriate to represent hierarchies where parent nodes minimize the distances to their descendants and have smaller Euclidean norm than their children. Our approach obtains state-of-the-art results in retrieval and classification tasks on different datasets.

Summary
Learning embeddings of graphs in hyperbolic space have become popular and yielded promising results. A core reason for that is learning hierarchical representations of the graphs is easier in hyperbolic space due to the curvature and the geometrical properties of the hyperbolic space. Similar to [1, 2], this paper uses Lorentzian model of the hyperbolic space in order to learn embeddings of the graph. The main difference of the proposed approach in this paper is that  they come up with a closed-form solution such that each node representation close to the centroid of their descendants. A curious property of the equation for the centroids proposed to learn the embeddings of each node also encodes information related to the specificity in the Euclidean norm of the centroid. Also this paper introduces two additional hyperparameters. Beta hyperparameter is selected to control the curvature of the space. Depending on the task the optimal curvature can be tuned to be a different value. This also ties closely with the de-Sitter spaces. Authors provide results on different graph embedding benchmark tasks. The paper claims that, an advantage of the proposed approach is that the embedding of the model can be tuned with regular SGD without needing to use Riemannian optimization techniques.
Questions
Have you tried learning beta instead of selecting as a hyperparameter?
The paper claims that Riemannian optimization is not necessary for this model, but have you tried optimizing the model with the Riemannian optimization methods?
Equation 11, bears lots of similarity to the Einstein gyro-midpoint method proposed by Abraham Ungar which is also used by [2]. Have you investigated the relationship between the two formulations?
On Eurovoc dataset the results of the proposed method is worse than the d_P in H^d. Do you have a justification of why that happens?
Pros
The paper delivers some interesting theoretical findings about the embeddings learned in hyperbolic space, e.g. a closed for equation in the
The paper is written well. The goal and motivations are clear.
Cons
Experiments are only limited to small scale-traditional graph datasets. It would be more interesting to see how those embeddings would perform on large-scale datasets such as to learn knowledge-base embeddings or for recommendation systems.
Although the idea is interesting. Learning graph embeddings have already been explored in [1]. The main contribution of this paper is thus mainly focuses on the close-form equation for the centroid and the curvature hyperparameter. These changes provide significant improvements on the results but still the novelty of the approach is in that sense limited compared to [1].
Minor comment:
It is really difficult to understand what is in Figure 2 and 3. Can you reduce the number of data points and just emphasize a few nodes in the graph that shows a clear hierarchy.
[1] Nickel, Maximilian, and Douwe Kiela. "Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry." arXiv preprint arXiv:1806.03417 (2018).
[2] Gulcehre, Caglar, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia et al. "Hyperbolic Attention Networks." arXiv preprint arXiv:1805.09786 (2018).


Graph prediction methods that work closely with the structure of the data, e.g., graph generation, commonly ignore the content of its nodes. On the other hand, the solutions that consider the node's information, e.g., classification, ignore the structure of the whole. And some methods exist in between, e.g., link prediction, but predict the structure piece-wise instead of considering the graph as a whole. We hypothesize that by jointly predicting the structure of the graph and its nodes' features, we can improve both tasks. We propose the Graph Learning Network (GLN), a simple yet effective process to learn node embeddings and structure prediction functions. Our model uses graph convolutions to propose expected node features, and predict the best structure based on them. We repeat these steps sequentially to enhance the prediction and the embeddings. In contrast to existing generation methods that rely only on the structure of the data, we use the feature on the nodes to predict better relations, similar to what link prediction methods do. However, we propose an holistic approach to process the whole graph for our predictions. Our experiments show that our method predicts consistent structures across a set of problems, while creating meaningful node embeddings.

This paper proposes a supervised learning method for predicting the connectivity of a graph based on both the features of nodes in the graph as well as the overall graph structure, rather than just the structure of the graph or just the node features. The approach is evaluated on two synthetic datasets, a “community” dataset and a “geometric figures” dataset.
Unfortunately, I do not think this paper as it stands currently is ready for publication at ICLR for the following reasons: (1) the comparison and discussion with prior literature is lacking; (2) the experiments are rather weak; and (3) the significance and novelty of the method itself seems limited.
1. I am very surprised there was no discussion of [1] (or even better, a comparison to), which is another method which uses information about the full graph (via message-passing) to infer the connectivity of the graph in an unsupervised way. The discussion of the literature on graph neural networks in general is a bit weak, missing important references such as [2-4]. Such approaches, especially message-passing style approaches like [4], do exactly what the current paper suggests has not been done: they make predictions based on information in the nodes while considering the structure of the graph as a whole (via message-passing). Although [4] does not explicitly make edge predictions the approach is straightforward to generalize to making edge predictions, see [5].
2. The experiments in the paper only test the proposed method on very toy domains, and thus feel weak. The results in Table 2, for example, suggest that the proposed method has reached ceiling-level performance and thus to really tell the difference between GLN_f, GLN_c, and any other methods, more difficult problems are called for. The geometric figures dataset, in particular, does not seem to me like it would test the claim that the paper would like to make: that it is important to take into account the fully structure of the graph when predicting edges. Indeed, there is a very simple rule that can be applied in the geometric figures case which does not use global graph information (if the two nodes have the same color, connect them, if they are different colors, do not connect them). It is therefore unsurprising that GLN_c actually does slightly better than GLN_f (according to Table 2) on geometric figures.
Additionally, the experiments do not provide much insight into the architecture itself. For example, the present architecture is meant to repeat the embedding and link-prediction steps some number of times, and in the experiments it seems that these steps are repeated four times. But how important is the repetition in practice? It would be nice to see the effect of repetitions on final performance, to demonstrate whether this is in fact an important component of the model or not. Similarly, there are several different loss functions but it is not obvious to what extent these losses contribute to the final performance of the model. It would be nice if there could be some ablation studies that train the model with different combinations of losses to see which are actually important.
3. Finally, I have some concerns about the model itself. If I understand correctly, both f_l and c_l depend on a number of learned parameters which is a function of the number of nodes in the graph. This is unfortunate, as one of the strengths of the graph neural network approach is that GNNs usually have a number of parameters that is independent of the size of the graph, thus allowing GNNs to scale to graphs of arbitrary size. However, that is not the case in this model. Moreover, the architecture of f_l and c_l do not seem particularly novel. f_l just involves passing the node embeddings through a MLP to produce the link predictions. c_l involves something closer to message-passing, though where weighted combinations are learned on a per-node basis (rather than sharing the same function across all nodes). This could be interesting, even though it sacrifices the scale-free nature of GNNs, if it could be shown to actually outperform existing GNN approaches on more realistic datasets. However, given the lack of experiments demonstrating this, it is hard to say how significant the approach is.
[1] Kipf, Fetaya, Wang, Welling & Zemel (2018). Neural Relational Inference for Interacting Systems. ICML 2018.
[2] Gori, M., Monfardini, G., and Scarselli, F. (2005). A new model for learning in graph domains. IJCNN 2005.
[3] Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2009). The graph
neural network model. IEEE Transactions on Neural Networks, 20(1):61–80.
[4] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212.
[5] Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., ... Pascanu, R. (2018). Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261.


K FOR THE PRICE OF 1: PARAMETER-EFFICIENT MULTI-TASK AND TRANSFER LEARNING
We introduce a novel method that enables parameter-efficient transfer and multitask learning with deep neural networks. The basic approach is to learn a model patch -a small set of parameters -that will specialize to each task, instead of finetuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match singletask performance.

Summary: the paper introduces a new way of fine-tuning neural networks. Instead of re-training the whole model or fine-tuning the last few layers, the authors propose to fine-tune a small set of model patches that affect the network at different layers. The results show that this way of fine-tuning is superior to above mentioned typical ways either in accuracy or in the number of tuned parameters in three different settings: transfer learning, multi-task learning and domain adaptation.
Quality: the introduced way of fine-tuning is interesting alternative to the typical last layer re-training. I like that the authors present an intuition behind their approach and justify it by an illustrative example. The experiments are fair, assuming the authors explain the choice of hyper-parameters during the revision.
Clarity: in general the paper is well-written. The discussion of multi-task and domain adaptation parts can be improved though.
Originality: the contributions are novel to my best knowledge.
Significance: high, I believe the paper may facilitate a further developments in the area.
I ask the authors to address the following during the rebuttal stage:
* explain the choice of the hyper-parameters of RMSProp (paragraph under Table 1).
* fix Figure 3, it's impossible to read in the paper-printed version
* explain how the average number of parameters per model in computed in Tables 4 and 5. E.g. 700K params/model in the first column of Table 4 is misleading - I suppose the shared parameters are not taken into account. The same holds for 0 in the second column, etc.
* add a proper discussion for domain adaptation part. The simple "The results are shown in Table 5" is not enough.
* consider leaving the discussion of cost-efficient model cascades out. The presented details are too condensed and do not add value to the paper.
* explain how different resolutions are managed by the same model in the domain adaptation experiments.


We consider the fundamental problem of semi-supervised node classification in attributed graphs with a focus on few-shot learning. Here, we propose Structural Regularized Graph Convolutional Networks (SRGCN), novel neural network architectures extending the well-known GCN structures by stacking transposed graph convolutional layers for reconstruction of input features. We add a reconstruction error term in the loss function as a regularizer. Unlike standard regularization such as L 1 or L 2 , which controls the model complexity by including a penalty term depends solely on parameters, our regularization function is parameterized by a trainable neural network whose structure depends on the topology of the underlying graph. The new approach effectively addresses the shortcomings of previous graph convolution-based techniques for learning classifiers in the few-shot regime and significantly improves generalization performance over original GCNs when the number of labeled samples is insufficient. Experimental studies on three challenging benchmarks demonstrate that the proposed approach has matched state-of-the-art results and can improve classification accuracies by a notable margin when there are very few examples from each class.

This paper proposes to regularize the training of graph convolutional neural networks by adding a reconstruction loss to the supervised loss. Results are reported on citation benchmarks and compared for increasing number of labeled data.
The presentation of the paper could be significantly improved. Details of the proposed model are missing and the effects of the proposed regularization w.r.t. other regularizations are not analyzed.
My main concerns are related to the model design, the novelty of the approach (adding a reconstruction loss) and its experimental evaluation.
Details / references of the transposed convolution operation are missing (see e.g. https://ieeexplore.ieee.org/document/7742951). It is not clear what the role of the transposed convolution is in that case. It seems that the encoder does not change the nodes nor the edges of the graph, only the features, and the filters of the transposed convolution are learnt. If the operation is analogous to the transposed convolution on images, then given that the number of nodes in the graph does not change in the encoder layers (no graph coarsening operations are applied), then learning an additional convolution should be analogous (see e.g. https://arxiv.org/pdf/1603.07285.pdf Figure 4.3.). Could the authors comment on that?
Details on the pooling operation performed after the transposed convolution are missing (see e.g. https://arxiv.org/pdf/1805.00165.pdf, https://arxiv.org/pdf/1606.09375.pdf). Does the pooling operation coarsen the graph? if so, how is it then upsampled to match the input graph?
Figure X in section 2.1. does no exist.
Supervised loss in section 2.2.1 seems to disregard the sum over the nodes which have labels.
\hat A is not defined when it is introduced (in section 2.2.2), it appears later in section 2.3.
Section 2.2.2 suggests that additional regularization (such as L2) is still required (note that the introduction outlines the proposed loss as a combination of reconstruction loss and supervised loss). An ablation study using either one of both regularizers should be performed to better understand their impact. Note that hyper-parameters chosen give higher weight to L2 regularizer.
Section 3 introduces a bunch of definitions to presumably compare GCN against SRGCN, but those measures of influence are not reported for any model.
Experimental validation raises some concerns. It is not clear whether standard splits for the reported datasets are used. It is not clear whether hyper-parameter tuning has been performed for baselines. Authors state "the parameters of GCN and GAT and SRGCN are the same following (Kipf et al; Velickovic et al.)". Note that SRGCN probably has additional parameters, due to the decoder stacked on top of the GCN. Reporting the number of parameters that each model has would provide more insights. Results are not reported following standards of running the models N times and providing mean and std. Moreover, there are no results using the full training set.


We describe MÊLÉE, a meta-learning algorithm for learning a good exploration policy in the interactive contextual bandit setting. Here, an algorithm must take actions based on contexts, and learn based only on a reward signal from the action taken, thereby generating an exploration/exploitation trade-off. MÊLÉE addresses this trade-off by learning a good exploration strategy based on offline synthetic tasks, on which it can simulate the contextual bandit setting. Based on these simulations, MÊLÉE uses an imitation learning strategy to learn a good exploration policy that can then be applied to true contextual bandit tasks at test time. We compare MÊLÉE to seven strong baseline contextual bandit algorithms on a set of three hundred real-world datasets, on which it outperforms alternatives in most settings, especially when differences in rewards are large. Finally, we demonstrate the importance of having a rich feature representation for learning how to explore.

This paper investigates a meta-learning approach for the contextual bandit problem. The goal is to learn a generic exploration policy from datasets, and then to apply the exploration policy to contextual bandit tasks. The authors have adapted an algorithm proposed for imitation learning (Ross & Bagnell 2014) to their setting. Some theoretical guarantees straightforwardly extracted from (Ross & Bagnell 2014) and from (Kakade et al 2008) are presented. Experiments are done on 300 supervised datasets.
Major concerns:
1 This paper investigates a problem that does not correspond to the real problem: how to take advantage of a plenty of logs generated by a known stochastic policy (or worst unknown deterministic policy) for the same (or a close) contextual bandit task?
Most of companies have this problem. I do not know a single use case, in which we have some full information datasets, which are representative of contextual bandit tasks to be performed. If the full information datasets does not correspond to the contextual bandit tasks, it is not possible to learn something useful for the contextual bandit task.
2 The experimental validation is not convincing.
The experiments are done on datasets, which are mostly binary classification datasets. In this case, the exploration task is easy. May be it is the reason why the exploration parameter \mu or \epsilon = 0 provides the best results for MELEE or \epsilon-greedy?
The baselines are not strong. The only tested contextual bandit algorithm is LinUCB. However a diagonal approximation of the covariance matrix is used when the dimension exceeds 150. In this case LinUCB is not efficient. There are a lot of contextual bandit algorithms that scale with the dimension.
3 The theoretical guarantees are not convincing.
The result of Theorem 1 is a weak result. A linear regret against the expected reward of the best policy is usually considered as a loosely result. Theorem 2 shows that there is no theoretical gain of the use of the proposed algorithm: the upper bound of the expected number of mistakes obtained when Banditron is used in MELEE is upper than the one of Banditron alone.
Minor concerns:
The algorithms are not well written. POLOPT function has sometimes one parameter, sometimes two and sometimes three parameters. The algorithm 1 is described in section 2, while one of the inputs of the algorithm 1 (feature extractor function) is described in section 3.1. The algorithm 1 seems to return all the N exploration policies. The choice of the returned policy has to be described.
In contextual bandits, the exploration policy is not handcrafted. The contextual bandit algorithms are designed to be optimal or near optimal in worst case: they are generic algorithms.


Anomaly detection is of the critical issue in both fundamental machine learning research area and industrial applications. A good anomaly detection should accurately discriminate anomalies from normal samples. Although most previous anomaly detection methods achieve good performances on various tasks, they do not perform well on high-dimensional imbalanced small data-sets with multimodal distribution. Unlike existing approaches, each mode in the distribution is individually detected by a single model, and thus handle multi-modalities. In this paper, we develop a multimodal one-class model based on generative adversarial network (GAN) to distinguish anomalies from normal samples. The generator of the GAN takes in a noise vector with a pseudo latent code to generate instances at the low-density area of normal samples within the same data space to simulate the anomalies. The discriminator of the GAN then is trained to distinguish the generated samples from the normal samples. Since the generated instances mimic the low-density distribution of the normal samples (outliers), the discriminator should directly detect real anomalies from normal samples. We test our methods on several datasets. In these experiments, our method outperforms the state-of-the-art anomaly detection methods on both the accuracy and F1 score.

This paper presents a GAN approach adapted for multi-modal distributions of single class data. The generator is trained to generate samples in the low density areas of the data distribution. The discriminator is training to distinguish between generated and real samples and hence is able to discriminate between normal data (=real) and anomalous data (=generated, in low density areas of the normal data).
To force the model to map the different modes of the data, a categorical latent variable is used that represents the potential distribution modes. Both a one-hot code and a Gaussian mixture model are explored. This is not a novel approach, however, no citations are provided.
To force the generator to produce samples in the low density areas of the data distribution, a Complementary GAN is used. The authors cite OCAN [zheng18], which in turns cites [Dai, NIPS17]. This approach has the advantage that no threshold needs to be fine-tuned since the discriminator can directly be used for anomaly detection.
Constraints derived from both these goals are included in the loss function, which, in addition, includes terms to encourage diversity and similarity of the generated samples.
The model is tested on a proprietary dataset of real manufacturing product. The dimension of the data is 280 (after proprietary feature extraction). The authors compare their approach to 9 other anomaly detection methods. The reported performance is the highest. The OCAN method has similar performance. The authors specify that fine-tuning is need for all other methods (except OCAN). Fine-tuning is performed on the same data used for testing, hence providing a marked advantage. However, I do not understand why OCAN is listed in table 1 with both fine-tuning and no fine-tuning (raw). This is not explained and should be clarified. In any case, the combination of Complementary-GAN and the multi-modal latent variable seem to be very effective on this dataset. To understand whether this approach is really superior, other benchmark datasets should be tested.
The article is technically sound. The citations are generally ok, except for the missing citation related to the use of latent categorical codes for push the model into mapping multiple modes of the data. The math is reasonable, although some notations are a bit hard to follow. The English needs to be improved. There are many grammatical errors and the paper needs to be proof-read. Some errors make it hard to understand the text. In particular the adjective modal is used throughout the paper as a noun instead of 'mode'. There are also several LaTex formatting errors which lead to some gibberish and some of the figures are too small making them unusable when the paper is printed.
Overall, I think the paper is incremental, as it combines two previously published methods. It also lacks generality as only one (proprietary) dataset is used. English needs to be proof-read and formatting errors fixed.


We propose an approach to predict the 3D shape and pose for the objects present in a scene. Existing learning based methods that pursue this goal make independent predictions per object, and do not leverage the relationships amongst them. We argue that reasoning about these relationships is crucial, and present an approach to incorporate these in a 3D prediction framework. In addition to independent per-object predictions, we predict pairwise relations in the form of relative 3D pose, and demonstrate that these can be easily incorporated to improve object level estimates. We report performance across different datasets (SUNCG, NYUv2), and show that our approach significantly improves over independent prediction approaches while also outperforming alternate implicit reasoning methods.

<Summary>: This paper presented a method for incorporating binary relationship between objects (relative location, rotation and scale) into single object 3d prediction. It is built on top of previously published work of [a] and used same network architecture and loss as of [a] and only added the binary relations between objects for object 3d estimation. The results are shown on SUNCG synthetic dataset and only *4 image* instances of NYUv2 dataset which is very small for a computer vision task.
[a] Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A Efros, and Jitendra Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In CVPR, 2018.
<Pros>: The paper tackles a problem of obvious interest to computer vision research community. It shows better results compared to previous similar work of [a] without considering binary relation between objects.
<Cons>:
*Technical details are missing:
The set of known and unknown variables are not clear throughout the paper:
-The extrinsic camera parameters are known or estimated by the method?
-The intrinsic camera parameters are known or estimated by the method?
-What are the properties of ground truth bounding boxes in 2D camera frame and 3D space?
-What is the coordinate of translation? is it in camera coordinate or world coordinate?
-What are the variations of camera poses in training and testing for synthetic dataset and how are the samples generated? Are the train/test images generated or are rendered images from previously published work of [b] used?
[b] Yinda Zhang, Shuran Song, Ersin Yumer, Manolis Savva, Joon-Young Lee, Hailin Jin, and Thomas Funkhouser. Physically-based rendering for indoor scene understanding using convolutional neural networks. In CVPR, 2017.
*The proposed method is trained on synthetic dataset of SUNCG and their object relations have biases from scene creators. While using binary relation between objects increase the recall in prediction it can also make the predictions bias to the most dominant relations and decrease the precision of detection in rare cases in synthetic dataset. Also, such bias can decrease prediction precision in images of real scenes.

*One of the main issues in this paper is that the result of fully automated pipeline versus having ground-truth annotation at test time are mixed up. For example, in the teaser figure (Figure 1-b), does the proposed method use ground truth bounding boxes or not? It is mentioned in figure caption: “(b) Output: An example result of our method that takes as input the 2D image and generates the 3D layout.”. Is the input only 2D image or 2D image + ground truth object bounding boxes?
In order to make sure that reader understands each qualitative result, there should be a column showing the “Input” to the pipeline (Not “Image”). For example, in Figure 3 and Figure 4, the image overlaid with input ground-truth bounding boxes should be shown as input to the algorithm.
*The experiments and results does not convey the effectiveness of the proposed approach. There are major issues with the quality of the experiments and results. Here are several examples:
- Missing baseline: Comparison with the CRF-based baseline is missing. This statement is not convincing in the introduction: “One classical approach is to use graphical models such as CRFs. However, these classical approaches have usually provided little improvements over object-based approaches.” For a fair comparison with prior works, reporting results on a CRF-based baseline using similar unary predictions is necessary.
-The experimental results are heavily based on ground truth boxes for the objects, but it is not clear how/where the ground truth boxes are given at the test time and which part is actually predicted.
-If the ground truth boxes are given at the test time, it means that the ground truth binary relations between objects are given and it makes the problem trivial.
-It is not clear what is the ground truth box in experimental setup. Is it amodal object box or the ground truth box contains only the visible part of the object?
-The qualitative results shown in Figure 4 have full objects in voxel space with predicted rotation, scale and translation. In the qualitative result of Figure 3 and Figure 5 the voxel prediction is shown as final output. Why the result of full object in voxel space with predicted (rotation, scale and translation) is not shown in Figure 3 and Figure 5 and why it is shown in Figure 4?
*Very limited results on real images:
-Quantitative result on a dataset of real images is missing. The results on synthetic datasets is not a good proxy for the actual performance of the algorithm in real use cases and applications.
- The paper only shows few results of NYUv2 on known ground truth boxes. The errors in object detection can be propagated to the 3D estimation therefore these qualitative results are not representative of the actual qualitative performance of the proposed algorithm. Several randomly selected qualitative results on a dataset of real images “without ground-truth boxes” are needed for evaluating the performance of the proposed method on real images.
-Reporting variation in all parameters of scale, rotation and translation is necessary in order to find the difficulty of the problem. For example, what is the distribution of object scale in different object categories. What is the error of scale prediction of we use mean object scale for each object category for all object instance at test set?
*Unclear statements and presentation:
- It is mentioned in the paper: “While the incorporation of unary and relative predictions can be expressed via linear constraints in the case of translation and scale, a similar closed form update does not apply for rotation because of the framing as a classification task and non-linearity of the manifold.”
-Is it necessary for the relative rotation to be formulated to classification task?
-If not the comparison of modeling relative rotation via linear constraints is missing.
- In some of the tables and figures the “know ground-truth boxes/detection setting” are in bold face and in some cases are not. This should be consistent throughout the paper.


Deep Neural Networks (DNNs) generalize well despite their massive size and capability of memorizing all examples. There is a hypothesis that DNNs start learning from simple patterns based on the observations that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite the importance of understanding the learning dynamics of DNNs, properties of easy and hard examples are not fully investigated. In this paper, we study the similarities of easy and hard examples respectively among different CNNs, assessing those examples contributions to generalization. Our results show that most easy examples are identical among different CNNs, as they share similar dataset-dependent patterns (e.g., colors, structures, and superficial cues in high-frequency). Moreover, while hard examples tend to contribute more to generalization than easy examples, removing a large number of easy examples leads to poor generalization, and we find that most misclassified examples in validation dataset are hard examples. By analyzing intriguing properties of easy and hard examples, we discover that the reason why easy and hard examples have such properties can be explained by biases in a dataset and Stochastic Gradient Descent (SGD).

The paper formulates a definition of easy and hard examples and studies the properties and the training implications of such examples. The paper does not attempt to present insights that change training for the better (although suggests this could be future work), so the primary value it claims to add is our understanding of neural networks. I think the paper presents findings that most deep learning practitioners already find intuitive, which is why I think the paper falls short in its primary mission. An exposé like this could be valuable for an introductory text in deep learning, but I do not think the analysis meets the bar for a cutting edge insight and do not think it should be accepted.
Strengths:
- Quantifying easy and hard and using that as a starting point for further analysis is not a bad starting point at all.
- The experiments form a good starting point for interesting analysis.
- The paper is easy to follow and understand.
Weaknesses:
- The biggest weakness is that I just didn't find any of conclusions from this study to be that surprising or interesting. I think it's pretty obvious that neural networks start by learning the most immediately discriminative features ("frequent patterns"). The visual examples of easy and hard examples are not surprising at all (I think you get similar clusters if you just show bottom and top of model confidences, which I think many of us have). In section 7.1, the result that most misclassified examples are hard examples is presented as a surprising result. This is confusing, because this exactly what I would have expected given how you define easy/hard. It would be far more surprising if misclassified examples were all considered easy under your definition.
- The paper only scratches the surface. In Figure 2, the results for the two different datasets are quite different. This means only two datasets is probably not enough for us to understand what is going on here in general. Just the conclusion that datasets may be different in terms of easy/hard samples does not take the analysis far enough. It's also unclear what the reader should make of these conclusions.
- In Table 1, let's say the bottom 30% of samples are actually equally easy. This would mean that the "easy" examples are just a random 1/3 of those samples. Basically, I'm worried about the implications of having a hard cut-off at 10% and if there are situations where the bottom 10% actually changed quite a bit, but the broader picture of easy really didn't change that much. I guess I'm saying that I didn't quite gain confidence that definitions of easy/hard and matching rate are the correct way to go here and there might be a better metric that can look at the continuum of easy/hard from e = 0 to 1. You could have some kind of distance function where if an example moved from 5th percentile to 12th percentile, it would constitute a distance of 7. This is perhaps not the right thing either, but presenting an alternative metric and showing that the numbers (up to scale) and conclusions are unchanged would be nice.
Other comments:
- The new terminology of "contradicted pattern" and "non-contradicted pattern" is a bit confusing. Why aren't you just calling these "non-discriminative" and "discriminative"? If a mantis and a ladybird are both typically on a leaf, the leaf is not discriminative for this task. However, if a mantis and a boat are typically on differently colored backgrounds, the background is discriminative.
Minor comments:
- page 1, "easy and hard examples differ on various CNNs architectures" -> "CNN"
- page 2, "as a criteria" -> "criterion"
- page 2, "We then redefine easy and hard" -> don't you mean just "define"? Or do you mean that the words already have casual meanings, so this is a redefinition? I still think "define" is less confusing here.
- page 2, I think it's confusing that both easy and hard use the threshold \tau, suggesting it is the same. Maybe put a subscript to make it clear that the two \taus are different.
- page 6, "accuracy does not drop" -> could use a "does not *even* drop" for clarity
- page 7, "7.1 Do misclassified examples in validation dataset are hard examples": "Do"->"Are", remove "are"


NEURAL GRAPH EVOLUTION: TOWARDS EFFICIENT AUTOMATIC ROBOT DESIGN
Despite the recent successes in robotic locomotion control, the design of robots, i.e., the design of their body structure, still heavily relies on human engineering. Automatic robot design has been a long studied subject, however, progress has been slow due to large combinatorial search space and the difficulty to efficiently evaluate the candidate structures. Note that one needs to both, search over many possible body structures, and choose among them based on how the robot with that structure performs in an environment. The latter means training an optimal controller given a candidate structure, which in itself is costly to obtain. In this paper, we propose Neural Graph Evolution (NGE), which performs evolutionary search in graph space, by iteratively evolving graph structures using simple mutation primitives. Key to our approach is to parameterize the control policies with graph neural networks, which allows us to transfer skills from previously evaluated designs during the graph search. This significantly reduces evaluation cost of new candidates and makes the search process orders of magnitude more efficient than that of past work. In addition, NGE applies Graph Mutation with Uncertainty (GM-UC) by incorporating model uncertainty, which reduces the search space by balancing exploration and exploitation. We show that NGE significantly outperforms previous methods in terms of convergence rate and final performance. As shown in experiments, NGE is the first algorithm that can automatically discover kinematically preferred robotic graph structures, such as a fish with two symmetric flat side-fins and a tail, or a cheetah with athletic front and back legs. NGE is extremely efficient, it finds plausible robotic structures within a day on a single 64 CPU-core Amazon EC2 machine.

This paper proposes an approach for automatic robot design based on Neural graph evolution.
The overall approach has a flavor of genetical algorithms, as it also performs evolutionary operations on the graph, but it also allows for a better mechanism for policy sharing across the different topologies, which is nice.
My main concern about the paper is that, currently, the experiments do not include any strong baseline (the ES currently is not a strong baseline, see comments below).
The experiments currently demonstrate that optimizing both controller and hardware is better than optimizing just the controller, which is not surprising and is a phenomenon which has been previously studied in the literature.
What instead is missing is an answer to the question: Is it worth using a neural graph? what are the advantages and disadvantages compared to previous approaches?
I would like to see additional experiments to answer this questions.
In particular, I believe that any algorithms you compare against, you should optimize both G and theta, since optimizing purely the hardware is unfair.
You should use an existing ES implementation (e.g., from some well-known package) instead of a naive implementation, and as additional baseline also CMA-ES.
If you can also compare against one or two algorithms of your choice from the recent literature it would also give more value to the comparison.
Detailed comments:
- in the abstract you say that "NGE is the first algorithm that can automatically discover complex robotic graph structures". This statement is ambiguous and potentially unsupported by evidence. how do you define complex? that can or that did discover?
- in the introduction you mention that automatic robot design had limited success. This is rather subject, and I would tend to disagree.  Moreover, the same limitations that apply to other algorithms to make them successful, in my opinion, apply to your proposed algorithm (e.g., difficulty to move from simulated to real-world).
- The digression at the bottom of the first page about neural architecture search seem out of context and interrupts the flow of the introduction. What is the point that you are trying to make? Also, note that some of the algorithms that you are citing there have indeed applied beyond architecture search, eg. Bayesian optimization is used for gait optimization in robotics, and Genetic algorithms have been used for automatic robot design.
- The stated contributions number 3 and 5 are not truly contributions. #3 is so generic that a large part of the previous literature on the topic fall under this category -- not new. #5 is weak, and tell us more about the limitations of random search and naive ES than necessarily a merit of your approach.
- Sec 2.2: "(GNNs) are very effective" effective at what? what is the metric that you consider?
- Sec 3 "(PS), where weights are reused" can you already go into more details or refer to later sections?
- First line page 4 you mention AF, without introducing the acronym ever before.
- Sec 3.1: the statements about MB and MF algorithms are inaccurate. Model-based RL algorithms can work in real-time (e.g. http://proceedings.mlr.press/v78/drews17a/drews17a.pdf) and have been shown to have same asymptotic performance of MB controllers for simple robot control (e.g. https://arxiv.org/abs/1805.12114)
- "to speed up and trade off between evaluating fitness and evolving new species" Unclear sentence. speed up what? why is this a trade-off?
- Sec 3.4 can you recap all the parameters after eq.11? going through Sec 3.2 and 2.2 to find them is quite annoying.
- Sec 4.1:  would argue that computational cost is rarely a concern among evolutionary algorithms. The cost of evaluating the function is typically more pressing, and as a result it is important to have algorithms that can converge within a small number of iterations/generations.
- Providing the same computational budget seem rather arbitrary at the moment, and it heavily depends from implementation. How many evaluations do you perform for each method? why not having the same budget of experiments?


THE DEEP WEIGHT PRIOR
Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution. In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (dwp), that exploit generative models to encourage a specific structure of trained convolutional filters e.g., spatial correlations. We define dwp in a form of an implicit distribution and propose a method for variational inference with such type of implicit priors. In experiments, we show that dwp improves the performance of Bayesian neural networks when training data are limited, and initialization of weights with samples from dwp accelerates training of conventional convolutional neural networks.

Summary:
This paper proposes the ‘deep weight prior’: the idea is to elicit a prior on an auxiliary dataset and then use that prior over the CNN filters to jump start inference for a data set of interest.  Both explicit and implicit priors are considered, with the latter having the benefit of increased flexibility but having the drawback of a lack of a parametric form to plug in to the ELBO.  The authors address this last point by extending the ELBO appropriately.  Experiments are performed testing the prior’s ability to capture trained filters (Figure 1), provide a good initialization (Figure 2), improve sample efficiency (Figure 3), improve training speed (Figure 4).
Pros:
I like this paper: it is a intuitive idea, and the experiments explore exactly what one would hope to gain from the prior (i.e. better initialization, improved sample efficiency).  I find the paper clearly written and to have a logical flow.  Furthermore, I think eliciting priors---while so crucial in more traditional Bayesian modeling---has been mostly overlooked by the Bayesian ML community, and this paper clearly shows that there are gains to be had from a fairly straightforward procedure.
Cons:
The only potential issue with the paper is the use of the implicit prior, as it complicates variational inference, requiring the extension to the ELBO described in Section 3.2.  As far as I can tell, all experiments use the implicit priors.  I would have liked to have seen an experiment using a parametric prior (eg Gaussian) that shows what gains the implicit prior provides.  Or is it simply a matter of memory efficiency?
Other comments:
-- Nice first sentence in the introduction!  I like how it’s a general statement but immediately focuses the reader’s attention to the paper’s topic.
-- While it doesn’t say so explicitly, the paper seems to imply it is the first to use implicit priors.  Some previous work that uses some form of implicit prior includes:
Runs a chain to refine the prior: Alex Lamb et al. "GibbsNet: Iterative Adversarial Inference for Deep Graphical Models." Advances in Neural Information Processing Systems. 2017.
Optimizes a NN implicit prior based on an invariance objective: Eric Nalisnick and Padhraic Smyth. "Learning priors for invariance." International Conference on Artificial Intelligence and Statistics. 2018.
Defines implicit priors over functions through samplers: Chao Ma, Yingzhen Li, and José Miguel Hernández-Lobato. "Variational Implicit Processes." arXiv preprint arXiv:1806.02390 (2018).
Evaluation:  I recommend this paper for acceptance.  It is a sensible idea with pointed experimental validation.


Generative Adversarial Networks (GANs) have shown great results in accurately modeling complex distributions, but their training is known to be difficult due to instabilities caused by a challenging minimax optimization problem. This is especially troublesome given the lack of an evaluation metric that can reliably detect non-convergent behaviors. We leverage the notion of duality gap from game theory in order to propose a novel convergence metric for GANs that has low computational cost. We verify the validity of the proposed metric for various test scenarios commonly used in the literature.

In this paper, the authors proposed the duality gap as the criterion for evaluating the training of GAN. To justify the proposed criterion, the authors designed empirical experiments on both synthetic and real-world datasets to demonstrate the ability of the duality gap for detecting divergence, mode collapse, sample equality, as well as the generalization to other application domains besides image generation. Comparing with the existing criteria, e.g., FID and INC, the duality gap shows better ability and computational efficiency.
However, the paper ignores rich literatures in optimization that uses the duality gap as the criterion for characterizing the convergence of algorithms for min-max saddle point problem, e.g., [1]. In fact, in optimization community, using duality gap to screening the convergence on saddle point problem is a common knowledge. [1] even provides the finite-step convergence rate when the saddle point problem is convex-concave. This paper is only introducing that into machine learning community. Therefore, the novelty of the paper seems not enough.
Secondly, the duality gap is only able to screen the optimization convergence and the solution quality w.r.t. **the same objective**. It is not valid to compare different GANs with different losses function using the duality gap. Theoretically, for any loss function derived from some divergences, e.g. [2], the global optimal solution  can always achieve zero duality gap. In other words, for different GANs, with different objectives, the duality gap cannot distinguish which one is better. In such sense, the title is very misleading.
Thirdly, how the evaluate such criterion in practice in GAN scenario is not clearly explained. Considering the neural network parametrization of both the generator and discriminator, the argmax_v M(u, v) and argmin_u M(u, v) is not tractable. Without the optimal solution, what is the meaning of the ``duality gap'' should be explained. What will happen if we only obtain the suboptimal solutions which themselves are model collapsed? Without such discussion in both theoretical and/or empirical aspects, I am not very convincing about the conclusion.
Finally, if one follows the Fenchel dual view of GAN in [2, 3], the min-max is the variational form of some divergences, which the GANs are directly optimizing. It is straightforwardly to see the better min-max value is, the smaller divergence between generated samples and ground-truth is, and thus, the better quality of the generator is. The fact that min-max objective is indeed able to characterize the quality of generator is obvious and well-known. Otherwise, there is not need to use such objective in the optimization to train the model.
[1] Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. (2009). Robust stochastic approximation approach to stochastic programming. SIAM J. on Optimization, 19(4):1574–1609.
[2]  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), pages 2672–2680, 2014.
[3] S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. arXiv:1606.00709, 2016


In Reinforcement Learning (RL), an agent explores the environment and collects trajectories into the memory buffer for later learning. However, the collected trajectories can easily be imbalanced with respect to the achieved goal states. The problem of learning from imbalanced data is a well-known problem in supervised learning, but has not yet been thoroughly researched in RL. To address this problem, we propose a novel Curiosity-Driven Prioritization (CDP) framework to encourage the agent to over-sample those trajectories that have rare achieved goal states. The CDP framework mimics the human learning process and focuses more on relatively uncommon events. We evaluate our methods using the robotic environment provided by OpenAI Gym. The environment contains six robot manipulation tasks. In our experiments, we combined CDP with Deep Deterministic Policy Gradient (DDPG) with or without Hindsight Experience Replay (HER). The experimental results show that CDP improves both performance and sampleefficiency of reinforcement learning agents, compared to state-of-the-art methods.

This paper addresses a problem that arises in "universal" value-function approximation (that is, reinforcement-learning when a current goal is included as part of the input);  when doing experience replay, the experience buffer might have much more representation of some goals than others, and it's important to keep the training appropriately balanced over goals.
So, the idea is to a kind of importance weighting of the trajectory memory, by doing a density estimation on the goal distribution represented in the memory and then sample them for training in a way that is inversely related to their densities.  This method results in a moderate improvement in the effectiveness of DDPG, compared to the previous method for hindsight experience replay.
The idea is intuitively sensible, but I believe this paper falls short of being ready for publication for three major reasons.
First, the mechanism provided has no mathematical justification--it seems fairly arbitrary.   Even if it's not possible to prove something about this strategy, it would be useful to just state a desirable property that the sampling mechanism should have and then argue informally that this mechanism has that property.  As it is, it's just one point in a large space of possible mechanisms.
I have a substantial concern that this method might end up assigning a high likelihood of resampling trajectories where something unusual happened, not because of the agent's actions, but because of the world having made a very unusual stochastic transition.   If that's true, then this distribution would be very bad for training a value function, which is supposed to involve an expectation over "nature"'s choices in the MDP.
Second, the experiments are (as I understand it, but I may be wrong) in deterministic domains, which definitely does not constitute a general test of a proposed RL  method.
- I'm not sure we can conclude much from the results on fetchSlide (and it would make sense not to use the last set of parameters but the best one encountered during training)
- What implementation of the other algorithms did you use?
Third, the writing in the paper has some significant lapses in clarity.  I was a substantial way through the paper before understanding exactly what the set-up was;  in particular, exactly what "state" meant was not clear.  I would suggest saying something like s = ((x^g, x^c), g) where s is a state from the perspective of value iteration, (x^g, x^c) is a state of the system, which is a vector of values divided into two sub-vectors, x^g is the part of the system state that involves the state variables that are specified in the goal, x^c (for 'context') is the rest of the system state, and g is the goal.  The dimensions of x^g and g should line up.
- This sentence  was particularly troublesome:  "Each  state s_t also includes the state of the achieved goal, meaning the goal state is a subset of the normal state.  Here, we overwrite the notation s_t  as the achieved goal state, i.e., the state of the object."
- Also, it's important to say what the goal actually is, since it doesn't make sense for it to be a point in a continuous space.  (You do say this later, but it would be helpful to the reader to say it earlier.)