Federated learning is a recent advance in privacy protection. In this context, a trusted curator aggregates parameters optimized in decentralized fashion by multiple clients. The resulting model is then distributed back to all clients, ultimately converging to a joint representative model without explicitly having to share the data. However, the protocol is vulnerable to differential attacks, which could originate from any party contributing during federated optimization. In such an attack, a client's contribution during training and information about their data set is revealed through analyzing the distributed model. We tackle this problem and propose an algorithm for client sided differential privacy preserving federated optimization. The aim is to hide clients' contributions during training, balancing the trade-off between privacy loss and model performance. Empirical studies suggest that given a sufficiently large number of participating clients, our proposed procedure can maintain client-level differential privacy at only a minor cost in model performance.

The main claim the authors make is that providing privacy in learning should go beyond just privacy for individual records to providing privacy for data contributors which could be an entire hospital. Adding privacy by design to the machine learning pipe-line is an important topic. Unfortunately, the presentation of this paper makes it hard to follow.
Some of the issues in this paper are technical and easy to resolve, such as citation format (see below) or consistency of notation (see below). Another example is that although the method presented here is suitable only for gradient based learning this is not stated clearly. However, other issues are more fundamental:
1.	The main motivation for this work is providing privacy to a client which could be a hospital as opposed to providing privacy to a single record – why is that an important task? Moreover, there are standard ways to extend differential privacy from a single record to a set of r records (see dwork & Rote, 2014 Theorem 2.2), in what sense the method presented here different than these methods?
2.	Another issue with the hospitals motivation is that the results show that when the number of parties is 10,000 the accuracy is close to the baseline. However, there are only 5534 registered hospitals in the US in 2018 according to the American Hospital Association (AHA): https://www.aha.org/statistics/fast-facts-us-hospitals. Therefore, are the sizes used in the experiments reasonable?
3.	In the presentation of the methods, it is not clear what is novel and what was already done by Abadi et al., 2016
4.	The theoretical analysis of the algorithm is only implied and not stated clearly
5.	In reporting the experiment setup key pieces of information are missing which makes the experiment irreproducible. For example, what is the leaning algorithm used? If it is a neural network, what was its layout? What type of cross validation was used to tune parameters?
6.	In describing the experiment it says that “For K\in\{1000,10000} data points are repeated.” This could mean that a single client holds the same point multiple times or that multiple clients hold the same data point. Which one of them is correct? What are the implications of that on the results of the experiment?
7.	Since grid search is used to tune parameters, more information is leaking which is not compensated for by, for example, composition bounds
8.	The results of the experiments are not contrasted against prior art, for example the results of Abadi et al., 2016.
Additional comments
9.	The introduction is confusing since it uses the term “federated learning” as a privacy technology. However federated learning discusses the scenario where the data is distributed between several parties. It is not necessarily the case that there are also privacy concerns associated, in many cases the need for federated learning is due to performance constraints.
10.	In the abstract the term “differential attacks” is used – what does it mean?
11.	“An independent study McMahan et al. (2018), published at the same time”- since you refer to the work of McMahan et al before your paper was reviewed, it means that the work of McMahan et al came out earlier.
12.	In the section “Choosing $\sigma$ and $m$” it is stated that the higher \sigma and the lower m, the higher the privacy loss. Isn’t the privacy loss reduced when \sigma is larger? Moreover, since you divide the gradients by m_t then the sensitivity of each party is of the order of S/m and therefore it reduces as m gets larger, hence, the privacy loss is smaller when m is large.
13.	At the bottom of page 4 and top of page 5 you introduce variance related terms that are never used in the algorithm or any analysis (they are presented in Figure 3). The variance between clients can be a function of how the data is split between them. If, for example, each client represents a different demography then the variance may be larger from the beginning.
14.	In the experiments (Table 1), what does it mean for \delta^\prime to be e-3, e-5 or e-6? Is it 10^{-3}, 10^{-5} and 10^{-6}?
15.	The methods presented here apply only for gradient descent learning algorithms, but this is not stated clearly. For example, would the methods presented here apply for learning tree based models?
16.	The citations are used incorrectly, for example “sometimes referred to as collaborative Shokri & Shmatikov (2015)” should be “sometimes referred to as collaborative (Shokri & Shmatikov, 2015)”. This can be achieved by using \citep in latex. This problem appears in many places in the paper, including, for example, “we make use of the moments accountant as proposed by Abadi et al. Abadi et al. (2016).” Which should be “we make use of the moments accountant as proposed by Abadi et al. (2016).” In which case you should use only \cite and not quote the name in the .tex file.
17.	“We use the same deﬁnition for differential privacy in randomized mechanisms as Abadi et al. (2016):” – the definition of differential privacy is due to Dwork, McSherry, Nissim & Smith, 2006
18.	Notation is followed loosely which makes it harder to follow at parts. For example, you use “m_t” for the number of participants in time t but in some cases,  you use only m as in “Choosing $\sigma$ and $m$”.
19.	In algorithm 1 the function ClientUpdate receives two parameters however the first parameter is never used in this function.
20.	Figure 2: I think it would be easier to see the results if you use log-log plot
21.	Discussion: “For K=10000, the differrntially private model almost reaches accuracies of the non-differential private one.” – it is true that the model used in this experiment achieves an accuracy of 0.97 without DP and the reported number for K=10000 is 0.96 which is very close. However, the baseline accuracy of 0.97 is very low for MNIST.
22.	In the bibliography you have Brendan McMahan appearing both as Brendan McMahan and H. Brendan McMahan
It is possible that underneath that this work has some hidden jams, however, the presentation makes them hard to find.


VARIATIONAL BAYESIAN PHYLOGENETIC INFERENCE
Bayesian phylogenetic inference is currently done via Markov chain Monte Carlo with simple mechanisms for proposing new states, which hinders exploration efficiency and often requires long runs to deliver accurate posterior estimates. In this paper we present an alternative approach: a variational framework for Bayesian phylogenetic analysis. We approximate the true posterior using an expressive graphical model for tree distributions, called a subsplit Bayesian network, together with appropriate branch length distributions. We train the variational approximation via stochastic gradient ascent and adopt multi-sample based gradient estimators for different latent variables separately to handle the composite latent space of phylogenetic models. We show that our structured variational approximations are flexible enough to provide comparable posterior estimation to MCMC, while requiring less computation due to a more efficient tree exploration mechanism enabled by variational inference. Moreover, the variational approximations can be readily used for further statistical analysis such as marginal likelihood estimation for model comparison via importance sampling. Experiments on both synthetic data and real data Bayesian phylogenetic inference problems demonstrate the effectiveness and efficiency of our methods.

This paper explores an approximate inference solution to the challenging problem of Bayesian inference of phylogenetic trees. Its leverages recently proposed subsplit Bayesian networks (SBNs) as a variational approximation over tree space and combines this with modern gradient estimators for VI. It is thorough in its evaluation of both methodological considerations and different datasets.
The main advantage would seem to be a large speedup over MCMC-based methods (Figure 4), which could be of significant value to the phylogenetics community. This point would benefit from more discussion. How do the number of iterations (reported in Figures 3&4, which was done carefully) correspond to wallclock time? Can this new method scale to numbers of sites and sequences that were previously unfeasible?
The main technical contribution is the use of SBNs as variational approximations over tree-space, but it is difficult to follow their implementation and parameter sharing without the explanation of the original paper. Additionally, the issue of estimating the support of the subsplit CPTs needs more discussion. As the authors acknowledge, complete parameterizations of these models scale in a combinatorial way with “all possible parent-child subsplit pairs”, and they deal with this by shrinking the support up front with various heuristics. It seems that these support estimation approaches would be feasible when the data are strong but would become challenging to scale when the data are weak. Since VB is often concerned with the limited-data regime, more discussion of when support estimation is feasible and when it is difficult would clarify how widely applicable the method is.
Overall, this work is an interesting extension of variational Bayes to a tree-structured inference problem and is thorough in its evaluation. While it is a bit focused on classical inference for ICLR, it could be interesting both for the VI community and as a significant application advancement.
Other notes:
In table 1, is the point that all methods are basically the same with different variance? This is not clear from the text. What about the variational bounds?


When applying machine learning to sensitive data one has to balance between accuracy, information leakage, and computational-complexity. Recent studies have shown that Homomorphic Encryption (HE) can be used for protecting against information leakage while applying neural networks. However, this comes with the cost of limiting the kind of neural networks that can be used (and hence the accuracy) and with latency of the order of several minutes even for relatively simple networks. In this study we improve on previous results both in the kind of networks that can be applied and in terms of the latency. Most of the improvement is achieved by novel ways to represent the data to make better use of the capabilities of the encryption scheme.

The paper proposes improvements on the area of neural network inference with homomorphically encrypted data, where a major drawback in current applications is the high computational cost.
The paper makes several minor contributions and, in my opinion, it fails to provide an unified message to the reader. The arguably independent contributions are:
1. the use of the faster SAEL 3.2.1 over CryptoNet — not really an innovation per se
2. flexible representation of message space — the main contribution
3. a much more complex network than CryptoNet for an experiment on CIFAR10 — minor contribution/application of 2
4. using pre-trained model for embedding images before encryption — minor contribution
I think the authors should refocus this work on point 2 and 3 only. 1 could simply be a note in the paper, as not a a real contribution and 4 may be even excluded as it goes toward a different direction.
I will mainly focus on the central contribution of the paper — the data representation — which in my opinion has the potential to progress this area further. Unfortunately, the current quality of presentation is suboptimal. The choices on architecture and intermediate representations of LoLa in Section 4 are hard to follow, and it is not clear to me when the authors are making a choice or satisfying a constraint. That is, for example when we aim to vs. we have to use one representation in place of another? Since this is the main contribution, I suggest the author to help the presentation with diagrams and formulae in this Section. Each component should be presented in modular fashion. In fact, this is what the authors did in Section 3. Yet, towards reuse of these concepts for machine learning applications, the authors could present their improvements as applied to *neural network layers*. E.g. responding to answers such as: what are the most efficient data representation in input and output if we want to compute a convolutional layer? It would be also interesting to break down cost requirements for each intermediate computation, to highlight bottlenecks and compare alternative choices.
Incidentally, LoLa-Conv improvements appear to be simply due to the use of a conv vs. dense layer as input.
Section 6 explains the use of a pre-trained AlexNet to embed data for learning a linear model (HE-aware) for Caltech 101. In the context of application of HE, the idea is novel to my knowledge, although it is a rather common practice in deep learning. The number reported in this section have no baseline to compare with and therefore it’s hard to evaluate their significance.
A missing reference is [A].
Minors
* section 4: “this saves a processing step which saves"
* typos in section 6: “a networks”, “in medical imagine"
* footnote 4: “convinient"
[A] TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service.


We propose sequenced-replacement sampling (SRS) for training deep neural networks. The basic idea is to assign a fixed sequence index to each sample in the dataset. Once a mini-batch is randomly drawn in each training iteration, we refill the original dataset by successively adding samples according to their sequence index. Thus we carry out replacement sampling but in a batched and sequenced way. In a sense, SRS could be viewed as a way of performing "mini-batch augmentation". It is particularly useful for a task where we have a relatively small images-per-class such as CIFAR-100. Together with a longer period of initial large learning rate, it significantly improves the classification accuracy in CIFAR-100 over the current state-of-the-art results. Our experiments indicate that training deeper networks with SRS is less prone to over-fitting. In the best case, we achieve an error rate as low as 10.10%.

In this paper, the authors introduce a sampling strategy that aims to combine the benefits of with- and without-replacement SGD. With-replacement strategies add more randomness to the process, which the authors claim helps convergence, while the without-replacement strategies ensure equal usage of all datapoints. The authors present numerical results showing better convergence and improved final accuracy. While I found the idea appealing, I felt that the paper needs more work before it can be published. I detail some of my primary concerns below:
- The entire motivation of the paper is predicated on the hypothesis that more randomness is better for training. This is not generally true. Past work has shown that specific kinds of random noise aid convergence through exploration/saddle point avoidance/escaping spurious minima while others either make no change, or hurt. Noise from sampling tends to be a structured noise that aids exploration/convergence over batch gradient descent, but it is not immediately clear to me why the choice between with- and without-replacement should imply exploration.
- Maybe it's obvious but I'm not grasping why, mathematically, the number of accessible configurations for SRS is the same as original replacement sampling (4th paragraph on page 3).
- Given that the central motivation of the work was to enable with-replacement strategies while still ensuring equal usage, I recommend that the authors include a histogram of datapoint usage for three strategies (with, without, hybrid). This should help convince the reader that the SRS indeed improves upon the usage statistics of with replacement.
- If one were to create a hybrid sampling strategy, one that is natural is doing 50-50 sampling with and without replacement. In other words, for a batch size of 64, say, 32 are sampled with replacement and 32 without. By changing the ratio, you can also control what end of the sampling spectrum you want to be on. Did you try such a strategy?
- For the numerical experiments, as I see it, there are 3 differences between the SRS setup and the baseline: location of batch normalization, learning rate, and batch size. The authors show (at the bottom of Page 6) that the performance boost does not come from learning rate or mini-batch size, but what about the placement of the BN layer? Seems like that still remains as a confounding factor?
- "SRS leads to much more fluctuations, and hence significantly more covariate shift". How do the authors define covariate shift? Can the authors substantiate this claim theoretically/empirically?
- The authors claim that the method works better when the dataset size is low compared to number of classes. Again, can the authors substantiate this claim theoretically/empirically? Maybe you can try running a sub-sampled version of CIFAR-10/100 with the baselines?
- The writing in the paper needs improving. A few sample phrases that need editing: "smaller mini-batch means a larger approximation", "more accessible configurations of mini-batches", "hence more exploration-induction", "less optimal local minimum"
- Minor comment: why is the queue filled with repeated samples? In Figure 1, why not have the system initialized with 1 2 3 in the pool and 4 5 in the queue? Seems like by repeating, there is an unnecessary bias towards those datapoints.


Likelihood-based Permutation Invariant Loss Function for Probability Distributions
We propose a permutation-invariant loss function designed for the neural networks reconstructing a set of elements without considering the order within its vector representation. Unlike popular approaches for encoding and decoding a set, our work does not rely on a carefully engineered network topology nor by any additional sequential algorithm. The proposed method, Set Cross Entropy, has a natural information-theoretic interpretation and is related to the metrics defined for sets. We evaluate the proposed approach in two object reconstruction tasks and a rule learning task.

This paper proposes an objective function for sets autoencoders such that the loss is permutation invariant with respect to the order of reconstructed inputs. I think that the problem of autoencoding sets is important and designing custom loss functions is a good way to approach it. Thus, I quite like the idea of SCE  from that point of view. However, I find the experiments not convincing for me to accept the paper.
While reading Section 3, I found it hard to keep in mind that x and y are discrete probability distributions and the notation like P(x=y) is not making things easier. Actually, I’ve never seen cross entropy written with P(x=y). Though is my personal opinion and I don’t have a suggestion on how to improve the explanations in Eq. 1-8. However, I’m glad there is an example at the end of Section 3.
I have some comments on the Experiments section.
* Puzzles:
(1) Figure 1 could have been prettier.
(2) The phrase “The purpose of this experiment is to reproduce the results from (Zaheer et al., 2017)” makes little sense to me.  In Deep Sets, there are many experiments and it’s not clear which experiment is meant here.
(3) Table 1 gives test error statistics for 10 runs. What is changed in every run? Does the test set stay the same in every run or is a kind of a cross-validation? Or is it just a different random seed for the initial weights? I could not find an explanation in the text, so there is no way I can interpret the results.
* Blocksworld: the reconstructions are nice, but the numbers in Table 2 are difficult to interpret.
For example, I cannot estimate how important the difference of 10 points in SH scores is.
* Rule learning ILP tasks: I don’t know enough about learning logic rules tasks to comment on those experiments, but Table 3 seems overwhelming and the concept of 10 runs is still unclear.
--- General comment on the experiments ---
I think an important goal of any autoencoder is to learn a representation that can be useful in other tasks. There is even an example in the paper: “set representation of the environment is crucial in the robotic systems”. Thus, the experiments I would like to see are about evaluating the quality of a representation from an SCE-trained autoencoder compared to other training methods.  Without those experiments, I cannot estimate how valuable the SCE loss function is.


Various architectural advancements in the design of recurrent neural networks (RNN) have been focusing on improving the empirical stability and representability by sacrificing the complexity of the architecture. However, more remains to be done to fully understand the fundamental trade-off between these conflicting requirements. Towards answering this question, we forsake the purely bottom-up approach of data-driven machine learning to understand, instead, the physical origin and dynamical properties of existing RNN architectures. This facilitates designing new RNNs with smaller complexity overhead and provable stability guarantee. First, we define a family of deep recurrent neural networks, n-t-ORNN, according to the order of nonlinearity n and the range of temporal memory scale t in their underlying dynamics embodied in the form of discretized ordinary differential equations. We show that most of the existing proposals of RNN architectures belong to different orders of n-t-ORNNs. We then propose a new RNN ansatz, namely the Quantum-inspired Universal computing Neural Network (QUNN), to leverage the reversibility, stability, and universality of quantum computation for stable and universal RNN. QUNN provides a complexity reduction in the number of training parameters from being polynomial in both data and correlation time to only linear in correlation time. Compared to Long Short Term Memory (LSTM), QUNN of the same number of hidden layers facilitates higher nonlinearity and longer memory span with provable stability. Our work opens new directions in designing minimal RNNs based on additional knowledge about the dynamical nature of both the data and different training architectures.

This paper attempts to do three things:
	1) introduce a generalization / formalism for describing RNN architectures
	2) demonstrate how various popular RNN architectures fit within the proposed framework
	3) propose a new RNN architecture within the proposed framework that overcomes some limitations in LSTMs
The ultimate goal of this work is to develop an architecture that:
	1) is better able to model long-term dependencies
	2) is stable and efficient to train
Some strengths, concerns, and questions loosely ordered by section:
Stable RNNs
	- it's not clear to me where equations (2) and (3) come from; what is the motivation? Is it somehow derived from this Runge-Kutta method (I'm not familiar with it)?
	- I don't understand what this t^th order time-derivative amounts to in practice. A major claim (in Table 4) is that LSTMs are time-order 2 whereas QUNNs are time-order L and the implication is that this means LSTMs are worse at modeling long term structure than QUNNs ; but how does that actually relate to practical ability to model long-term dependencies? It certainly doesn't seem correct to me to say that LSTMs can only memorize sequences of length 2, so I don't know why we should care about this time-derivative order.
	- I thought this section was poorly written. The notation was poorly chosen at times, e.g. the t_k notation and the fact that some $l$ have subscripts and some don't. There were also some severe typos, e.g. I think Claim 1 should be "L-2-ORNN". Furthermore, there were crucially omitted definitions: what is reversibility and why should we care? Relatedly, the "proofs" are extremely hand-wavy and just cite unexplained methods with no further information.
QUNNs
	- The practical difference between QUNNs and LSTMs seems to be that the weights of the QUNN are dynamic within a single forward prop of the network, whereas LSTM weights are fixed given a single run (although the author does admit that the forget gates adds some element of dynamism, but there's no concrete evidence to draw conclusions about differences in practice).
	- I don't understand the repeated claim that LSTMs don't depend on the data; aren't the weights learned from data?

There may be something interesting in this paper, but it's not clear to me in its current incarnation and I'm not convinced that an eight-page conference paper is the right venue for such a work. There's a substantial of amount of exposition that needs to be there and is currently missing. I suspect the author knows this, but due to space constraints had to omit a lot of definitions and explanations.
I don't think all papers need experiments, but this paper I think would have greatly benefited from one. Community knowledge of LSTMs has reached a point where they are in practice easy to train and fairly stable (though admittedly with a lot of tricks). It would have been much more convincing to have simple examples where LSTMs fail due to instability and QUNNs succeed. Similarly, regarding long-term dependencies, my sense is that LSTMs are able to model some long-term dependencies. Experimental evidence of the gains offered by QUNNs would have also been very convincing.
Note: It looks like there's some funkiness in the tables on page 8 to fit into the page limit.


Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answering over a context. Furthermore, we present a new multitask question answering network (MQAN) that jointly learns all tasks in decaNLP without any task-specific modules or parameters more effectively than sequence-to-sequence and reading comprehension baselines. MQAN shows improvements in transfer learning for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification. We demonstrate that the MQAN's multi-pointer-generator decoder is key to this success and that performance further improves with an anti-curriculum training strategy. Though designed for decaNLP, MQAN also achieves state of the art results on the WikiSQL semantic parsing task in the single-task setting. We also release code for procuring and processing data, training and evaluating models, and reproducing all experiments for decaNLP.

I appreciate the work that went into creating this paper, but I'm afraid I see little justification for accepting it.  I have three major complaints with this paper:

1. I think the framing of decaNLP presented in this paper does more harm than good, because it perpetuates a misguided view of question answering.

Question answering is not a unified phenomenon.  There is no such thing as "general question answering", not even for humans.  Consider "What is 2 + 3?", "What's the terminal velocity of a rain drop?", and "What is the meaning of life?"  All of these questions require very different systems to answer, and trying to pretend they are the same doesn't help anyone solve any problems.

Question answering is a _format_ for studying particular phenomena.  Sometimes it is useful to pose a task as QA, and sometimes it is not.  QA is not a useful format for studying problems when you only have a single question (like "what is the sentiment?" or "what is the translation?"), and there is no hope of transfer from a related task.  Posing translation or classification as QA serves no useful purpose and gives people the wrong impression about question answering as a format for studying problems.
We have plenty of work that studies multiple datasets at a time (including in the context of semi-supervised / transfer learning), without doing this misguided framing of all of them as QA (see, e.g., the ELMo and BERT papers, which evaluated on many separate tasks).  I don't see any compelling justification for setting things up this way.

2. One of the main claims of this paper is transfer from one task to another by posing them all as question answering.  There is nothing new in the transfer results that were presented here, however.  For QA-SRL / QA-ZRE, transfer from SQuAD / other QA tasks has already been shown by Luheng He (http://aclweb.org/anthology/N18-2089) and Omer Levy (that was the whole point of the QA-ZRE paper), so this is merely reproducing that result (without mentioning that they did it first).  For all other tasks, performance drops when you try to train all tasks together, sometimes significantly (as in translation, unsurprisingly).  For the Czech task, fine tuning a pre-trained model has already been shown to help.  Transfer from MNLI to SNLI is known already and not surprising - one of the main points of MNLI was domain transfer, so obviously this has been studied before.  The claims about transfer to new classification tasks are misleading, as you really have the _same_ classification task, you've just arbitrarily changed how you're encoding the class label.  It _might_ be the case that you still get transfer if you actually switch to a related classification task, but you haven't examined that case.

3. This paper tries to put three separate ideas into a single conference paper, and all three ideas suffer as a result, because there is not enough space to do any of them justice.  Giving 15 pages of appendix for an 8 page paper, where some of the main content of the paper is pushed to the appendix, is egregious.  Putting your work in the context of related work is not something that should be pushed into an appendix, and we should not encourage this behavior.

The three ideas here seem to me to be (1) decaNLP, (2) the model architecture of MQAN, (3) transfer results.  Any of these three could have been a single conference paper, had it been done well.  As it stands, decaNLP isn't described or motivated well enough, and there isn't any space left in the paper to address my severe criticisms of it in my first point.  Perhaps if you had dedicated the paper to decaNLP, you could have given arguments that the framing is worthwhile, and described the tasks and their setup as QA sufficiently (as it is, I don't see any description anywhere of how the context is constructed for WikiSQL; did I miss it somewhere?).  For MQAN, there's more than a page of the core new architecture that's pushed into the appendix.  And for the transfer results, there is very little comparison to other transfer methods (e.g., ELMo, CoVe), or any deep analysis of what's going on - as I mentioned above, basically all of the results presented are just confirming what has already been done elsewhere.


Deep latent variable models, such as variational autoencoders, have been successfully used to disentangle factors of variation in image datasets. The structure of the representations learned by such models is usually observed after training and iteratively refined by tuning the network architecture and loss function. Here we propose a method that can explicitly place information into a specific subset of the latent variables. We demonstrate the use of the method in a task of disentangling global structure from local features in images. One subset of the latent variables is encouraged to represent local features through an auxiliary modelling task. In this auxiliary task, the global structure of an image is destroyed by dividing it into pixel patches which are then randomly shuffled. The full set of latent variables is trained to model the original data, obliging the remainder of the latent representation to model the global structure. We demonstrate that this approach successfully disentangles the latent variables for global structure from local structure by observing the generative samples of SVHN and CIFAR10. We also clustering the disentangled global structure of SVHN and found that the emerging clusters represent meaningful groups of global structures -including digit identities and the number of digits presence. Finally, we discuss the problem of evaluating the clustering accuracy when ground truth categories are not expressive enough.

This work proposes an approach for explicitly placing information in a subset of the latent variables. The approach is to construct an auxiliary generative model that takes as input the set of latent variables subtracted from the target subset, which is used to model modified data samples that do not contain the desired information.
Experiments focus on learning global information. The auxiliary model is then given data that have their global information destroyed via random shuffling of image patches.
# Approach seems limited.
 - This approach seems very limited, as there must exist a known transformation that removes the desired information. Apart from global vs. local, can the authors provide more examples of what sort of information this approach can disentangle? (Even for global vs. local, is there a transformation that can remove local information as opposed to global information?)
 - Can this approach learn multiple factors as opposed to just two?
 - What if the desired factors are not clearly disjoint and collectively exhaustive? (e.g. mustache vs. gender on human faces.)
# More ablations or experiments with comparable settings would be desirable.
 - What is the choice of beta in the beta-VAE training objective? Apart from 1.2, this isn't mentioned. My concern here is that beta might be affecting the result more than the proposed training algorithm. Can the proposed approach perform just as well without a modified objective? Ablation studies that show the proposed algorithm can improve upon the baseline in all settings would make this a stronger paper. (e.g. this approach with normal VAE objective, and normal VAE objective without auxiliary task for the clustering experiment.)
 - Why were 30 discrete categories used in the clustering experiment? Is this still comparable to the approaches that use 10, which would correspond to the number of classes?
# Related work.
There are some well-cited works that the authors may have missed. These are ultimately different approaches, but perhaps the authors can obtain some inspiration from these:
- Tranforming autoencoders [1] also apply a transformation to the image, but the goal is to learn the factor corresponding to the transformation, rather than the complement as in this work.
- An opposing approach for explicit information placement with a modified training procedure (where the target information is directly placed in the target subset and can handle multiple factors) is DC-IGN [2]. I believe the DC-IGN approach is more general and can handle a superset of the tasks of this approach, without requiring an auxiliary decoder. Comparing to this approach, I wonder if it would be better to provide samples that exhibit a particular factor, or samples that conceal the factor?
[1] Hinton, Geoffrey E., Alex Krizhevsky, and Sida D. Wang. "Transforming auto-encoders." International Conference on Artificial Neural Networks. Springer, Berlin, Heidelberg, 2011.
[2] Kulkarni, Tejas D., et al. "Deep convolutional inverse graphics network." Advances in neural information processing systems. 2015.
---- Update since rebuttal ----
I thank the authors for clarifying how this work fits in with related works and clarifying the hyperparameters. I maintain my concerns that the experiments are limited and do not showcase the individual benefit of using explicit information placement. More experiments based on different transformations that the authors have mentioned would make this a stronger contribution. The use of beta>1 is fine if it helps alongside the use of this approach, but it would have been better to see the effects of this approach and beta>1 (and other hyperparameters such as k in Table 1) in isolation.


We study model recovery for data classification, where the training labels are generated from a one-hidden-layer fully-connected neural network with sigmoid activations, and the goal is to recover the weight vectors of the neural network. We prove that under Gaussian inputs, the empirical risk function using cross entropy exhibits strong convexity and smoothness uniformly in a local neighborhood of the ground truth, as soon as the sample complexity is sufficiently large. This implies that if initialized in this neighborhood, which can be achieved via the tensor method, gradient descent converges linearly to a critical point that is provably close to the ground truth without requiring a fresh set of samples at each iteration. To the best of our knowledge, this is the first global convergence guarantee established for the empirical risk minimization using cross entropy via gradient descent for learning one-hidden-layer neural networks, at the near-optimal sample and computational complexity with respect to the network input dimension.

The paper presents theoretical analysis for recovering one-hidden-layer neural networks using logistic loss function. I have the following major concerns:
(1.a) The paper does not mention identifiability at all. As has been known, neural networks with even only one hidden layer are not identifiable. The authors need to either prove the identifiability or cite existing references on the identifiability. Otherwise,  the parameter recovery does not make sense.
Example: The linear network takes f(x) = 1'Wx/k, where 1 is a vector with every entry equal to one. Then two models with parameters W and V are identical as long 1'W = 1'V.
(1.b) If the equivalent parameters are not isolated, the local strong convexity is impossible to hold. The authors need to carefully justify their claim.
(2) When using Sigmoid or Tanh activation functions, the output is bounded between [0,1] or [-1,+1]. This is unrealistic for logistic regression: The output of [0,1] means that the posterior probability has to be bounded between 1/2 and e/(1+e); The output of [-1,1] means that the posterior probability has to be bounded between 1/(1+e) and e/(1+e).
(3) The most challenging part of the logistic loss is the lack of curvature, when neural networks have large magnitude outputs. Since this paper assumes that the neural networks takes very small magnitude outputs, the extension from Zhong et al. 2017b to the logistic loss is very straightforward.
(4) Spectral initialization is very impractical. Nobody is using it in practice. The spectral initialization avoids the challenging global convergence analysis.
(5) Theorem 3 needs clarification. Please explicitly write the RHS of (7). The result would become meaningless, if under the scaling of Theorem 2, is the RHS of (7) smaller than RHS of (5).
I also have the following minor concerns on some unrealistic assumptions, but these concerns do not affect my rating. These assumptions have been widely used in many other papers, due to the lack of theoretical understanding of neural networks in the machine learning community.
(6)	The neural networks take independent Gaussian input.
(7)	The model is assumed to be correct.
(8)	Only gradient descent is considered.


Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achieves performance equivalent to that of best fixed batch-size. At each epoch, the RMGD samples a batch size according to a certain probability distribution proportional to a batch being successful in reducing the loss function. Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success. After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size. Experimental results show that the RMGD achieves performance better than the best performing single batch size. It is surprising that the RMGD achieves better performance than grid search. Furthermore, it attains this performance in a shorter amount of time than grid search.

The authors consider the problem of determining the minibatch size for SGD by first fixing a set of candidate sizes, and then learning a distribution over those sizes using a MAB algorithm. A minibatch size is first sampled from the distribution, then one training epoch is performed. A validation error is then computed, and if it is lower than that of the last epoch, the cost of the minibatch is taken to be zero (otherwise one), and the distribution is updated. This is Algorithm 1.
In Section 4.2, they prove a regret bound, but I don’t think that regret is really the correct notion, here (although it’s very close). This is a subtle point, so I’ll set up some notation. Let w(b_1, ..., b_t) be the result at the tth epoch, if the batch sizes b_1, …, b_t were used at the 1st through tth epochs. Let y(w,b) be 0 if training one epoch starting at w with batch size b would improve the validation error, and 1 otherwise.
They show (unnumbered inequality on the middle of page 5) that \sum_t y(w(b_1,...,b_{t-1}),b_t) is close to \sum_t y(w(b_1,...,b_{t-1}),b^*), where b_t is the batch size that was chosen at time t, and b^* is the best fixed batch size. The key point here is that the comparator (the second sum) starts each epoch at the result that was found by their adaptive algorithm, *not* what would have been found if a batch size of b^* had been used from the beginning.
In other words, their result does *not* show that their algorithm is close to outperforming a fixed choice of batch size (for that to hold, the comparator would need to be \sum_t y(w(b^*,...,b^*),b^*)). What they show is similar, but subtly different. They don’t put too much weight on this theoretical result, and in fact don’t even explicitly claim that the comparator in this result is that for a fixed choice of batch size, so really this is a minor issue, but I think that this is something that should be clarified, since it would be easy for a reader to draw an incorrect conclusion.
With that said, their approach is well-motivated, and their experiments seem to show consistent small improvements in performance. I don’t think the performance improvements are totally conclusive, but one of the most appealing properties of their proposal is that it shouldn’t be much more computationally expensive than using a fixed minibatch size. Furthermore, their approach is potentially more robust, since you can presumably be less careful about choosing the set of candidate minibatch sizes, than you would be for choosing only one. So while the experiments don’t show a big improvement, their proposal has other benefits.


DEEPOBS: A DEEP LEARNING OPTIMIZER BENCHMARK SUITE
There is significant past and ongoing research on optimization methods for deep learning. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of such optimizers. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DEEPOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on IMAGENET or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce L A T E X code for inclusion in academic publications. It supports TENSORFLOW and is available open source.

This paper presents a new benchmark suite to compare optimizer on deep neural networks. It provides a pipeline to help streamlining the analysis of new optimizers which would favor easily reproducible results and fair comparisons.
Quality
The paper covers well the problems underlying the construction of such a benchmark, discussing the problems and models selection, runtime estimation, hyper-parameter selection and visualizations. It falls short however in some cases:
1. Hyper-parameter optimization
    While they mention the importance of hyper-parameter tuning for the benchmark, they leave it to the user to tune them without providing any standard procedure. Furthermore, they use grid search to build the baselines while this is known to be a poor optimizer [1].
2. Estimated runtime
    Runtime is estimated for a single set of hyper-parameters of the optimizer, but some optimizer may have similar or roughly similar results for a large set of hyper-parameters that widely affects the runtime. The effect of the hyper-parameters should be taken into account for this part of the benchmark.
3. Interpretation
    Such a benchmark should makes it easier for interpretation of results as the authors suggests. However, the paper does not convey much interpretation in section 4, beside the fact that results are not conclusive for any baseline. Results of the paper seem low, but they are difficult to verify since the plots are not very precise. For instance Wide ResNet-18-8 reports 1.54% test accuracy on SVHN [6] while this paper reports ~ 15% for the Wide ResNet 18-4 version. Figure 2 is a good attempt at making interpretations of sensitivity of optimizers' hyper-parameters but has limited interpretability compared to what can be found in the literature [2].
4. Problems
    There is an effort to provide varied types of problem, including classical optimization functions, image classification, image generation and language modeling. The number of problems consists mostly of image classification however and is very limited for image generation and language modeling.
Clarity
The paper is well written and easy to understand in general.
On a minor note, most figures are difficult to read. Side nodes on figure 1 does not divide clearly without any capital letter or punctuation at the end of sentence. Figure 2 should be self contained with its own legend. Figure 3 is useful for a visual impression of the speed of convergence but a histogram would be necessary for a better visual comparison of the different performances.
Section 2.2 has a confusing terminology for the "train valid set". Is it a standard validation set?
Originality
There is virtually no benchmarks for optimizers available for the community. I believe a standardized procedure for comparing optimizers can be viewed as an original contribution.
Significance
Reproducibility is a problem in machine learning [3, 4] and optimizers' efficiency on deep neural networks generalization performance is still not very well understood [5]. Therefore, there is a strong need for a benchmark for sound comparisons and to favor better reproducibility.
Conclusion
The benchmark presented in this paper would be an important contribution to the community but lacks a few important features in my opinion, in particular, sound hyper-parameter optimization procedure and sound interpretation tools. On a skeptical note, I doubt the benchmark will be used extensively if the results it provides yield no conclusive interpretation as reported for the baselines. As I feel there is more work needed to support the goals of the paper, I would suggest this paper for a workshop. Nevertheless, I would not be upset if it was accepted because of the importance of the subject and the originality of this work.
[1] Bergstra, James, and Yoshua Bengio. "Random search for hyper-parameter optimization." Journal of Machine Learning Research 13, no. Feb (2012): 281-305.
[2] Biedenkapp, Andre, Joshua Marben, Marius Lindauer and Frank Hutter. “CAVE : Configuration Assessment , Visualization and Evaluation.” In International Conference on Learning and Intelligent Optimization (2018).
[3] Lucic, Mario, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. “Are GANs Created Equal? A Large-Scale Study.” arXiv preprint arXiv:1711.10337 (2017).
[4] Melis, Gábor, Chris Dyer, and Phil Blunsom. “On the state of the art of evaluation in neural language models.” arXiv preprint arXiv:1707.05589 (2017).
[5] Wilson, Ashia C., Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. "The marginal value of adaptive gradient methods in machine learning." In Advances in Neural Information Processing Systems, pp. 4148-4158. 2017.
[6] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016
-----------
Revision
-----------
In light of the discussion with the authors, the revision made to chapter 4 and in particular the proposed modifications to section 2.4 for a camera-ready paper, I revise my score to 6.


We describe a simple ensemble approach that, unlike conventional ensembles, uses multiple random data sketches ('pseudosaccades') rather than multiple classifiers to improve classification performance. Using this simple, but novel, approach we obtain statistically significant improvements in classification performance on AlexNet, GoogLeNet, ResNet-50 and ResNet-152 baselines on Imagenet datae.g. of the order of 0.3% to 0.6% in Top-1 accuracy and similar improvements in Top-k accuracy -essentially nearly for free.

The paper proposes a data augmentation technique where the input image is sub-sampled by randomly sampling rows and columns without replacement, which the authors call ‘pseudosaccades’. Rather than multiple classifiers, the authors ensemble using multiple ‘pseudosaccades’ as input, with the same network.
Comments:
I think that the proposed augmentation is a neat trick. However, the inner-workings of the method are poorly presented (or not well understood). For eg. In section 3.5, while discussing the effects of the method on individual classes, the authors mention ‘different architectures do tend to be affected by the pseudosaccades differently’ and provide no further insights.
There are no experiments that compare this method with other standard data augmentation techniques. For instance, one could use a similar ensembling technique for transformations like shear, translation, rotation, etc. by randomly sampling their corresponding parameters. I would be interested in experimental results that compare the proposed ensemble with ensembles constructed using these common techniques.
Since there is no reason for this technique to be used in isolation (I found no such motivation in the paper), it would be insightful to have experimental results where this technique is combined with the aforementioned standard augmentation techniques. Will this method’s impact on the accuracy change with these other augmentations? (Ablation studies would be useful).
This is a a form of regularization and can be thought of reverse structured dropout. Also have the authors compared this with Cutout [1, 2]? Similar experiments and comparisons would be insightful.
[1] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.
[2] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. arXiv preprint arXiv:1708.04896, 2017.
In summary:
The performance improvements are incremental. The paper lacks sufficient technical contribution. Further, it does not provide comparisons with standard techniques and similar augmentation methods to demonstrate the usefulness of the method.


Recent literature has demonstrated promising results on the training of Generative Adversarial Networks by employing a set of discriminators, as opposed to the traditional game involving one generator against a single adversary. Those methods perform single-objective optimization on some simple consolidation of the losses, e.g. an average. In this work, we revisit the multiple-discriminator approach by framing the simultaneous minimization of losses provided by different models as a multi-objective optimization problem. Specifically, we evaluate the performance of multiple gradient descent and the hypervolume maximization algorithm on a number of different datasets. Moreover, we argue that the previously proposed methods and hypervolume maximization can all be seen as variations of multiple gradient descent in which the update direction computation can be done efficiently. Our results indicate that hypervolume maximization presents a better compromise between sample quality and diversity, and computational cost than previous methods.

Clarity:
The work is a clear introduction/overview of this area of research. The reviewer enjoyed the connections to Multiple-Gradient Descent and clear distinctions/contrasts with previous approaches to weighting the outputs of multiple discriminators. All in all, the paper is quite clear in what its contributions are and how it differs from previous approaches. The details and motivations of the Hypervolume Maximization  (HVM) method (especially as it relates to and interacts with the slack method of picking the nadir point) were a bit harder to follow intuitively given the standalone information in the paper.
Originality:
Adapts a technique to approximate MGD called HVM (Miranda 2016) and applies it to multi-discriminator training in GANs. As far as the reviewer is aware, this is a novel application of HVM to this task and well motivated under the MGD interpretation of the problem.
Significance:
Unclear. This work in isolation appears to present an improvement over prior work in this sub-field, but it is not obvious that the findings in these experiments will continue to be robust in more competitive settings. For instance, the worst performing model on CIFAR10, WGAN-GP (according to the experiments run) WGAN-GP also holds near SOTA Inception scores on CIFAR10 when appropriately tuned. Without any experimental results extending beyond toy datasets like MNIST and CIFAR10 the reviewer is not confident whether fundamental issues with GAN training are being addressed or just artifacts of small scale setups. Closely related previous work (Neyshabur 2017) scaled to 128x128 resolution on a much more difficult dataset - Imagenet Dogs but the authors did not compare in this case.
Quality:
Some concerns about details of experiments (see cons list and significance section for further discussion).
Pros:
+ The work provides a clear overview of previous work on approaches using multiple discriminators.
+ The connections of this line of work to MGD and the re-interpretation of various other approaches in this framework is valuable.
+ The author provides direct comparisons to similar methods, which increases confidence in the results.
+ On the experiments run, the HVM method appears to be an improvement over the two previous approaches of softmax weighting and straightforward averaging for multiple discriminators.
Cons:
- Performance of GANs is highly dependent on both model size and compute expended for a given experiment (see Miyato 2018 for model size and training iterations and Brock 2018 for batch size). Training multiple discriminators (in this paper up to 24) significantly increases compute cost and effective model size. No baselines controlling for the effects of larger models and batch sizes are done.
- The paper lacks experiments beyond toy-ish tasks like MNIST and CIFAR10 and does not do a good job comparing to the broader established literature and contextualizing its results on certain tasks such as CIFAR10 (reporting ratios to a baseline instead of absolute values, for instance). The absolute inception score of the baseline DCGAN needs to be reported to allow for this. Is the Inception Score of the authors DCGAN implementation similar to the 6 to 6.5 reported in the literature?
- Figure 3 is slightly strange in that the x axis is time to best result result instead of just overall wallclock time. Without additional information I can not determine whether it is admissible. Do all models achieve their best FID scores at similar points in training? Why is this not just a visualization of FID score as a function of wallclock time? A method which has lower variance or continues to make progress for longer than methods which begin to diverge would be unfairly represented by the current Figure.
Additional comments:
In section 3.1 Eq 5 appears to be wrong. The loss of the discriminator is presented in a form to be minimized so exponentiating the negative loss in the softmax weighting term as presented will do the opposite of what is desired and assign lower weight to higher loss discriminators.
In Fig 6 FID scores computed on a set of 10K samples are shown. The authors appear to draw the line for the FID score of real data at 0. But since it is being estimated with only 10K samples there will be sampling error resulting in non-zero FID score. The authors should update this figure to show the box-plot for FID scores computed on random draws of 10K real samples. I have only worked with FID on Imagenet where FID scores for random batches of 10K samples are much higher than 0. I admit there is some chance the value is extremely low on CIFAR10 to make this point irrelevant, however.


While reinforcement learning (RL) shows a lot of promise for natural language processing-e.g. when fine-tuning natural language systems for optimizing a certain objective-there has been little investigation into potential language drift: when an external reward is used to train a system, the agents' communication protocol may easily and radically diverge from natural language. By re-casting translation as a communication game, we show that language drift indeed happens when pre-trained agents are fine-tuned with policy gradient methods. We contend that simply adding a "naturalness" constraint to the reward, e.g. by using language model log likelihood, does not fully address the issue, and argue that (perceptual) grounding is required. That is, while language model constraints impose syntactic conformity, they do not lead to semantic correspondence. Our experiments show that grounded models give the best communication performance, while retaining English syntax along with the ability to convey the intended semantics.

This paper poses and addresses the problem of language drift in multi-agent communication paradigms. When two pretrained natural-language agents are jointly optimized to communicate and solve some external non-linguistic objective, their internal communication often diverges to a code-like, unnatural communication system. This paper solves this “language drift” problem by requiring that the messages between agents be usable as inputs to an image caption retrieval system. They demonstrate that the jointly optimized agents perform best when regularized in this manner to prevent language drift.
1. Framing: I’m uncertain about the framing of this paper. The authors pose the problem of “language drift,” which is indeed a frequent problem in multi-agent communication tasks where the principle supervision involves non-linguistic inputs and outputs. They then design a three-language MT task as a test case, where the inputs and outputs are both linguistic. Why attack this particular task and grounding solution? I can imagine some potential goals of the paper, but also see more direct ways to address each of the potential goals than what the authors have chosen:
1a. Study how to counter language drift in general — why not choose a more intuitive two-agent communication task, e.g. navigation, game playing, etc.?
1b. Study how to counter language drift in the MT task — aren’t there simpler solutions to prevent language drift in this particular task? e.g. require “cycle-consistency” – that it be possible to reconstruct the French input using the French output? Why pick multimodal grounding, given that it imposes substantial additional data requirements?
1c. Build a better/more data-efficient machine translation system — this could be an interesting goal and suitable for the paper, but this doesn’t seem to be the framing that the authors intend.
2. Interpretation of first results:
2a. Thanks for including standard deviation estimates! I think it’s also important that you do some sort of significance testing on the comparison between PG+LM+G and PG+LM performance for Fr->En->De — these numbers look pretty close to me. You could run e.g. a simple sign test on examples within each corpus between the two conditions.
2b. It would also be good to know how robust your results are to hyperparameter settings (especially the entropy regularization hyperparameter).
3. Token frequency results: These are intriguing but quite confusing to me!
3a. How sensitive are these results to your entropy regularization setup? How does PG behave without entropy regularization?
3b. Table 6 shows that the PG model has very different drift for different POS categories. Does this explain away the change in the token frequency distribution? What do the token frequency effects look like for PG within the open-class / content word categories (i.e., controlling for the huge difference in closed-class behavior)?
4. Minor comments:
4a. There’s a related problem in unsupervised representation learning for language. Work on VAEs for language, for example, has shown that the encoder often collapses meaning differences in the latent representation, and leans on an overly powerful decoder in order to pick up all of the lost information. It would be good to reference this work in your framing (see e.g. Bowman et al. (2015)).
4b. In sec. 3.1 you overload notation for R. Can you subscript these so that it’s especially clear in your results which systems are following which reward function?
4c. Great to show some qualitative examples in Table 7 — can you explicitly state where these are from (dev set vs. test set?) and whether they are randomly sampled?
References:
Bowman et al. (2015). Generating sentences from a continuous space. https://arxiv.org/abs/1511.06349


INSTAGAN: INSTANCE-AWARE IMAGE-TO-IMAGE TRANSLATION
Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan.

This paper does unpaired cross-domain translation of multi-instance images, proposing a method -- InstaGAN -- which builds on CycleGAN by taking into account instance information in the form of per-instance segmentation masks.
=====================================
Pros:
The paper is well-written and easy to understand. The proposed method is novel, and does a good job of handling a type of information that previous methods couldn’t.
The motivation for each piece of the model and training objective is clearly explained in the context of the problem. Intuitively seems like a nice and elegant way to take advantage of the extra segmentation information available.
The results look pretty good and clearly compare favorably with CycleGAN and other baselines. The tested baselines seem like a fair comparison -- for example, the model capacity of the baseline is increased to compensate for the larger proposed model.
=====================================
Cons / suggestions:
The results are somewhat limited in terms of the number of domains tested -- three pairs of categories (giraffe/sheep, pants/skirts, cup/bottle).  In a sense, this is somewhat understandable -- one wouldn’t necessarily expect the method to be able to translate between objects with different scale or that are never seen in the same contexts (e.g. cups and giraffes). However, it would still have been nice to see e.g. more pairs of animal classes to confirm that the category pairs aren’t the only ones where the method worked.
Relatedly, it would have been interesting to see if a single model could be trained on multiple category pairs and benefit from information sharing between them.
The evaluation is primarily qualitative, with quantitative results limited to Appendix D showing a classification score. I think there could have been a few more interesting quantitative results, such as segmentation accuracy of the proposed images for the proposed masks, or reconstruction error. Visualizing some reconstruction pairs (i.e., x vs. Gyx(Gxy(x))) would have been interesting as well.
I would have liked to see a more thorough ablation of parts of the model. For example, the L_idt piece of the loss enforcing that an image in the target domain (Y) remain identical after passing through the generator mapping X->Y. This loss term could have been included in the original CycleGAN as well (i.e. there is nothing about it that’s specific to having instance information) but it was not -- is it necessary?
=====================================
Overall, while the evaluation could have been more thorough and quantitative, this is a well-written paper that proposes an interesting, well-motivated, and novel method with good results.
==========================================================================
REVISION
The authors' additional results and responses have addressed most of my concerns, and I've raised my rating from 6 to 7.
> We remark that the identity mapping loss L_idt is already used by the authors of the original CycleGAN (see Figure 9 of [2]).
Thanks, you're right, I didn't know this was part of the original CycleGAN. As a final suggestion, it would be good to mention in your method section that this loss component is used in the original CycleGAN for less knowledgeable readers (like me) as it's somewhat hard to find in the original paper (only used in some of their experiments and not mentioned as part of the "main objective").


VISCERAL MACHINES: RISK-AVERSION IN REINFORCEMENT LEARNING WITH INTRINSIC PHYSIOLOGICAL REWARDS
As people learn to navigate the world, autonomic nervous system (e.g., "fight or flight") responses provide intrinsic feedback about the potential consequence of action choices (e.g., becoming nervous when close to a cliff edge or driving fast around a bend.) Physiological changes are correlated with these biological preparations to protect one-self from danger. We present a novel approach to reinforcement learning that leverages a task-independent intrinsic reward function trained on peripheral pulse measurements that are correlated with human autonomic nervous system responses. Our hypothesis is that such reward functions can circumvent the challenges associated with sparse and skewed rewards in reinforcement learning settings and can help improve sample efficiency. We test this in a simulated driving environment and show that it can increase the speed of learning and reduce the number of collisions during the learning stage.

Summary:
This submission proposes a reinforcement learning framework based on human emotional reaction in the context of autonomous driving. This relies on defining a reward function as the convex combination of an extrinsic (goal oriented) reward, and an intrinsic reward. This later reward is learnt from experiments with humans performing the task in a virtual environment, for which emotional response is quantified as blood volume pulse wave (BVP). The authors show that including this intrinsic reward lead to a better performance of a deep Q networks, with respect to using the extrinsic reward only.
Evaluation:
Overall the proposed idea is interesting, and the use of human experiments to improve a reinforcement learning algorithm offers interesting perspectives. The weakness of the paper in my opinion is the statistical analysis of the results, the lack of in depth evaluation of the extrinsic reward prediction and the rather poor baseline comparison.
Detailed comments:
1.	Statistical analysis
The significance of the results should be assessed with statistical methods in the following results:
Section 4.1: Please provide and assessment of the significance of the testing loss of the prediction. For example, one could repetitively shuffle blocks of the target time series and quantify the RMSE obtained by the trained algorithm to build an H0 statistic of random prediction.
Section 4.2: the sentence “improves significantly when lambda is either non-zero or not equal to 1” does not seem valid to me and such claim should in any case be properly evaluated statistically (including correction for multiple comparison etc…).
Error bars: please provide a clear description in the figure caption of what the error bars represent. Ideally in case of small samples, box plots would be more appropriate.
2.	Time lags in BVP
It would be interesting to know (from the literature) the typical latency of BVP responses to averse stimuli (and possible the latency of the various mechanisms, e.g. brain response, in the chain from stimuli to BVP). Moreover, as latency is likely a critical factor in anticipating danger before it is too late, it would important to know how the prediction accuracy evolves when learning to predict at different time lags forward in time, and how such level of anticipation influence the performance of the Q-network.
3.	Poor baseline comparison
The comparison to reward shaping in section 4.4 is not very convincing. One can imagine that what counts is not the absolute distance to a wall, but the distance to a wall in the driving direction, within a given solid angle. As a consequence, a better heuristic baseline could be used.
Moreover, it is unclear whether the approaches should be compared with the same lambda: the authors need to provide evidence that the statistics (mean and possibly variance) of the chosen heuristic is match to the original intrinsic reward, otherwise it is obvious that the lambda should be adapted.
4.	Better analysis of figure 5-6(Minor)
I find figure 5-6 very interesting and I would suggest that the authors fully comment on these results. E.g. : (1) why the middle plot of Fig. 6 mostly flat, and why such differences between each curve from the beginning of the training. (2) Why the goal oriented task leads to different optimal lambda, is this just a normalization issue?


SOLVING THE RUBIK'S CUBE WITH APPROXIMATE POLICY ITERATION
Recently, Approximate Policy Iteration (API) algorithms have achieved superhuman proficiency in two-player zero-sum games such as Go, Chess, and Shogi without human data. These API algorithms iterate between two policies: a slow policy (tree search), and a fast policy (a neural network). In these two-player games, a reward is always received at the end of the game. However, the Rubik's Cube has only a single solved state, and episodes are not guaranteed to terminate. This poses a major problem for these API algorithms since they rely on the reward received at the end of the game. We introduce Autodidactic Iteration: an API algorithm that overcomes the problem of sparse rewards by training on a distribution of states that allows the reward to propagate from the goal state to states farther away. Autodidactic Iteration is able to learn how to solve the Rubik's Cube without relying on human data. Our algorithm is able to solve 100% of randomly scrambled cubes while achieving a median solve length of 30 moves -less than or equal to solvers that employ human domain knowledge.

This paper introduces a deep RL algorithm to solve the Rubik's cube. The particularity of this algorithm is to handle the huge state space and very sparse reward of the Rubik's cube. To do so, a) it ensures each training batch contains states close to the reward by scrambling the solution; b) it computes an approximate value and policy for that state using the current model and c) it weights data points based by the inverse of the number of random moves from the solution used to generate that training point. The resulting model is compared to two non-ML algorithms and shown to be competitive either on computational speed or on the quality of the solution.
This paper is well written and clear. To the best of my knowledge, this is the first RL-based approach to handle the Rubik's cube problem so well. The specificities of this problem make it interesting. While the idea of starting from the solution seemed straightforward at first, the paper describes more advanced tricks claimed to be necessary to make the algorithm work. The algorithm seems to be quite successful and competitive with expert algorithms, which I find very nice. Overall, I found the proposed approach interesting and sparsity of reward is an important problem so I would rather be in favor of accepting this paper.
On the negative side, I am slightly disappointed that the paper does not link to a repository with the code. Is this something the authors are considering in the future? While it does not seem difficult to code, it is still nice to have the experimental setup.
There has been (unsuccessful) attempts to solve the Rubik's cube using deep RL before. I found some of them here: https://github.com/jasonrute/puzzle_cube . I am not sure whether these can be considered prior art as I could not find associated accepted papers but some are quite detailed. Some could also provide additional baselines for the proposed methods and highlight the challenges of the Rubik's cube.
I am also curious whether/how redundant positions are handled by the proposed approach and wished this would be discussed a bit. Considering the nature of the state space and the dynamics, I would have expected this to be a significant problem, unlike in Go or chess. Does the algorithm forbid the reverse of the last action? Is the learned value/policy function good enough that backwards moves are seldom explored? Since the paper mention that BFS is interesting to remove cycles, I assume identical states are not duplicated. Is this correct?


Naively applying Reinforcement Learning algorithms to continuous control problems -such as locomotion and robot control -to maximize task reward often results in policies which rely on high-amplitude, high-frequency control signals, known colloquially as bang-bang control. While such policies can implement the optimal solution, particularly in simulated systems, they are often not desirable for real world systems since bang-bang control can lead to increased wear and tear and energy consumption and tends to excite undesired second-order dynamics. To counteract this issue, multi-objective optimization can be used to simultaneously optimize both the reward and some auxiliary cost that discourages undesired (e.g. high-amplitude) control. In principle, such an approach can yield the sought after, smooth, control policies. It can, however, be hard to find the correct trade-off between cost and return that results in the desired behavior. In this paper we propose a new constraint-based approach which defines a lower bound on the return while minimizing one or more costs (such as control effort). We employ Lagrangian relaxation to learn both (a) the parameters of a control policy that satisfies the desired constraints and (b) the Lagrangian multipliers for the optimization. Moreover, we demonstrate policy optimization which satisfies constraints either in expectation or in a per-step fashion, and we learn a single conditional policy that is able to dynamically change the trade-off between return and cost. We demonstrate the efficiency of our approach using a number of continuous control benchmark tasks as well as a realistic, energy-optimized quadruped locomotion task.

This paper proposes a model free reinforcement learning algorithm with constraint on reward, with demonstration on cartpole and quadruped locomotion.
strength: (1) challenging examples like the quadruped.
                (2) result seems to indicate the method is effective
There are several things I would like the authors to clarify:
(1) In section 3.2, why is solving (4) would give a "exactly the desired trade-off between reward and cost"? First of all, how is the desired trade-off defined? And how is (4) solved exactly? If it is solved iteratively, i.e, alternating between the inner min and outer max, then during the inner loop, wouldn't the optimal value for \lambda be infinity when constrained is violated (which will be the case at the beginning)? And when the constrained is satisfied, wouldn't \lambda = 0? How do you make sure the constrained will still be satisfied during the outer loop since it will not incurred penalty(\lambda=0). Even if you have a lower bound on \lambda, this is introducing additional hyperparameter, while the purpose of the paper is to eliminate hyperparamter?
(2) In section 3.2, equation 6. This is clearly not a convex combination of Qr-Vr and Qc, since convex combination requires nonnegative coefficients. The subtitle is scale invariance, and I cannot find what is the invariance here (in fact, the word invariance\invariant only appears once in the paper). By changing the parametrization, you are no longer solving the original problem (equation 4), since in equation (4), the only thing that is related to \lambda is (Qr-Vr), and in (6), you introduce \lambda to Qc as well. How is this change justified?
(3)If I am not mistaken, the constrained can still be violated with your method. While from the result it seems your method outperforms manually selecting weights to do trade off,  I don't get an insight on why this automatic way to do tradeoff is better. And this goes back to "exactly the desired trade-off between reward and cost" in point(1), how is this defined?
(3) The comparison in the cartpole experiment doesn't seem fair at all, since the baseline controller is not optimized for energy, there is no reason why it would be comparable to one that is optimized for energy. And why would a controller " switch between maximum and minimum actuation is indeed the optimal solution" after swingup? Maybe it is "a" optimal solution, but wouldn't a controller that does nothing is more optimal(assuming there is no disturbance)?
(4)For Table I, the error column is misleading. If I understand correctly, exceeding the lower bound is not an error (If I am wrong, please clarify it in the paper). And it is interesting that for target=0.3, the energy consumption is actually the lowest.
(5)Another simple way to impose constrained would be to terminate the episode and give large penalty, it will be interesting to see such comparison.
minor points:
* is usually used for optimal value, but is used in the paper as a bound.


Timestamped sequences of events, pervasive in domains with data logs, e.g, health records, are often modeled as point processes with rate functions over time. Leading classical methods for risk scores such as Cox and Hawkes processes use such data but make strong assumptions about the shape and form of multivariate influences, resulting in time-to-event distributions irreflective of many real world processes. Recent methods in point processes and recurrent neural networks capably model rate functions but may be complex and difficult to interrogate. Our work develops a high-performing, interrogable model. We introduce wavelet reconstruction networks, a multivariate point process with a sparse wavelet reconstruction kernel to model rate functions from marked, timestamped data. We show they achieve improved performance and interrogability over baselines in forecasting complications and scheduled care visits in patients with diabetes.

[PROS]
[originality]
The paper proposed to construct the intensity function of a point process using wavelet, in order to improve its expressiveness, e.g. allowing non-additivity.
The authors did extensive experiments to investigate their model performance compared to many appropriate baselines, on both synthetic and real-world datasets.
[CONS]
[clarity]
The major drawback of this submission is its clarity. The paper is vague at various important points in both method and experiments, thus leaving their correctness and soundness undetermined.
In the method section, the authors did not specify a well-defined and self-consistent notation system. This makes the paper really hard to understand. For example, one may be easily confused with things like:
1) How q, q(g(t), t_i), g(t), g_{es}, g_{d} are connected and distinguished?
2) The function g_{es} maps from t to R, then is t a space or a variable?
3) The state s seems crucial in the function g_{es}, but why it is only mentioned one time in the paper? How it is defined and how it is used?
4) How the Hadamard product is applied to two matrices of different sizes?
5) j=1 is time dimension and j=2 is time and value dimension, then why j=1 is not part of j=2? Time is needed in both cases and it seems natural that the associated parameters are shared.
6) Figure-3 has t_i in figure, t’ in caption, but the text in main paper mentions t_0 for Figure-3. How are they related? Are they actually the same?
The most confusing part is the censoring distance c. Its introduction around eqn-(2) suggests that c > 0 and the censoring section clearly mentions that. But c is also used to denote the forecast distance, which is clearly < 0 according to Figure-6 and Figure-7. What’s worse, there is also a term called forecast censoring distance. What are the relationships among these terms? If they are all the same, then is the c actually a model parameter or an evaluation control knob? Such things are very important to clarify.
Moreover, the paper did not clearly explain how the model is trained in each case, especially for (multi-)forecasting. In details:
1) What is the training objective?.
2) What is the optimization method for this objective?
3) How is it implemented and would the code be released?
It is good that the experimental section lists many appropriate baseline models and multiple evaluation metrics, but it is not clear how they are used. For example:
1) Fourier methods and Hawkes process do not deal with the value v, then how are they fairly compared to the proposed model which takes v into account as in eqn-(2)?
2) How is the Goodman-Kruskal gamma exactly computed? On all the instances of the held-out set? What is exactly the rank in this case?
3) The authors also leave out the positiveness constraints of a Hawkes process to incorporate inhibitory interactions, but how the positivity of the intensity function is ensured in this case?
[quality and significance]
The method is well-motivated and innovative. But details of the model and experiments are very unclear, so its overall soundness is hard to judge. For example:
1) The authors claim that, compared to neural models, their model has the advantage of the interpretability (for small datasets), but they also have neural components in their model. So why their model is more interpretable than others (e.g. Mei and Eisner 2017 as they cited) is not clear to me.
2) It is not clear why the interpretability is associated with the size of the dataset (quote `remains interpretable for small data sets’). What’s worse, the interpretability seems the only advantage of the model over other neural models (please correct me if I am wrong). If this edge could not scale up to large datasets, then does it mean on large datasets, a neural model should always be preferred over this model, because they are supposed to be more expressive?


We present Predicted Variables (PVars), an approach to making machine learning (ML) a first class citizen in programming languages. There is a growing divide in approaches to building systems: using human experts (e.g. programming) on the one hand, and using behavior learned from data (e.g. ML) on the other hand. PVars aim to make using ML in programming easier by hybridizing the two. We leverage the existing concept of variables and create a new type, a predicted variable. PVars are akin to native variables with one important distinction: PVars determine their value using ML when evaluated. We describe PVars and their interface, how they can be used in programming, and demonstrate the feasibility of our approach on three algorithmic problems: binary search, QuickSort, and caches. We show experimentally that PVars are able to improve over the commonly used heuristics and lead to a better performance than the original algorithms. As opposed to previous work applying ML to algorithmic problems, PVars have the advantage that they can be used within the existing frameworks and do not require the existing domain knowledge to be replaced. PVars allow for a seamless integration of ML into existing systems and algorithms. Our PVars implementation currently relies on standard Reinforcement Learning (RL) methods. To learn faster, PVars use the heuristic function, which they are replacing, as an initial function. We show that PVars quickly pick up the behavior of the initial function and then improve performance beyond that without ever performing substantially worse -allowing for a safe deployment in critical applications.

This paper proposes using predicted variables(PVars) - variables that learn
their values through reinforcement learning (using observed values and
rewards provided explicitly by the programmer). PVars are meant to replace
variables that are computed using heuristics.
Pros:
* Interesting/intriguing idea
* Applicability discussed through 3 different examples
Cons:
* Gaps in explanation
* Exaggerated claims
* Problems inherent to the proposed technique are not properly addressed, brushed off as if unimportant
The idea of PVars is potentially interesting and worth exploring; that
being said, the paper in its current form is not ready for
publication.
Some criticism/suggestions for improvement:
While the idea may be appealing and worth studying, the paper does not address several problems inherent to the technique, such as:
- overheads (computational cost for inference, not only in
  prediction/inference time but also all resources necessary to run
  the RL algorithm; what is the memory footprint of running the RL?)
- reproducibility
- programming overhead: I personally do not buy that this technique -
  at least as presented in this paper - is as easy as "if statements"
  (as stated in the paper) or will help ML become mainstream in
  programming. I think the programmer needs to understand the
  underpinnings of the PVars to be able to meaningfully provide
  observations and rewards, in addition to the domain specific
  knowledge. In fact, as the paper describes, there is a strong
  interplay between the problem setting/domain and how the rewards should be
  designed.
- applicability: when and where such a technique makes sense
The interface for PVars is not entirely clear, in particular the
meaning of "observations" and "rewards" do not come natural to
programmers unless they are exposed to a RL setting. Section 2 could
provide more details such that it would read as a tutorial on
PVars. If regular programmers read that section, not sure they
understand right away how to use PVars. The intent behind PVars
becomes clearer throughout the examples that follow.
It was not always clear when PVars use the "initialization function"
as a backup solution. In fact, not sure "initialization" is the right
term, it behaves almost like an "alternative" prediction/safety net.
The examples would benefit from showing the initialization of the PVars.
The paper would improve if the claims would be toned down, the
limitations properly addressed and discussed and the implications of
the technique honestly described. I also think discussing the
applicability of the technique beyond the 3 examples presented needs
to be conveyed, specially given the "performance" of the technique
(several episodes are needed to achieve good performance).
While not equivalent, I think papers from approximate computing (and
perhaps even probabilistic programming) could be cited in the related
work. In fact, for an example of how "non-mainstream" ideas can be
proposed for programming languages (and explained in a scientific
publication), see the work of Adrian Sampson on approximate computing
https://www.cs.cornell.edu/~asampson/research.html
In particular, the EnerJ paper (PLDI 2011) and Probabilistic Assertions (PLDI 2014).
Update: I maintain my scores after the rebuttal discussion.


GLUE: A MULTI-TASK BENCHMARK AND ANALYSIS PLATFORM FOR NATURAL LANGUAGE UNDERSTAND- ING
For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.

The paper proposes a new benchmark for natural language understanding: GLUE. Models will be evaluated based on a diverse set of existing language understanding tasks which encourages the models to learn shared knowledge across different tasks. The authors empirically show that models trained with multiple tasks in the dataset perform better than models that focused on one specific task. They also point out existing methods are not able achieve good performance in this dataset and request for more general natural language understanding system. The work also collects an expert evaluated diagnostic evaluation dataset for further examination for the models.
Quality: borderline, clarity:good, originality: borderline, significance: good,
Pros:
- The benchmark is set up in a online platform with leaderboard which can be easily accessible to people.
- The benchmark comes with a diagnostic evaluation dataset with coarse-grained and fine-grained categories that examine different aspect of language understanding abilities.
- Baseline results for major existing models are provided
Cons:
- The author should provide more detailed analysis and interpretable explanations for the results as opposed to simply stating that the overall performance is better.
For example, why attention hurts performance in single task training? Why multi-tasks training actually leads to worse performance on some of the dataset? Do these phenomenons still exist if you train on a different subset of the dataset?
What are the samples that the models failed to perform well? It would be nice to get some more insights and conclusions based on the results obtained from this benchmark to shed some lights on how to improve these models. The results section should be seriously revised.
- The diagnostic evaluation dataset seems to be a way to better understand the model, however, it is hard to see the scope of the data (are the samples under each categories balanced?). Besides, the examples in the dataset seems very confusing even for humans (Table 2).  The evaluation with NLP expert is also far from perfect. I wonder how accurate is this dataset annotated (or even the sentences make sense or not), and how suitable it is for evaluating model’s language understanding abilities. It would be nice if the authors can include some statistics about the dataset.
The paper proposes a useful benchmark that measures different aspects of language understanding abilities which would be helpful to the community. However, I feel the novelty or take away messages from the experiment section is limited.


EFFICIENT AUGMENTATION VIA DATA SUBSAMPLING
Data augmentation is commonly used to encode invariances in learning methods. However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set. The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply. In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset. We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.

Summary: The authors study the problem of identifying subsampling strategies for data augmentation, primarily for encoding invariances in learning methods. The problem seems relevant with applications to learning invariances as well as close connections with the covariate shift problem.
Contributions: The key contributions include the proposal of strategies based on model influence and loss as well as empirical benchmarking of the proposed methods on vision datasets.
Clarity: While the paper is written well and is easily accessible, the plots and the numbers in the tables were a bit small and thereby hard to read. I would suggest the authors to have bigger plots and tables in future revisions to ensure readability.
>> The authors mention in Section 4.1 that "support vector are points with non-zero loss": In all generality, this statement seems to be incorrect. For example, even for linearly separable data, a linear SVM would have support vectors which are correctly classified.
>> The experiment section seems to be missing a table on the statistics of the datasets used: This is important to understand the class distribution in the datasets used and if at all there was label imbalance in any of them. It looks like all the datasets used for experimentation had almost balanced class labels and in order to fully understand the scope of these sampling strategies, I would suggest the authors to also provide results on class imbalanced datasets where the distribution over labels is non-uniform.
>> Incomprehensive comparison with benchmarks:
a) The comparison of their methods with VSV benchmark seems incomplete. While the authors used the obtained support vectors as the augmentation set and argued that it is of fixed size, a natural way to extend these to any support size is to instead use margin based sampling where the margins are obtained from the trained SVM since these are inherently margin maximizing classifiers. Low margin points are likely to be more influential than high margin points.
b) In Section 5.3, a key takeaway is "diversity and removing redundancy is key in learning invariances". This leads to possibly other benchmarks to which the proposed policies could be compared, for example those based on Determinantal point processes (DPP) which are known for inducing diversity in subset selection. There is a large literature on sampling diverse subsets (based on submodular notions of diversity) which seems to be missing from comparisons. Another possible way to overcome this would be to use stratified sampling to promote equal representation amongst all classes.
c) In Section 2, it is mentioned that general methods for dataset reduction are orthogonal to the class of methods considered in this paper. However, on looking at the data augmentation problem as that of using fewest samples possible to learn a new invariance, it can be reduced to a dataset reduction problem. One way of using these reduction methods is to use the selected set of datapoints as the augmentation set and compare their performance. This would provide another set of benchmarks to which proposed methods should be compared.
>> Accuracy Metrics: While the authors look at the overall accuracy of the learnt classifiers, in order to understand the efficacy of the proposed sampling methods at learning invariances, it would be helpful to see the performance numbers separately on the original dataset as well as the transformed dataset using the various transformations.
>> Experiments in other domains: The proposed schemes seem to be general enough to be applicable to domains other than computer vision. Since the focus of the paper is the proposal of general sampling strategies, it would be good to compare them to baselines on other domains possibly text datasets or audio datasets.