Extracting relevant information, causally inferring and predicting the future states with high accuracy is a crucial task for modeling complex systems. The endeavor to address these tasks is made even more challenging when we have to deal with high-dimensional heterogeneous data streams. Such data streams often have higher-order inter-dependencies across spatial and temporal dimensions. We propose to perform a soft-clustering of the data and learn its dynamics to produce a compact dynamical model while still ensuring the original objectives of causal inference and accurate predictions. To efficiently and rigorously process the dynamics of soft-clustering, we advocate for an information theory inspired approach that incorporates stochastic calculus and seeks to determine a trade-off between the predictive accuracy and compactness of the mathematical representation. We cast the model construction as a maximization of the compression of the state variables such that the predictive ability and causal interdependence (relatedness) constraints between the original data streams and the compact model are closely bounded. We provide theoretical guarantees concerning the convergence of the proposed learning algorithm. To further test the proposed framework, we consider a high-dimensional Gaussian case study and describe an iterative scheme for updating the new model parameters. Using numerical experiments, we demonstrate the benefits on compression and prediction accuracy for a class of dynamical systems. Finally, we apply the proposed algorithm to the real-world dataset of multimodal sentiment intensity and show improvements in prediction with reduced dimensions.

The paper proposes a method to learn the conditional distribution of a random variable in order to minimize and maximize certain mutual information terms.  Interestingly, the proposed method can be applied to sentiment prediction and outperforms a 2018 method based on SVM.
Overall, the ideas seem intriguing, and the results seem promising, but I really cannot understand what the paper is saying, and I think the paper would be much stronger if it was written more clearly (to make individual sentences more clear, but also to make the broader picture more clear). Not only is the writing hard to understand (some sentences lack a verb!), but it is vague, and the notion of a "complex system" is never defined.  It seems that the technique can be applied to any (potentially non-stationary) Markov process?
Additionally, due to the lack of clarity in the writing and lack of mathematical rigor, Theorem 1 does not seem to be true as stated. I think this is an issue of stating the assumptions, and not due to a mistake in the derivation.  Right now, the actual conclusion of theorem 1 is not even clear to me.
Quality: poor/unclear
Clarity: very poor
Originality: unclear, perhaps high? Not clear how related it is to the methods of Tishby et al.
Significance: unclear, as clarity was poor, and there was minimal discussion of alternative methods.
Specific points:
- Eq (2), the first term is included because it is for the "information compression task", but I do not understand that. Where is the actual compression?  This is not traditional compression (turning a large vector into a smaller vector), but more like turning one PDF into a PDF with lower entropy?
- This paper seems to fall into the subfield of system identification (at which I am not an expert), so I'd expect to see some related literature in the field. The only compared method was the IF method of Tishby et al. from 18 years ago (and the current work seems to be a generalization of that).
- Equation (4): what exactly is the object being minimized? Is it a PDF/probability measure? Is it an *instance* of a random variable?  If it is a PDF, is it the PDF of B_k | X_{k-1} ?
- The statement of Theorem 1 is either too vague or wrong. To say "The solution... is given by" makes it sound like you are giving equations that define a unique solution. Perhaps you mean, "Any solution ... must necessarily satisfy..." ? And that is not clearly true without more work. You are basically saying that any minimizer must be a stationary point of the objective (since you are not assuming convexity). It seems everything is differentiable?  How do you know solutions even exist -- what if it is unbounded? In that case, these are not necessary conditions.
- Lemma 1: "The iterative procedure... is convergent."  The iterative procedure was never defined, so I don't even know what to make of this.
- Section 3.2: "As proved by prior work, the optimum solution obtained by a stochastic transformation that is jointly Gaussian with bottleneck's input."  I do not know what you are trying to say here. There's no predicate.
- Section 4 wasn't that interesting to me yet, since it was abstract and it seemed possible that you make a model to fit your framework well. But section 5 is much better, since you apply it to a real problem. However, what you are actually solving in section 5 is unclear. The entire setup is poorly described, so I am very confused.


The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.

This paper is thorough and well-written. On first look, the paper seems to be addressing a topic that I believe is already well-known in the DL community that has typically been explained by memory constraints or light empirical evidence. However, a more in-depth reading of this paper shows that the authors provide a serious attempt at implementing SVRG methods. This is demonstrated by their detailed implementation that attempts to overcome the main practical algorithmic concerns for neural networks (which may be beneficial even in the implementation of other optimization algorithms for deep learning) and their in-depth experiments that give concrete evidence towards a reasonable explanation of why SVRG methods currently do not work for deep learning. In particular, they claim that because the SVRG estimator fails to significantly decrease the variance, the increased computation is not worthwhile in improving the efficiency of the algorithm. Because the empirical study is fairly in-depth and thorough and the paper itself is well-written (particularly for DL), I’m more inclined to accept the paper; however, I still do not believe that the explanation is significant and novel enough to be worthy of publication, as I will explain below.
1. Performance of SVRG methods in convex setting
It is fairly well-known that even in the convex optimization community (training logistic regression), SVRG methods fail to improve the performance of SG in the initial phase; see [1, 5]. Often, the improvements in the algorithm are seen in the later phases of the algorithm, once the iterates is sufficiently close to the solution and the linear convergence rate kicks in.
The experiments for neural networks presented here seem to corroborate this; in particular, the variance reduction introduces an additional cost but without much benefit in the main initial phase (epochs up to 150).
One also typically observes (in the convex setting) very little difference in test error between SG and SVRG methods since it is unnecessary to train logistic regression to a lower error for the test error to stabilize. Hence, the test error results in the neural network setting feels unsurprising.
2. Comments/questions on experiments
(i) Batch Normalization: When you were obtaining poor results and divergence with applying SVRG directly with training mode on, what batch size were you using for the batch normalization? In particular, did you use full batch when computing the batch norm statistics at the snapshot point? Did you try fixing the batch normalization “ghost batch size” [4]?
(ii) Measuring Variance Reduction: When training the models, what other hyperparameters were tried? Was SVRG sensitive to the choices in hyperparameters? What happened when momentum was not used? How did the training loss behave? It may also be good to mention which datasets were used since these networks have been applied to a wider set of datasets.
(iii) Streaming SVRG Variants: Have you considered SVRG with batching, as proposed in [3]?
(iv) Convergence Rate Comparisons: Is it reasonable to use the same hyperparameter settings for all of the methods? One would expect that each method needs to be tuned independently; otherwise, this may indicate that SVRG/SCSG and the SG method are so similar that they can all be treated the same as the SG method.
3. Generalization
For neural networks, the question of generalization is almost as important as finding a minimizer efficiently, which is not addressed in-depth in this paper. The SG method benefits from treating both the empirical risk and expected risk problems “equally”, whereas SVRG suffers from utilizing this finite-sum/full-batch structure, which may potentially lead to deficiencies in the testing error. In light of this, I would suggest the authors investigate more carefully the generalization properties of the solutions of SVRG methods for neural networks. This may be highly relevant to the work on large-batch training; see [6].
Summary:
Overall, the paper is quite thorough and well-written, particularly for deep learning. However, the paper still lacks enough content and novelty, in my opinion, to warrant acceptance. They appeal to a simple empirical investigation of the variance as supportive evidence for their claim; if the paper had some stronger mathematical justification specific for neural networks demonstrating why the theory does not hold in this case, the paper would be a clear accept. For these reasons, I have given a weak reject. A response addressing my concerns above and emphasizing the novelty of these results for neural networks may push me the other way.
References:
[1] Bollapragada, Raghu, et al. "A progressive batching L-BFGS method for machine learning." arXiv preprint arXiv:1802.05374(2018).
[2] Friedlander, Michael P., and Mark Schmidt. "Hybrid deterministic-stochastic methods for data fitting." SIAM Journal on Scientific Computing 34.3 (2012): A1380-A1405.
[3] Harikandeh, Reza, et al. "Stopwasting my gradients: Practical svrg." Advances in Neural Information Processing Systems. 2015.
[4] Hoffer, Elad, Itay Hubara, and Daniel Soudry. "Train longer, generalize better: closing the generalization gap in large batch training of neural networks." Advances in Neural Information Processing Systems. 2017.
[5] Johnson, Rie, and Tong Zhang. "Accelerating stochastic gradient descent using predictive variance reduction." Advances in neural information processing systems. 2013.
[6] Keskar, Nitish Shirish, et al. "On large-batch training for deep learning: Generalization gap and sharp minima." arXiv preprint arXiv:1609.04836 (2016).
[7] Smith, Samuel L., Pieter-Jan Kindermans, and Quoc V. Le. "Don't Decay the Learning Rate, Increase the Batch Size." arXiv preprint arXiv:1711.00489 (2017).


We propose a novel algorithm, Difference-Seeking Generative Adversarial Network (DSGAN), developed from traditional GAN. DSGAN considers the scenario that the training samples of target distribution, p t , are difficult to collect. Suppose there are two distributions pd and p d such that the density of the target distribution can be the differences between the densities of pd and p d . We show how to learn the target distribution p t only via samples from p d and pd (relatively easy to obtain). DSGAN has the flexibility to produce samples from various target distributions (e.g. the out-of-distribution). Two key applications, semi-supervised learning and adversarial training, are taken as examples to validate the effectiveness of DSGAN. We also provide theoretical analyses about the convergence of DSGAN.

The paper presents a new Generative Adversarial Network (GAN) for learning a
target distribution that is defined as the difference between two other
distributions. Applications in semi-supervised learning and adversarial training
are considered in the experimental evaluation and results are presented in
computer vision tasks.
The paper is not very well written and can be hard to follow. One very important
issue for me was motivation for defining the target distribution as a difference
between two other distributions. I am not familiar with this area, but reading
through the introduction it was never clear to me why this is a useful scenario,
in practice. Furthermore, some statements in the introduction felt quite
arbitrary. For example, the authors state that PixelCNN "does not have a latent
representation" in a manner that makes it sound as if that is a bad thing. If
indeed it is, then why so? It would be very helpful to motivate the setting more
and to provide a couple of examples of where this method would be useful, in the
introduction. Also, regarding the MNIST example in the end of page 1, what is
the "universal set"? This paragraph also felt a bit arbitrary and unclear.
Some comments about the rest of the paper:
  - The theoretical results of section 3 are just stated/listed, but are not
    connected to algorithm 1. Please connect them to the different parts of the
    algorithm and state in a couple sentences what they imply for the algorithm.
  - Right after theorem 1, which assumption are you referring to when you say
    "the assumption in Theorem 1"?
  - The reformulation of section 3.1 is never justified. What led you to use
    this reformulation and why do you think it is more stable in practice?
  - You should mention in the caption of table 4, what quantity you are
    computing.
Note that my evaluation for this paper is based mainly on the way it is written
as, in its current state, it is hard for me to judge what is novel and what is
useful, and what readers are supposed to take in by reading this paper. The main
question that the paper definitely needs to answer, but does not do so currently
(in my opinion) is:
  When is this method useful to readers? For solving which problems and under
  what conditions? And also, when is this method bad and should not be used?
== Experiments ==
Section 5.1 is hard to follow and I don't quite get how it connects to the rest.
Also, in section 5.1.2 you mention that in comparison to Dai et al. (2017) your
method does not need to rely on an additional density estimation network. Even
if that is true, I cannot see how it is a useful remark given that the method of
Dai et al. seems to always beat your method.
== Style ==
In figure 1, no labels or legends are provided making it hard to figure out
what's going on at a glance. It would be very helpful to include labels and a
legend.
Equation 2 is not written correctly. The equals sign only refers to "V(G, D)"
and not the min-max of that, right? Please make that explicit by first defining
"V(G, D)" alone.


We propose a general method for various image restoration problems, such as denoising, deblurring, super-resolution and inpainting. The problem is formulated as a constrained optimization problem. Its objective is to maximize a posteriori probability of latent variables, and its constraint is that the image generated by these latent variables must be the same as the degraded image. We use a Generative Adversarial Network (GAN) as our density estimation model. Convincing results are obtained on MNIST dataset.

The authors propose a method for image restoration, where the restored image is the MAP estimate. A pretrained GAN is utilized to approximate the prior distribution of the noise-free images. Then, the likelihood induces a constraint which is based on the degradation function. In particular, the method tries to find the latent point for which the GAN generates the image, which if gets degraded will match the given degraded image. Also, an optimization algorithm is presented that solves the proposed constrained optimization problem.
I find the paper very well written and easy to follow. Also, the idea is pretty clean, and the derivations are simple and clear. Additionally, the Figures 2,3 are very intuitive and nicely explain the theory. However, I think that there are some weaknesses (see comments):
Comments:
#1) I do not understand exactly what the "general method" means. Does it mean that you propose a method, where you can just change the F, such that to solve a different degradation problem? So you provide the general framework where somebody has to specify only the F?
#2) Clearly, the efficiency of the method is highly based on the ability of the GAN to approximate well the prior distribution of the noise-free images.
#3) There are several Equations that can be combined, such that to save enough white space in order to discuss further some actual technical details. For instance, Eq. 2,3 can be easily combined using the proportional symbol, Eq. 8,9,10,11 show actually the same thing.
#4) I think that the function F has to be differentiable, and this should be mentioned in the text. Also, I believe that some actual (analytic) examples of F should be provided, at least  in the experiments. The same holds for the p(Omega). This parameter Omega is estimated individually for each degraded image?
#5) Before Eq. 8 the matrix V is a function of z and should be presented as such in the equations.
#6) I believe that it would be nice to include a magnified image of Fig. 3, where the gradient steps are shown. Also, my understanding is that the optimization goal is to find first a feasible solution, and then find the point that maximizes f. I think that this can be clarified in the text.
#7) The optimization steps seem to be intuitive, however, there is not any actual proof of converge. Of course, the example in the Figure 3 is very nice and intuitive, but it is also rather simple. I would suggest, at least, to include some empirical evidences in the experiments that show convergence.
#8) In the experiments I think that at least one example of F and p(Omega) should be presented. Also, what the numbers in Table 4 show? Which is the best value that can be achieved? These numbers correspond to several images, or to a unique image?
#9) I think that MNIST is almost a toy experiment, since the crucial component of the proposed method is the prior modeling with the GAN. I believe that a more challenging experiment should be conducted e.g. using celebA dataset.
Minor comments:
#1) In the paragraph after Eq. 4 the equality p_r(x)=p_G(x) is very strong assumption. I would suggest to use the \simeq symbol instead.
#2) After Eq. 6 the "nonnegative" should be "nonzero".
#3) Additional density estimation models can be used e.g. VAEs, GMM. Especially, I believe that the VAE will provide a way to approximate the prior easier than the GAN.
#4) In Section 2 paragraph 2, the sentence "However, they only ... and directly" is not clear what means.
In general, I find both the proposed model and optimization algorithm interesting. Additionally, the idea is nicely presented in the paper. Most of my comments are improvements which can be easily included. The two things that make me more skeptical, is the convergence of the proposed algorithm and the experiments. The MNIST is a relatively simple experiment, and I would like to see how the method works in more challenging problems. Also, I think that additional methods to compute the image prior should be included in the experiments.


Recent advances in neural Sequence-to-Sequence (Seq2Seq) models reveal a purely data-driven approach to the response generation task. Despite its diverse variants and applications, the existing Seq2Seq models are prone to producing short and generic replies, which blocks such neural network architectures from being utilized in practical open-domain response generation tasks. In this research, we analyze this critical issue from the perspective of the optimization goal of models and the specific characteristics of human-to-human conversational corpora. Our analysis is conducted by decomposing the goal of Neural Response Generation (NRG) into the optimizations of word selection and ordering. It can be derived from the decomposing that Seq2Seq based NRG models naturally tend to select common words to compose responses, and ignore the semantic of queries in word ordering. On the basis of the analysis, we propose a max-marginal ranking regularization term to avoid Seq2Seq models from producing the generic and uninformative responses. The empirical experiments on benchmarks with several metrics have validated our analysis and proposed methodology.

The paper investigates the problem of universal replies plaguing the Seq2Seq neural generation models. The problem is indeed quite important because for problems with high entropy solutions the seq2seq models have been shown to struggle in past literature. While the authors do pick a good problem, that's where the quality of the paper ends for me. The paper goes on an endless meandering through a lot of meaningless probabilistic arguments.  First of all, factorizing a seq2seq model as done in equation 1 is plain wrong. The model doesn't operate by first selecting a set of words and then ordering them. On top of this wrong factorization, section 2.2 & 2.3 derives a bunch of meaningless lemmas with extremely crude assumptions. For example, for lemma 3, M is supposed to be some universal constant defined to be the frequency of universal replies while all other replies seem to have a frequency of 1. Somehow through this wrong factorization and some probabilistic jugglery, we arrive at section 3 where the takeaway from section 2 is the rather known one that the model promotes universal replies regardless of query.
In section 3, the authors then introduce the "max-marginal regularization" which is a linear combination of log-likelihood and max-margin (where the score is given by log-likelihood) losses. Firstly, the use of word "marginal" instead of "margin" seems quite wrong to say the least.  Secondly, the stated definition seems to be wrong. In the definition the range of values for \gamma is not stated. I consider the two mutual exclusive and exhaustive cases (assuming \gamma not equals 0) below and show that both have issues:
(a) \gamma > 0: This seems to imply that when the log-likelihood of ground-truth is already \gamma better than the log-likelihood of the random negative, the loss comes to life. Strange!
(b) \gamma < 0: This is again weird and doesn't seem to be the intended behavior from a max-margin{al} loss.
I'm assuming the authors swapped y with y^{-} in the "regularization" part.
Anyways, the loss/regularization doesn't seem to be novel and should have been compared against pure max-margin methods as well.
Coming to the results section, figure 3 doesn't inspire much confidence in the results. For the first example in figure 3, the baseline outputs seem much better than the proposed model, even if they follow a trend, it's much better than the ungrammatical and incomprehensible sentences generated by the proposed model. Also there seems to be a discrepancy in figure 3 with the baseline output for first query having two "Where is your location?" outputs.  The human column of results for Table 3 is calculated over just 100 examples which seems quite low for any meaningful statistical comparison. Moreover, not quite sure why the results used the top-10 results of beam instead of the top-1.
A lot of typos/wrong phrasing/wrong claims and here are some of them:
(a) Page 1, "lead to the misrecognition of those common replies as grammatically corrected patterns"? - No idea what the authors meant.
(b) Page 1, "unconsciously preferred" - I would avoid attaching consciousness before AGI strikes us.
(c) Page 1, "Above characters" -> "Above characteristics"
(d) Page 1, "most historical" -> "most previous"
(e) Page 2, "rest replies" -> "rest of the replies"
(f) Page 3, "variational upper bound" -> Not sure what's variational about the bound
(g) "Word Perplexity (PPL) was used to determine the semantic context of phrase-level utterance"? - No idea what the authors meant.


Meta-reinforcement learning aims to learn fast reinforcement learning (RL) procedures that can be applied to new tasks or environments. While learning fast RL procedures holds promise for allowing agents to autonomously learn a diverse range of skills, existing methods for learning efficient RL are impractical for real world settings, as they rely on slow reinforcement learning algorithms for metatraining, even when the learned procedures are fast. In this paper, we propose to learn a fast reinforcement learning procedure through supervised imitation of an expert, such that, after meta-learning, an agent can quickly learn new tasks through trial-and-error. Through our proposed method, we show that it is possible to learn fast RL using demonstrations, rather than relying on slow RL, where expert agents can be trained quickly by using privileged information or off-policy RL methods. Our experimental evaluation on a number of complex simulated robotic domains demonstrates that our method can effectively learn to learn from spare rewards and is significantly more efficient than prior meta reinforcement learning algorithms.

This work addresses the problem of learning a policy-learning-procedure, through meta-learning, that can adapt quickly to new tasks. This work uses MAML for meta-learning, and with this choice, the problem can be broken down into two loops:
1) inner loop: adapting a policy \pi_phi based on unseen rollouts, where initial parameters phi were provided by the meta-trainer in the outer loop
2) outer loop: the meta-trainer tries to learn parameters phi on batches of tasks that provide good initial parameters
In prior work on meta-reinforcement learning via MAML, both the outer as well as inner objective attempt to minimize a RL objective, leading to an algorithm that has very high sample-complexity. This work uses imitation learning for the outer loop procedure, to significantly decrease sample-complexity.
Technical Contribution:
-----------------------
The idea of using imitation learning for reinforcement learning is well explored in the literature, and so using this idea in itself is not real contribution. There are several issues with the presentation of this work, that make it incredibly difficult to identify a technical contribution:
1. overreaching statements without details to backup: you are writing the paper as if you are learning a "RL algorithm" that can be used to quickly learn new tasks. your manuscript does not really provide a description for this "algorithm". After re-reading several other papers I concluded that what you mean is that you learn an initial set of policy parameters that can quickly adapt to new related tasks and an update rule with which you update these parameters. However, standard MAML uses SGD as an update rule so there is really nothing to be learned here. Unfortunately, your paper provides zero detail on these claims of learning a "RL procedure", so for now I have to assume that you are simply learning a good initial set of policy parameters through meta-learning. If that is the case, then using imitation learning in this setting is really not novel, this has been done by a lot of other people before (you're just using MAML to learn "better" initial parameters).
2. you're technical section (section 4) provides some details on the technical challenges of using demonstrations to perform the outer loop optimization step. Unfortunately, you are not putting your work in the context of existing work ([1], [2]), that discuss and address the importance/issue of sampling in meta-rl with MAML. So it's impossible to know whether there is any new insight here
Experimental Evaluation:
-------------------------
The experimental evaluation is very "thin", other than the original MAML-RL and pure imitation learning no other more recent baselines ([1], [2]) have been compared to. And only 2 relatively simple simulation settings are tested.
Summary:
-----------
Very minor contribution, a manuscript that is lacking important details and does not relate it's technical section to existing work, with very thin evaluation.
[1] The Importance of Sampling in Meta-Reinforcement Learning, NIPS 2018
[2] CONTINUOUS ADAPTATION VIA META-LEARNING IN NONSTATIONARY AND COMPETITIVE ENVIRONMENTS, ICLR 2018


Question generation is an important task for improving our ability to process natural language data, with additional challenges over other sequence transformation tasks. Recent approaches use modifications to a Seq2Seq architecture inspired by advances in machine translation, but unlike translation the input and output vocabularies overlap significantly, and there are many different valid questions for each input. Approaches using copy mechanisms and reinforcement learning have shown promising results, but there are ambiguities in the exact implementation that have not yet been investigated. We show that by removing inductive bias from the model and allowing the choice of generation path to become latent, we achieve substantial improvements over implementations biased with both naive and smart heuristics. We perform a human evaluation to confirm these findings. We show that although policy gradient methods may be used to decouple training from the ground truth and optimise directly for quality metrics that have previously been assumed to be good choices, these objectives are poorly aligned with human judgement and the model simply learns to exploit the weaknesses of the reward source. Finally, we show that an adversarial objective learned directly from the ground truth data is not able to generate a useful training signal.

The paper studies question generation, which is an important problem in many real applications. The authors propose to use better caching model and more evalution methods to deal with the problem. However, the paper is poorly written and hard to follow, and the proposed model lacks of novelty. The main reasons are as below:
1) In model section, the task definition is not clear. It is expected to see what's the question generation task studied in this paper. An example or a model overview will definitly help.
2) The encoder and decoder are not novel, it is expected to cite and compare with the existing similar encoder architecture, such as the encoder proposed in bidaf "Seo, Minjoon, et al. "Bidirectional attention flow for machine comprehension." arXiv preprint arXiv:1611.01603 (2016)."  The math symbols are aligned, for example, h_a or h^a is used to represent the encoding. Besides, adding the binary feature in the embedding is not necessary, the LSTM model could learn such sequential correlation. The decoder description is not clear as well and expected to compare with existing work (e.g., bidaf) to show the difference.
3) The proposed copy mecahnism is not clear. A formal definition of s_t, v_t and y_(t-1) should be given before defining the p_t. A more serious question, what is the fuse operation used to define p_t? concat, elementwise_plus or others?
4) In the training, how to deal the ground truth that are not in the vocab? The authors stated "using a modified heuristic described below", but no follow-ups in the paper.
5) The paper is not well written and organized. Small typos: in introduction, 'and and answer span', 'and output and output sequences'. In model, 'Glorot initialization', 'Bahdanau attention', it is not the common way to cite others' work. In encoder, the defintion of the state for decoder could be reorganized to the decoder.
I have read the authors' detailed rebuttal. Thanks.


We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained on the full data. We show that the improved performance is due to the increased diversity of reformulation strategies.

The authors proposed a variant of ensemble method in reinforcement learning for query reformulation. They train multiple specialized sub-agents on disjoint partitions of the training data, and use a meta-agent, which can see all the training data, to decide the final answer. This can speed up the training thanks to parallelization. They observed that this can improve the diversity of learnt reformulations and the overall performance in some cases.
Strengths
1. The paper is clear and easy to follow.
2. Multiple evaluation metrics and baseline models are considered
Weaknesses
1. The proposed method is simple and lacks novelty.
2. The performance improvement is marginal and some empirical results are not carefully analyzed.
Significance
Exploring a diverse set of strategies is beneficial in reinforcement learning. This paper focuses on two aspects of this problem. One is how to learn diverse agents and the other one is how to efficiently learn these agents efficiently, which are important concerns in practice.
Originality
The model learning approach they proposed is merely a simple variant of ensemble learning. The main difference is they train sub-agents on disjoint partitions of the training data, which seems a trivial modification although this shows to improve the overall model performance.
Technical Quality
Overall, the experiments are well-thought, but the following questions need to be explained:
1.	In the Introduction, the authors claim three contributions they made in this paper. My question is, if the third one is really an important contribution, why didn’t the authors demonstrate it in detail in the main text? Attaching it to the appendix could make the reader confused about its significance.
2.	In Table-1, the authors claim that their proposed architectures can outperform the baseline RL-10-Ensemble with only 1/10 time. The sub-agents are trained on a partition of the training set. My question is, are these sub-agents trained in parallel on different machine? If so, why cannot the RL-10-Ensemble be trained in parallel through some multithread or distributed computation? The implementation of the proposed model and the baseline seems not that fair.
3.	The main architecture is described in section 3.3 and 3.4, including the Sub-agents and the Aggregator. However, in Appendix C.1, the authors claim that the gains the proposed method comes mostly from the pool of diverse reformulators, and not from the simple use of a re-ranking function (Aggregator). This is confusing because if it is true, the proposed method is really reduced to an ensemble of the baseline model.
4.	In Table-2, some of the results are worse than the baseline methods like Re-Ranker. Although the authors claim the re-ranking is a post-processing, Re-Ranker performs significantly better than the proposed model. If the authors want to better demonstrate the advantages of the proposed model, a comparison between the proposed model with re-ranking and the Re-Ranker is required.
5.	In Table 10, why the proposed method fails to produce the right answer whereas the other methods perform well?


ATTENTION, LEARN TO SOLVE ROUTING PROBLEMS!
The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development. However, to push this idea towards practical implementation, we need better models and better ways of training. We contribute in both directions: we propose a model based on attention layers with benefits over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we find is more efficient than using a value function. We significantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes. With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.

This paper proposes an alternative deep learning model for use in combinatorial optimization. The attention model is inspired by the Transformer architecture of Vaswani et al. (2017). Given a distribution over problem instances (e.g. TSP), the REINFORCE update is used to train the attention model. Interestingly, the baseline used in the REINFORCE update is based on greedy rollout using the current model. Experimentally, four different routing problems are considered. The authors show that the proposed method often outperforms some other learning-based methods and is competitive with existing (non-learned) heuristics.
Overall, this is a good piece of work. Next, I will touch on some strengths and weaknesses which I hope the authors can address/take into account. My main concern is the lack of comparison with Deudon et al. (2018).
Strengths:
- Writing: beautifully written and precise even with respect to tiny technical details; great job!
- Versatility: the experimental evaluation on four different routing problems with different kinds of objectives and constraints, different baseline heuristics, etc., is quite impressive (irrespective of the results). The fact that the proposed model can be easily adapted to different problems is encouraging, since many real-world operational problems may be different from textbook TSP/VRP, and hard to design algorithms for; a learned algorithm can greatly expedite the process. This versatility is shared with the model in Dai et al. (2017) which applied to different graph optimization problems.
- Choice of baseline: the use of the greedy policy is definitely the right thing to do here, as one wants to beat "simpler" baselines.
- Results: the proposed method performs very well and does not seem hard to tune, in that the same model hyperparameters work well across different problems.
Weaknesses:
- Comparison to Deudon et al. (2018): I believe the authors should do more work to compare against Deudon et al. (2018). This includes expanding the sentence in related work, describing the differences in more detail; perhaps a side-by-side graphical comparison of the two models in the appendix would help; reporting results from or running the code of that paper for the relevant problems (TSP?). This is crucial, since that paper also builds on the Transformer architecture, attention, etc. Its code is also online and seems to have been up for a while (https://github.com/MichelDeudon/encode-attend-navigate). There is quite some overlap, and the reader should be able to understand how the two models/papers differ.
- Intuition: One thing that is lacking here is some intuitive explanation of *why* this particular attention model is a reasonable choice for guiding a combinatorial algorithm. For instance, earlier work such as Pointer networks or S2V-DQN each addressed certain issues with other models of the time (e.g. capturing graph structure in S2V-DQN). If the choice of the model is purely performance-driven, that is completely fine, but then it makes sense to walk the reader through the process that got you to the final model. You do some of that in the ablation study in 5.2, for the baseline. Additionally, I am wondering about why this attention model is good for a combinatorial problem.
Questions/suggestions:
- Performance metric: if I understand correctly, Table 1 reports objective values. Could you additionally report optimality gaps compared to the best solution found *across* methods (including Gurobi, especially when it solves to optimality for the smaller problems/all of TSP)? Otherwise, it is very hard to interpret the differences in absolute objective values across methods.
- Baseline: could you use a non-learned baseline (e.g. 2-opt for the case of TSP) at the beginning of the training (then go to your learned but greedy baseline)? Might this give a stronger baseline at the beginning and accelerate training?


We demonstrate how to learn efficient heuristics for automated reasoning algorithms through deep reinforcement learning. We focus on backtracking search algorithms for quantified Boolean logics, which already can solve formulas of impressive size -up to 100s of thousands of variables. The main challenge is to find a representation of these formulas that lends itself to making predictions in a scalable way. For challenging problems, the heuristic learned through our approach reduces execution time by a factor of 10 compared to the existing handwritten heuristics.

The paper proposes an approach to automatically learning variable selection
heuristics for QBF using deep learning. The evaluation presented by the authors
shows the promise of the method and demonstrates significant performance
improvements over a variable selection heuristic that does not use machine
learning.
In practice, the overhead of the proposed method is likely to be a major
obstacle in its adoption. The authors note the difficulty of finding suitable
benchmarks and restrict the set of instances they use for evaluation to formulae
where the proposed method is likely to achieve improvements. This skews the
evaluation in favor of the proposed method; in particular, the 90% improvement
figure mentioned in the abstract is not representative of the general case.
Indeed, on another set of instances the proposed method falls significantly
short of the performance of a state-of-the-art heuristic that does not employ
learning.
A drawback of the paper is that there is no comparison to related work. I
realize that this is difficult to achieve because other approaches are in
related, but different areas and may be difficult to adapt for this case, but a
general comparison to the improvements other approaches achieve would be
helpful.
Nevertheless, the work is interesting and presents a new angle on using machine
learning to speed up combinatorial problem solving. While several issues hinder
practical adoption, this is likely to lead to interesting follow-up work that
will improve problem solving in practice.
The description of the method (Section 4.1) is short and not detailed enough to
reproduce the approach the authors are proposing. However, the code is
available.
In summary, I feel that the paper can be accepted.


OFF-POLICY EVALUATION AND LEARNING FROM LOGGED BANDIT FEEDBACK: ERROR REDUCTION VIA SURROGATE POLICY
When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges. Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a collection of logged feedback from the actions taken by a historical policy, and expect to learn a policy that takes good actions in possibly unseen contexts. Such a batch learning setting is ubiquitous in online and interactive systems, such as ad platforms and recommendation systems. Existing approaches based on inverse propensity weights, such as Inverse Propensity Scoring (IPS) and Policy Optimizer for Exponential Models (POEM), enjoy unbiasedness but often suffer from large mean squared error. In this work, we introduce a new approach named Maximum Likelihood Inverse Propensity Scoring (MLIPS) for batch learning from logged bandit feedback. Instead of using the given historical policy as the proposal in inverse propensity weights, we estimate a maximum likelihood surrogate policy based on the logged action-context pairs, and then use this surrogate policy as the proposal. We prove that MLIPS is asymptotically unbiased, and moreover, has a smaller nonasymptotic mean squared error than IPS. Such an error reduction phenomenon is somewhat surprising as the estimated surrogate policy is less accurate than the given historical policy. Results on multi-label classification problems and a large-scale ad placement dataset demonstrate the empirical effectiveness of MLIPS. Furthermore, the proposed surrogate policy technique is complementary to existing error reduction techniques, and when combined, is able to consistently boost the performance of several widely used approaches.

The paper proposes to fit a model of the logging policy that generates bandit feedback data, and use this model's propensities when performing off-policy optimization. When the model is well-specified (i.e. the logging policy indeed lies within the parametric class of models we are fitting), and we use maximum likelihood estimation to fit the model, this approach can yield a lower error when evaluating a policy's performance using off-policy data. The paper then shows how this improved off-policy estimation can also yield better off-policy optimization, and demonstrate this in semi-synthetic experiments.
Specific Comments:
Eq2.4: Lambda is overloaded (context distribution vs. regularization hyper-parameter).
Eq3.3: E[.] is used before defining it (i.e., E[.] should be interpreted as E_(x,a)~mu(.|beta*) [.])
Eq3.5: I^-1(beta*) makes sense, but the second term E[ d/d beta (S(x,a; beta*)) ] uses a notation that needs to be introduced (you mean || (E[ d/d beta (S(x,a; beta)) ] |_at beta=beta* )^-1 ||).
After Eq3.7: It will be instructive to specify some examples of logging policies mu which satisfy these assumptions (and how big the O(.) constants are for those examples).
Section 3.2: In practical considerations, expected a discussion of how robust things are when the logging policy class is mis-specified (i.e. assuming there is a beta* such that mu(.|beta*) created the data is unlikely to be true).
For ML- approaches, was a clipping constant M still used? If so, was it crucial and why?
Lemma D.1: The lemmas in appendix should be accompanied by a proof. E.g. what is C_beta? I don't immediately see why D.3 suggests that the inverse of the Fisher matrix has bounded norm (for instance, if x=0 the inverse is undefined).
General Comments:
Clarity: Good. The paper is easy to follow. Some examples from the Appendix can be moved to the main text (especially to provide a firm grounding for the constants appearing in Section3.1)
Correctness: I did not step through Appendix A-C. In Appendix D, there was a questionable claim. The stated theorems in the main text are believable [not surprising that asymptotic bias vanishes when the logging policy model is well-specified].
Originality: This builds on several previous works on off-policy optimization in bandit settings, and proposes a simple addition to improve performance.
Significance: The paper seems to have missed an opportunity; it can be substantially stronger with a more careful study of when fitting the logging policy will help vs. hurt, and what kinds of regularization or alternatives to maximum likelihood estimation can yield similar improvements (e.g. regularizing propensities close to uniform, ensure no small propensities).


In autonomous vehicle (AV) control, requiring an agent to make mistakes, or even allowing mistakes, can be quite dangerous and costly in the real world. For this reason we investigate methods of training an AV without allowing the agent to explore and instead having a human explorer collect the data. Supervised learning has been explored for AV control, but it encounters the issue of the covariate shift. That is, training data collected from an optimal demonstration consists only of the states induced by the optimal control policy, but at runtime, the trained agent may encounter a vastly different state distribution and has seen little relevant training data. To mitigate this issue, we intentionally have our human explorer make sub-optimal decisions. In order to have our agent not replicate these suboptimal decisions, supervised learning requires that we either erase these actions, or replace these action with the correct action. Erasing these actions is wasteful and replacing these actions is difficult, since it is not easy to know the correct action without driving the car. Since supervised learning falls short, we introduce an alternate framework that includes continuous scalar feedback on each action, marking which actions we should replicate, which we should avoid, and how sure of this we are. Our framework, called "ReNeg", learns from from human demonstration and human evaluative feedback, collected entirely before any training begins. Our agent learns continuous control from sub-optimal behavior, but sub-optimal behavior that is safely performed by a human. We find that a human demonstrator can explore sub-optimal states in a safe and controlled manner, while still getting enough gradation in the states to benefit learning. The way we collect data and the algorithm we use to compute feedback on the data we call "Backseat Driver." Backseat Driver gives us state-action pairs matched with scalar values representing the score for those action in those states. We call the more general learning framework ReNeg, since it learns a regression from states to actions given negative as well as positive examples. We empirically validate several models in the ReNeg framework, testing on lane-following with limited data. We find that the best solution is a generalization of mean-squared error and outperforms supervised learning on the positive examples alone.

Despite many high profile successes in research, DeepRL is still not widely used in applications. One reason for this is that current methods typically assume the agent can learn by exploring many states and actions, however, in many real world tasks, such as driving used here, poor actions can be dangerous. Therefore, methods that can provides the flexible benefits of RL while avoiding this are of significant interest, one promising general ideas pursued for this has been to use human demonstrations.
A number of approaches to Inverse RL have been studied, but many make the assumption that the demonstrations are all examples of optimal behavior. This can be challenging if, for example, some examples contain suboptimal behavior, and it also means that the agent does not get to observe non-optimal trajectories and how to correct for them; the resulting policy often performs poorly due to the distributional shift between the demonstration trajectories and the trajectories induced by the learned policy.
This work attempts to correct for these problems by labeling the demonstration actions between $[-1, 1]$ indicating how good or bad the demonstration actions are. This introduces a challenge for learning, since good actions can be copied, but a bad action is more ambiguous: it does not necessarily imply the action are far away from the bad action is a good action.
One view of this work is that they introducing 3 losses for behavior cloning with weighted labels: A weighted (based on the label) L2 loss, an exponential loss and directly fitting the loss and searching over a discrete set of actions to find the highest estimate weighting. Note the current equation for $Loss_{FNET}$ doesn't make sense because it simply minimizing the output of the network, from the text it should be something like $(f - \hat{\theta})^2$?
The text discusses why rescaling the negative examples may be beneficial, but as far as I can tell, figure 4 you only consider $\alpha=\{0, 1\}$? Based on the text, why weren't intermediate values of $\alpha$ considered?
It could benefit from copy-editing, checking the equations and in some cases describing concepts more concisely using clear mathematical notation instead of wordy descriptions that are difficult to follow.
``Thus the assumption
that our training data is independent and identically distributed (i.i.d.) from the agent’s encountered
distribution goes out the window'' This is a misleading statement regarding the challenge of distributional shift in off-policy RL. The challenge is that state distribution between the behavior policy and the learned policy may be quite different, not just not iid. Even in on-policy RL the state visitation is certainly not usually iid.
``In the off-policy policy gradient RL framework, this issue is typically circumvented by changing the
objective function from an expectation of the learned policy’s value function over the learned policy
state-visitation distribution to an expectation of the learned policy’s value function over the behavior
(exploratory) state-visitation distribution (Degris et al., 2012). In the RL framework, this could be
dealt with by an approximation off-policy stochastic policy gradient that scales the stochastic policy
gradient by an importance sampling ratio (Silver et al., 2014) (Degris et al., 2012). ''. The importance sampling in Degris is not to correct for the objective being under the behavior state policy and DPG (Silver et al., 2014) specifically does not require importance sampling so it shouldn't be referenced here. This paragraph seems to be conflating two issues: the distributional shift between the behavior state distribution and the policy state distribution that can make off-policy learning unstable, and importance sampling to estimate outcome likelihoods using behavior experience.
This work is on a very important topic. However, in its current form it is not well-communicated. Additionally, the best performing method is not novel (as the author's state $\alpha=1$, the best performing setting, is essentially the same as COACH but with scalar labels). For these reasons reason, I think this work may be of limited interested.


Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation. We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation. We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufficient for the learning to take off, showing the practical aspect of the method.

Paper Summary:
The idea of the paper is to improve Hindsight Experience Replay by providing natural language instructions as intermediate goals.
Paper Strengths:
Unfortunately, there is not many positive points about the paper except that it explores an interesting direction.
Paper Weaknesses:
I vote for rejection of the paper due to the following issues:
- It is not clear how a description for a point along the way is provided (when the agent is not at a target). It is not clear how those feedback sentences are generated. That is the main claim of the paper and it is not clear at all.
- The result of DQN is surprising (it is always zero). DQN is not that bad. Probably, there is a bug in the implementation. There should be comments on this in the rebuttal.
- According to several recent works, algorithms like A3C work much better than DQN. Does the proposed method provide improvements over A3C as well?
- The only measure that is reported is success rate. The episode length should be reported as well. I suggest using the SPL metric proposed by Anderson et al. in "On Evaluation of Embodied Navigation Agents".
- Replacing one word with its synonym is considered as zero-shot. That is not really a zero-shot setting. Please refer to the following paper, which is missing in the related work:
Interactive Grounded Language Acquisition and Generalization in a 2D World, ICLR 2018
- The environments are toy environments. The experiments should be carried out in more complex environments such as THOR or House3D that include more semantics.
- What is the difference between this method and providing a large negative reward at a non-target object?
- The paper discusses the advantages of word embeddings over one-hot vectors. That is obvious and not the goal of this paper.
- It seems the same environment is used for train and test.
------------------------
Post rebuttal comments:
Most of my concerns have been addressed. My new rating is 5. I like the idea of having a compact representation for the hindsight experience replay, but there are still a few issues:
- I expected more complexity in vision and language. I do not agree with the rebuttal that AI2-THOR or House3D are not suitable. This level of complexity would be ok if this paper was among the first ones to explore this domain, but there are already several works. The zero-shot setting (changing the word with its synonym) is also so simplistic.
- The proposed method uses much more annotations than the baselines so the comparisons are not really fair. This information should have been added to the baseline to see how this additional information changes the performance. Basically, it is not clear if the improvement should be attributed to the extra annotation or the way the advice is given.
- The writing is still confusing. For instance, it is mentioned that "Concretely, for each state s ∈ S, we define T as a teacher that gives an advice T(s)", while that is not true since later it is mentioned that "the teacher give advice based solely on the terminal state". These statements are contradictory, and it is not trivial at all to provide an advice for each state.


VON MISES-FISHER LOSS FOR TRAINING SEQUENCE TO SEQUENCE MODELS WITH CONTINUOUS OUTPUTS
The Softmax function is used in the final layer of nearly all existing sequence-tosequence models for language generation. However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint. We propose a general technique for replacing the softmax layer with a continuous embedding layer. Our primary innovations are a novel probabilistic loss, and a training and inference procedure in which we generate a probability distribution over pre-trained word embeddings, instead of a multinomial distribution over the vocabulary obtained via softmax. We evaluate this new class of sequence-to-sequence models with continuous outputs on the task of neural machine translation. We show that our models train up to 2.5x faster than the state-of-the-art models while achieving comparable translation quality. These models are capable of handling very large vocabularies without compromising on translation quality or speed. They also produce more meaningful errors than the softmax-based models, as these errors typically lie in a subspace of the vector space of the reference translations 1 .

This paper describes a technique for replacing the softmax layer in sequence-to-sequence models with one that attempts to predict a continuous word embedding, which will then be mapped into a (potentially huge) pre-trained embedding vector via nearest neighbor search. The obvious choice for building a loss around such a prediction (squared error) is shown to be inappropriate empirically, and instead a von Mises-Fisher loss is proposed. Experiments conducted on small-data, small-model, greedy-search German->English, French->English and English->French scenarios demonstrate translation quality on par with BPE, and superior performance to a number of other continuous vector losses. They also provide convincing arguments that this new objective is more efficient in terms of both time and number of learned parameters.
This is a nice innovation for sequence-to-sequence modeling. The technical contribution required to make it work is non-trivial, and the authors have demonstrated promising results on a small system. I’m not sure whether this has any chance of supplanting BPE as the go-to solution for large vocabulary models, but I think it’s very healthy to add this method to the discussion.
Other than the aforementioned small baseline systems, this paper has few issues, so I’ll take some of my usual ‘problems with the paper’ space to discuss some downsides with this method. First: the need to use pre-trained word embeddings may be a step backward. It’s always a little scary to introduce more steps into the pipeline, and it’s uncomfortable to hear the authors state that they may be able to improve performance by changing the word embedding objective. As we move to large training sets, having pre-trained embeddings is likely to stop being an advantage and start being a hindrance. Second: though this can drastically increase vocabulary sizes, it is still a closed vocabulary model, which is a weakness when compared to BPE (though I suppose you could do both).
Smaller issues:
First paragraph after equation (1): “the hidden state … t, h.” -> “the hidden state h … t.”
Equation (2): it might help your readers to spell out how setting \kappa to ||\hat{e}|| allows you to ignore the unit-norm assumption of \mu.
“the negative log-likelihood of the vMF…” - missing capital
Unnumbered equation immediately before “Regularization of NLLvMF”: C_m||\hat{e}|| is missing round brackets around ||\hat{e}|| to make it an argument of the C_m function.
Is predicting the word vector whose target embedding has the highest value of vMF probability any more expensive than nearest neighbor search? Does it preclude the use of very fast nearest neighbor searches?
It might be a good idea to make it clear in 4.3 that you see an extension to beam search for your method to be non-trivial (and that you aren’t simply leaving out beam search for comparability to the various empirical loss functions). This didn’t become clear to me until the Future Work section.
In Table 5, I don’t fully understand F1 in terms of word-level translation accuracy. Recall is easy to understand (does the reference word appear in the system output?) but precision is harder to conceptualize. It might help to define the metric more carefully.


DYNAMIC SPARSE GRAPH FOR EFFICIENT DEEP LEARNING
We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimensionreduction search and obtains the BN compatibility via a double-mask selection. Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.

[Overview]
In this paper, the authors proposed to use dynamic sparse computation graph for reducing the computation memory and time cost in deep neural network (DNN). This method is applicable in both DNN training and inference. Unlike most of previous work that focusing on the reduction of computation during the inference time, this new method propose a dynamic computation graph by pruning the activations on the fly during the training of inference, which is an interesting and novel exploration. In the experiments, the authors performed extensive experiments to demonstrate the effectiveness of the proposed method compared with several baseline methods and original models. It is clear to me that this method helps to reduce the memory cost and computation cost for both DNN training and inference.
[Strengthes]
1. This paper addresses the computational burden in both memory and time from a novel angle than previous network pruning methods. It can be applied to reduce the computation in both network training and inference, but also preserve the representation ability of the network.
2. To endow the network compression in training and inference, the authors proposed to mute the low-activated neurons so that the computations merely happened on those selected neurons.
3. For the selection, the authors proposed a simple but efficient dimension reduction methods, random sparse projection, to project the original activations and weights into a lower-dimensional space and compute the approximated response map in such a lower dimension space, which the selection is based on.
4. The authors performed comprehensive experiments to demonstrate the effectiveness the proposed method for network compression. Those results are insightful and solid.
[Questions]
1. Is the sparsity of each layer the same across the whole network? It would be nice if the authors could perform some ablation studies on varied sparsity in different layers, maybe just with some heuristic methods, e.g., decreasing the sparsity from lower layer to upper layers. As the authors mentioned, higher sparsity causes a larger degradation on deeper network. I am curious that whether there are some better way to set the sparsity.
2. During the training of the network, how the activation evolve? It would be interesting to show how the selected activation changes across the training time for the same training sample.  This might provide some insights on when the activations begin to converge to a stable state, and how it varies layer by layer.
3. Following the above questions, is there any stage that the sparsity can be fixed without further computation for selection. In generally, the training proceeds for a number of epochs. It would be nice if we can observe some convergence on the selected activations and then we can suspend the selection for saving the computation burden.
[Conclusion]
This paper present an interesting and novel approach for network pruning in both training and inference. Unlike most of the previous work, it pruning the activations in each layer though a dimension reduction strategy. From the experiments, this method achieved an obvious improvement for reducing the computation memory and time cost in training and inference stages. I think this paper has prompted a new direction of efficient deep neural network.


UNSUPERVISED LEARNING OF THE SET OF LOCAL MAXIMA
This paper describes a new form of unsupervised learning, whose input is a set of unlabeled points that are assumed to be local maxima of an unknown value function v in an unknown subset of the vector space. Two functions are learned: (i) a set indicator c, which is a binary classifier, and (ii) a comparator function h that given two nearby samples, predicts which sample has the higher value of the unknown function v. Loss terms are used to ensure that all training samples x are a local maxima of v, according to h and satisfy c(x) = 1. Therefore, c and h provide training signals to each other: a point x in the vicinity of x satisfies c(x) = −1 or is deemed by h to be lower in value than x. We present an algorithm, show an example where it is more efficient to use local maxima as an indicator function than to employ conventional classification, and derive a suitable generalization bound. Our experiments show that the method is able to outperform one-class classification algorithms in the task of anomaly detection and also provide an additional signal that is extracted in a completely unsupervised way.

This paper describes a new form of one-class/set beloning learning, based on definition of 4 player game:
- Classifier player (c), which is a typical one-class classifier model
- Comparator player (h), which given two instances answers if first is "not smaller" (wrt. set belonging) than the other
- Classifier adversary player (Gc), which tries to produce hard to distinguish samples for (c)
- Comparator adversary player (Gh), which tries to produce hard to classify samples for (h)
This way authors end up with cooperative-competitive game, where c and h act cooperatively to solve the problem, while Gc and Gh constantly try to "beat" them.
Overall I find this paper to be interesting and worth presenting, however I strongly encourage authors to rethink the way story is presented so that it is more approachable by people who do not have much experience with viewing typical classification problems as games. In particular, one could completely avoid talking about "sets of local maxima" and just talk about the density estimation problem, with c being characteristic function (of belonging to the support) and h being comparator of the pdf.
Strong points:
- Novel, multi-agent in nature, approach to one-class classification
- Proposed method build a complex system, which can be used in much wider class of problems than just classification (due to joint optimisation of classifier and comparator)
- Extensive evaluation on 4 problems
- Nice ablation study showing that most of the benefits come from pure c/Gc game (on average 68.8% acc vs 65.2% of just c, and 69.8% of entire system) but that h/Gh players do indeed still improve (an extra 1%). It might be interesting to investigate what exactly changed in c due to existance of h in training. Are there any identifiable properties of the model that can now be analysed?
Weak points:
In general I believe that theoretical analysis is the weakest part of the paper, and while interesting - it is actually a minor point, and shows interesting properties, but not the ones that would guarantee anything in "practical setup". I would suggest "downplaying" this part of the paper, maybe moving most of it to the appendix.
To be more specific:
- Theorem 1 shows that representation can be more compact, however existance of compactness does not rely imply that this particular solution can ever be learned or that it is a good thing (number of parameters is not correlated with generalisation capabilities of the model).
- Lemma 1 seems a bit redundant for the story. While it is nice to be able to show generalisation bounds in general, this paper is not really introducing new class of models (since in the end c is going to be used for actual classification), but rather training regime, and generalisations bounds do not tell us anything about the emerging dynamical system. The fact that adding v does not constrain c too much seems quite obvious, and as a result I would suggest moving this section to appendix.
Instead, if possible, the actual tricky mathematical bit for methods like this would be, in reviewers opinion, any analysis of learning dynamics of the system like this. Multi-agent systems cannot be optimised with independent gradient descent in general (convergence guarantees are lost). Consequently many papers focus on methods that bring these properties back (e.g. Consensus Optimisation or Symplectic Gradient Ascent). It would be beneficial for the reader to spend some time discussing stability of the system proposed, even if only empirically and on small problems.
Other remarks:
- eq. (1) is missing \cdot
- it could be useful to include explicit parameters dependences in (1) and (2) so that one sees how losses really define asymmetric game between the players
- why do we need 4 players and not just 3, with Gc and Gh being a single player/neural network? can we consider this as another ablation?
- given small performance gaps in Table 1 can we get error estimates/confidence intervals there? Deep SVDD paper includes error estimates of the baseline methods
- since training is performed in mini batch (it does not have to be decomposible over samples) shouldn't equations be based on expectations rather than sums?
-


Unsupervised representation learning has succeeded with excellent results in many applications. It is an especially powerful tool to learn a good representation of environments with partial or noisy observations. In partially observable domains it is important for the representation to encode a belief state-a sufficient statistic of the observations seen so far. In this paper, we investigate whether it is possible to learn such a belief representation using modern neural architectures. Specifically, we focus on one-step frame prediction and two variants of contrastive predictive coding (CPC) as the objective functions to learn the representations. To evaluate these learned representations, we test how well they can predict various pieces of information about the underlying state of the environment, e.g., position of the agent in a 3D maze. We show that all three methods are able to learn belief representations of the environment-they encode not only the state information, but also its uncertainty, a crucial aspect of belief states. We also find that for CPC multi-step predictions and action-conditioning are critical for accurate belief representations in visually complex environments. The ability of neural representations to capture the belief information has the potential to spur new advances for learning and planning in partially observable domains, where leveraging uncertainty is essential for optimal decision making.

** Summary **
The authors evaluate three different representation learning algorithms for partially observable environments. In particular, they investigate how well the learned representation encodes the true belief distribution, including its uncertainty.
They propose an extension to a previous algorithm and evaluate all three algorithms on a range of tasks.
** Clarity **
The paper is well written and overall easy to follow.
** Quality **
The paper evaluates the described algorithms on a sufficiently large set of tasks. There is no theoretical analysis.
** Originality & Significance **
While the authors propose a novel extension to an existing algorithm, I believe the value of this work lies in the detailed empirical analysis.
** Missing Citations **
I believe two recent papers (this year's ICML) should be mentioned in the related work section as they propose two representation learning algorithms for POMDPs that, as far as I can tell, are not yet mentioned in the paper but quite relevant to the discussed topic. [1] Because it also uses PSRs and [2] because it explicitly learns a belief state. It would be interesting to see how [2] compares in terms of performance to FP and CPC(|Action).
[1] Hefny, A., Marinho, Z., Sun, W., Srinivasa, S. & Gordon, G.. (2018). Recurrent Predictive State Policy Networks. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:1949-1958
[2] Igl, M., Zintgraf, L., Le, T.A., Wood, F. & Whiteson, S.. (2018). Deep Variational Reinforcement Learning for POMDPs. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:2117-2126
** Question **
I have several questions where I'm not sure I understand the paper correctly:
1.) Why can FP only predict the mean? For example, one could use a PixelCNN as decoder, which would allow to learn an entire distribution, not just the mean over images.
2.) The problem that CPC and CPC|Action is unable to predict objects if they don't influence the future trajectory doesn't seem surprising to me because whether an image is a positive or negative example can usually be determined by the background, the object is not necessary to do so. In other words, this is a problem of how the negative samples are chosen: If they were constructed using a simulator that shows the same background but without the objects, the belief would need to start encoding the presence of objects. Is this correct or am I missing something?
3.) Am I correct in thinking that CPC(|Action) would not be applicable to properly estimate the belief distribution in the presence of noise, i.e. for example when estimation the exact location based on sensors with Gaussian noise?
** Overall **
* Pros:
- Extensive, interesting evaluation
- Novel CPC|Action algorithm
* Cons:
- No theoretical analysis/justification for claims
- There are several subtleties that I am not sure are sufficiently discussed in the paper (see my questions above)


A UNIVERSAL MUSIC TRANSLATION NETWORK
We present a method for translating music across musical instruments and styles. This method is based on unsupervised training of a multi-domain wavenet autoencoder, with a shared encoder and a domain-independent latent space that is trained end-to-end on waveforms. Employing a diverse training dataset and large net capacity, the single encoder allows us to translate also from musical domains that were not seen during training. We evaluate our method on a dataset collected from professional musicians, and achieve convincing translations. We also study the properties of the obtained translation and demonstrate translating even from a whistle, potentially enabling the creation of instrumental music by untrained humans.

The paper proposes a multi-domain music translation method. The model presents a Wavenet auto-encoder setting with a single (domain independent) encoder and multiple (domain specific) decoders.  From the model perspective, the paper builds up on several exciting ideas such as Wavenet and autoencoder based translation models that can perform the domain conversion without relying on parallel datasets. The two main modifications are the use of data augmentation, the use of multiple decoders (rather a single decoder conditioned on the output domain identity) and the use of a domain confusion loss to prevent the latent space to encode domain specific information. This last idea has been also used on prior work.
Up to my knowledge, this is the first autoencoder-based music translation method. While this problem is very similar to that of speaker conversion, modeling musical audio signal (with many instruments) is clearly more challenging.
Summarizing, I think that the contributions in terms of methods are limited, but the results are very interesting. The paper gives an affirmative answer to the question of whether existing models could be adapted to handle the case of music translation, which is of value. The paper would be stronger in my view, if stronger baselines would be included. This would show that the technical contributions are better than alternative methods. Please read bellow some further comments and questions.
The authors perform two ablation studies: eliminating data augmentation and the domain confusion network. In both cases, the model without this add on fails to train. However, it seems to me that different studies are important.
The paper seems to be missing baselines. The authors could compare their work with that of VQ-VAE. The authors claim that they could not make VQ-VAE work on this problem. The cited work by Dieleman et al provides some improvements to adapt VQ-VAE to be better suited to the music domain. Did you evaluate also autoregressive discrete autoencoders?
The proposed method uses an individual decoder per domain. This is unlike other conversion methods (such as the speech conversion studied in VQ-VAE). This modification is very costly and provides a very large capacity. Have you tried having a single decoder which is also conditioned on a one-hot vector indicating the domain? Is it reasonable to expect some transfer between domains or are they too different? Maybe this is the motivation behind using many decoders. It would be good to clarify.
I understand that the emphasis of this work is on music translation, however, the model doesn't have anything specific to music. In that regard, maybe a way to compare to VQ-VAE is to run the proposed method to the voice conversion of the VQ-VAE.
Have you tried producing samples using the decoder in an unconditional setting?
The authors claim that the learned representation is disentangled. Why is this the case? Normally a representation is said to be disentangled if different properties are represented in different (disjoint) coordinates. I might not be understanding what is meant here.
The loss used by the authors, encourages the latent representation to not have domain specific information. The authors should cite the work [A], which has very similar motivation. It would be interesting to report the classification accuracy of the classifier to see how much of the domain information is left in the latent codes. Is it reduced to chance?
In Section 3.1 the authors describe some modifications to nv-wavenet. I imagine that this is because it leads to better performance or faster training. It would be good to give some more information. Did you perform ablation studies for these?
In the human lineup experiment (Figure 2 b,c and d). While the listeners fail to select the correct source, many of the domains are never chosen. This could suggest that some translations are consistently poorer than others or the translations themselves are poor. This cannot be deduced from this experiment. Have you evaluated this?  Maybe it would be better to present pairs of audios with reconstruction and a translation.
While I consider the results quite good, I tend to agree with the posted public comment. It is very hard to claim that the model is effectively transferring styles. A perceptual test should include the question: is this piece on this given style? As the authors mentioned, it is clearly very difficult to evaluate generative models. But maybe the claims could be toned down.
[A] Louizos, Christos, et al. "The variational fair autoencoder." arXiv preprint arXiv:1511.00830 (2015).


Deep Convolutional Neural Networks (CNNs) have been repeatedly shown to perform well on image classification tasks, successfully recognizing a broad array of objects when given sufficient training data. Methods for object localization, however, are still in need of substantial improvement. In this paper, we offer a fundamentally different approach to the localization of recognized objects in images. Our method is predicated on the idea that a deep CNN capable of recognizing an object must implicitly contain knowledge about object location in its connection weights. We provide a simple method to interpret classifier weights in the context of individual classified images. This method involves the calculation of the derivative of network generated activation patterns, such as the activation of output class label units, with regard to each input pixel, performing a sensitivity analysis that identifies the pixels that, in a local sense, have the greatest influence on internal representations and object recognition. These derivatives can be efficiently computed using a single backward pass through the deep CNN classifier, producing a sensitivity map of the image. We demonstrate that a simple linear mapping can be learned from sensitivity maps to bounding box coordinates, localizing the recognized object. Our experimental results, using real-world data sets for which ground truth localization information is known, reveal competitive accuracy from our fast technique.

===================
SUMMARY
===================
The paper proposes a method to extend the output of a network trained for visual object recognition (i.e. image classification) with bounding box coordinates that can serve to localize the recognized object.
This process is referred to as "object localization", and it resembles to some extent the object detection task.
Object localization is achieved by analyzing the absolute values of the gradients in the last convolutional layer (referred to as "attention map" in the paper) with regard to the pixel locations of the input image.
Then, a linear model is trained in order to predict the bounding box coordinates that serve to localize the recognized object instance. This is different from traditional object detection methods, which learn how to predict bounding boxes directly from image pixels. The paper claims that by learning how to predict bounding boxes from sensitivity maps the amount of needed training data is reduced.
Experiments on the PASCAL VOC'07 and a subset of the ImageNet'12 dataset show the performance of the proposed method.
===================
REVIEW
===================
The content of the paper is clear, has a good flow. Moreover, the proposed method is sound and relatively easy to follow. The reported experiments show good performance of the proposed method. In addition, there seems to be hints that the proposed method has a computation speed that makes it suitable for on-the-fly computations.
My main concerns with the manuscript are the following:
The manuscript proposes to analyze the internal spatial activations/gradients of a network trained for object recognition with the goal of exploiting internally encoded cues that can serve for object localization. At the high level, as recognized in the manuscript, this is very similar to the work from Zhou et al.,CVPR'16 with some differences regarding how the attention maps are defined, and if additional components are added to the network.
In addition, the way in which the proposed object localization problem is formulated bears resemblance to work on weakly-supervised object detection where object detectors are trained by using only image-level annotations (no bounding box annotations are used).
Keeping these two groups of work in mind, I find the contribution of the proposed method limited.
Perhaps a explicit positioning with respect to this work is required.
Section 2.3 states that there two possibilities, i.e. max or average. to aggregate the sensitivity maps obtained for each of the RGB channels from images. However, in Section 3.2, it is stated that the max operation is used for the Pascal VOC'07 dataset while the average operation was used for ImageNet data. This inconsistency in the way these maps are aggregated makes it hard to observe if there is a difference or trend across these two operations. Is there a reason why different operations were applied to different datasets? I recommend applying both operations on both datasets and discuss observations from the results.
While the proposed method have been evaluated on two datasets, and compared with respect to standard classic baselines, a deeper evaluation analyzing the inner workings of the method is not conducted.
In this regard, in addition to evaluating the effect of the method to aggregate sensitivity maps accross color channels (previously mentioned), I suggest reporting results by changing the threshold used in intersection over union (IoU) measure used to verify matching (Section 3,1). In addition, I am curious about the performance of the method in the two following scenarios: i) when multiple instances of the recognized object are present in the image, and ii) when instances of other distractor classes are present in the image.
In several parts of the manuscript, e.g. Sec. 1, 3.4, and 4, claims regarding the speed of the proposed method are made. However, there is no evaluation focusing on this aspect that could be used to support this claims. In this regard, either a evaluation focusing on the computation speed of the proposed method should be conducted or claims regarding to computation speed should be toned down.

Finally, the first title of the manuscript, i.e. "Interpreting Deep Neural Networks", suggests that the manuscript will cover topics regarding model interpretation of DNNs. However, model interpretation as such is never touched wich makes this title somewhat misleading. In fact the main theme of the manuscript is about object localization/detection, hence the second part of its title: "Fast Object Localization via Sensitivity Analysis".
If this manuscript is to be accepted, the first part of the title should be removed.
I would appreciate if my main concerns are addressed in the rebuttal.


PROBGAN: TOWARDS PROBABILISTIC GAN WITH THEORETICAL GUARANTEES
Probabilistic modelling is a principled framework to perform model aggregation, which has been a primary mechanism to combat mode collapse in the context of Generative Adversarial Networks (GAN). In this paper, we propose a novel probabilistic framework for GANs, ProbGAN, which iteratively learns a distribution over generators with a carefully crafted prior. Learning is efficiently triggered by a tailored stochastic gradient Hamiltonian Monte Carlo with a novel gradient approximation to perform Bayesian inference. Our theoretical analysis further reveals that our treatment is the first probabilistic framework that yields an equilibrium where generator distributions are faithful to the data distribution. Empirical evidence on synthetic high-dimensional multi-modal data and image databases (CIFAR-10, STL-10, and ImageNet) demonstrates the superiority of our method over both start-of-the-art multi-generator GANs and other probabilistic treatment for GANs.

Summary
=========
The paper extends Bayesian GANs by altering the generator and discriminator parameter likelihood distributions and their respective priors.
The authors further propose an SGHMC algorithm to collect samples of the resulting posterior distributions on each parameter set and evaluate their approach on both a synthetic and the CIFAR-10 data set.
They claim superiority of their method, reporting a higher distance to mode centers of generated data points and better generator space coverage for the synthetic data set and better inception scores for the real world data for their method.
Review
=========
As an overall comment, I found the language poor, at times misleading.
The authors should have their manuscript proof-read for grammar and vocabulary.
Examples:
- amazing superiority (page 1, 3rd paragraph)
- Accutally... (page 1, end of 3rd paragraph)
- the total mixture of generated data distribution (page 3, mid of 3.1)
- Similarity we define (page 3, end of 3.1)
- etc.
Over the whole manuscript, determiners are missing.
The authors start out with a general introduction to GANs and Bayesian GANs in particular,
arguing that it is an open research question whether the generator converges to the true data generating distribution in Bayesian GANs.
I do not agree here. The Bayesian GAN defines a posterior distribution for the generator that
is proportional to the likelihood that the discriminator assigns to generated samples.
The better the generator, the higher the likelihood that the discriminator assign to these samples.
In the case of a perfect generator, here the discriminator is equally unable to distinguish real and generated samples and consequently degenerates to a constant function.
Using the same symmetry argument as the authors, one can show that this is the case for Bayesian GANs.
While defining the likelihood functions, the iterator variable t is used without introduction.
Further, I a confused by their argument of incompatibility.
First, they derive a Gibbs style update scheme based on single samples for generator and discriminator parameters using
posteriors in which the noise has been explicitly marginalized out by utilizing a Monte Carlo estimate.
Second, the used posteriors are conditional distributions with non-identical conditioning sets.
I doubt that the argument still holds under this setting.
With respect to the remaining difference between the proposed approach and Bayesian GAN,
I'd like the authors elaborate where exactly the difference between expectation of objective value
and objective value of expectation is.
Since the original GAN objectives used for crafting the likelihoods are deterministic functions,
randomness is introduced by the distributions over the generator and discriminator parameters.
I would have guessed that expectations propagate into the objective functions.
It is, however, interesting to analyze the proposed inference algorithm, especially the introduced posterior distributions.
For the discriminator, this correspond simply to the likelihood function.
For the generator, the likelihood is combined with some prior for which no closed form solution exists.
In fact, this prior changes between iterations of the inference algorithm.
The resulting gradient of the posterior decomposes into the gradient of the current objective and the sum over all previous gradients.
While this is not a prior in the Bayesian sense (i.e. in the sense of an actual prior belief), it would be interesting to have a closer look at the effect this has on the sampling method.
My educated guess is, that this conceptually adds up to the momentum term in SGHMC and thus slows down the exploration of the parameter space and results in better coverage.
The experiments are inspired by the ones done in the original Bayesian GAN publication.
I liked the developed method to measure coverage of the generator space although I find the
term of hit error misleading.
Given that the probabilistic methods all achieve a hit rate of 1, a lower hit error actually points to worse coverage.
I was surprised to see that hit error and coverage are only not consistently negatively correlated.
Adding statistics over several runs of the models (e.g. 10) would strengthen the claim of superior performance.