Adversarial examples are considered a serious issue for safety critical applications of AI, such as finance, autonomous vehicle control and medicinal applications. Though significant work has resulted in increased robustness of systems to these attacks, systems are still vulnerable to well-crafted attacks. To address this problem several adversarial attack detection methods have been proposed. However, system can still be vulnerable to adversarial samples that are designed to specifically evade these detection methods. One recent detection scheme that has shown good performance is based on uncertainty estimates derived from Monte-Carlo dropout ensembles. Prior Networks, a new method of estimating predictive uncertainty, have been shown to outperform Monte-Carlo dropout on a range of tasks. One of the advantages of this approach is that the behaviour of a Prior Network can be explicitly tuned to, for example, predict high uncertainty in regions where there are no training data samples. In this work Prior Networks are applied to adversarial attack detection using measures of uncertainty in a similar fashion to Monte-Carlo Dropout. Detection based on measures of uncertainty derived from DNNs and Monte-Carlo dropout ensembles are used as a baseline. Prior Networks are shown to significantly out-perform these baseline approaches over a range of adversarial attacks in both detection of whitebox and blackbox configurations. Even when the adversarial attacks are constructed with full knowledge of the detection mechanism, it is shown to be highly challenging to successfully generate an adversarial sample.

Summary:
The authors propose a new method to detect adversarial attacks (examples). This approach relies on prior networks to estimate uncertainty in model predictions. Prior-networks are then trained to identify out-of-distribution inputs, and thereby used to detect adversarial examples. The authors evaluate their methodology on CIFAR-10 in different white-box and blackbox settings.
This work addresses an important question - detecting adversarial examples. Since it may not always be possible to build models that are completely robust in their predictions, detecting adversarial examples and/or identifying points where the model is uncertain is important. However, I am not convinced by the specific methodology as well as the proposed evaluation.
Detailed comments:
- This work is largely based on the recent work by Malinin and Gales, 2018, where prior networks are developed as a scheme to identify out-of-distribution inputs. As a result, the authors rely fundamentally on the assumption that adversarial examples lie off-the data manifold. There has been no convincing evidence for this hypothesis in the literature thus far. Adversarial examples are also likely to be on the data manifold, but form a small enough set that it doesn’t affect standard generalization. But because of the high-dimensional input space, a member of this small set is still close to every “natural” data point.
- I do not find the specific choice of attacks the authors consider convincing. (These being the attacks studied in Smith & Gal, 2018 does not seem to be a sufficient explanation). Specifically, the authors should evaluate on stronger Linf attacks such as PGD [Madry et al., 2017]. Further, it seems that the authors consider Linf eps around 80. These values seem extremely large given that eps=32 (possibly even > 16) causes perceptible changes in the images. Did the authors look at the adversarial examples created for these large eps values?
- The authors should include evaluation on a robust DNN (for example PGD trained VGG network) in the comparison. I believe that the joint success rate for this robust model will already be comparable to the proposed approach.
- I am not convinced by the attack that the authors provide for the setting where the detection scheme is known. This attack seems similar to the approach studied in Carlini and Wagner, 2017 (Perfect-Knowledge Attack Evaluation) which was insufficient to break the randomization defense. Why did the authors not try something along the lines of the attack in Carlini and Wagner, 2017 (Looking deeper) that actually broke the aforementioned defense? Specifically, trying to find adversarial examples that have low uncertainty as predicted by the prior networks. The uncertainty loss -- minimizing KL between p_in(\pi|x_adv) and p(\pi|x_adv, \theta) -- could be added to cross entropy loss.
- In Section 4.2, how do the authors generate black box attacks? If they are white box attacks on models trained with a different seed (as in Section 4.1) the results in 4.2 are surprising. Carlini and Wagner, 2017 found white-box attacks for randomization schemes transferrable and as per my understanding, this should be reflected in Fig 3, at least for prior work.
- I am confused by the authors comment - “Figure 3c shows that in almost 100% of cases the attack yields the target class.” The joint success rate being lower than the success rate should convey that adversarial examples couldn’t be found in many of these cases. What was the value of the epsilon that was used in these plots?
Quality, Novelty and Significance:
The paper is written well, but clarity about the evaluation procedures is lacking in the main manuscript. I am also not convinced by the rigor of the evaluation of their detection methodology. Specifically: (1) they do not consider state-of-the-art attack models such as PGD and (2) the scheme they propose for a perfect knowledge attack seems insufficient. While the paper asks an important question, I do not find the results sufficiently novel or convincing. More broadly, I find the idea of using a secondary network to detect adversarial examples somewhat tenuous as it should be fairly easy for an adversary to break this other network as well.


Programming languages are emerging as a challenging and interesting domain for machine learning. A core task, which has received significant attention in recent years, is building generative models of source code. However, to our knowledge, previous generative models have always been framed in terms of generating static snapshots of code. In this work, we instead treat source code as a dynamic object and tackle the problem of modeling the edits that software developers make to source code files. This requires extracting intent from previous edits and leveraging it to generate subsequent edits. We develop several neural networks and use synthetic data to test their ability to learn challenging edit patterns that require strong generalization. We then collect and train our models on a large-scale dataset consisting of millions of fine-grained edits from thousands of Python developers. From the modeling perspective, our main conclusion is that a new composition of attentional and pointer network components provides the best overall performance and scalability. From the application perspective, our results provide preliminary evidence of the feasibility of developing tools that learn to predict future edits.

Summary: The authors study building models for edits in source code. The application is obvious: a system to accurately predict what the next edit should be would be very valuable for developers. Here, edits are modeled by two types of sequences: one that tracks the state of all edits at each time step (and is thus very long), and one that contains the initial step and a changelist that contains the minimal information required to derive the state at any time. The authors train models on top of both of these representations, with the idea being to match the performance of the explicit (heavy) model with the implicit model. This is shown to be challenging, but a clever model is introduced that achieves this, and is thus the best of both worlds. There are synthetic and real-world code (text) edit experiments.
Strengths: The problem is well-posed and well-motivated. There's a nice application of powerful existing models, combined and tailored to the current work. The writing is generally quite clear. The number of experiments is quite solid.
Weaknesses: The main flaw is that nothing here is really specifically for souce code; the authors are really just modeling edits in text sequences. There's not an obvious way to integrate the kinds of constraints that source code typically satisfies either. There's some confusion (for me) about the implicit/explicit representations. More questions below.
Verdict: This is a pretty solid paper. It doesn't quite match up to its title, but it sets out a clearly defined problem, achieves its aims, and introduces some nice tricks. Although it doesn't produce anything genuinely groundbreaking, it seems like a nice step forward.
Comments and Questions:
- The problem is written in the context of source code, but it's really setup just for text sequences, which is a broader problem. Is there a way the authors take can advantage of the structural requirements for source code? I don't see an obvious way, but I'm curious what the authors think.
- What's the benefit of using the implicit representation for the positions? The explicit/implicit  position forms are basically just using the permutation or the inverse permutation form, which are equivalent. I don't see directly what's saved here, the alphabet size and the number of integers to store is the same.
- Similar question. The implicit likelihood is s^0, e^(1),...,e^(t-1), with the e^(i)'s being based on the implicit representations of the positions. Seems like you could do this with the *explicit* positions just fine, they carry enough information to derive s^(i) from s^(i-1). That is, the explicit/implicit problems are not really related to the explicit/implicit position representations.
- Just wanted to point out that this type of approach to sequences and edits has been studied pretty often in the information/coding theory communities, especially in the area of synchronization. There, the idea is to create the minimal "changelist" of insertions/deletions from two versions of a file. This could come in handy when building the datasets. See, for example, Sala et al "Synchronizing files from a large number of insertions and deletions".
- The problem statement should be stated a bit more rigorously. We'd like to say that the initial state is drawn from some distribution and that the state at each time forms a stochastic process with some transition law. As it stands the problem isn't well-defined, since with no probability distribution, there's nothing to predict and no likelihood.
- The "analogical decoder" idea is really nice.
- For the synthetic dataset, why are you selecting a random initial string, rather than using some existing generative text or source code model, which would get you synthetic data that more closely resembles code?
- I really liked the idea of using an oracle that gives the position as upper bound. Would it make sense to also have the opposite oracle that gives the edit symbol, but doesn't tell the location? I'm really curious which is the "harder" task, predicting the next symbol or the next location. In the information-theory setting, these two are actually equally hard, but the real-world setting might be pretty different. It would also be interesting to train models on top of the POMP. That would produce genuine upper bounds to the model performances.
- The explicit baseline model performs very well on all the edit types in Table 1. Are there cases where even this explicit case works poorly? Is the improved implicit model *always* upper bounded by the explicit model (to me it seems like the answer should always be yes, but it would be interesting to check it out for cases where explicit is not very high accuracy).


ON RANDOM DEEP WEIGHT-TIED AUTOENCODERS: EXACT ASYMPTOTIC ANALYSIS, PHASE TRANSI- TIONS, AND IMPLICATIONS TO TRAINING
We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs "approximate inference" as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.

This paper studies auto-encoders under several assumptions: (a) the auto-encoder's layers are fully connected, with random weights, (b) the auto-encoder is weight-tied, (c) the dimensions of the layers go to infinity with fixed ratios. The main contribution of the paper is to point out that this model of random autoencoder can be elegantly and rigorously analysed with one-dimensional equations. The idea is original and will probably lead to new directions of research. Already the first applications that the paper suggests are exciting.
The paper does a good job in justifying assumptions (a), (b) and (c) in the introduction. It is convincing in the fact that this point of view may bring practical insights on training initialization for real-world autoencoders. Thus my opinion is that this paper brings original and significant ideas in the field.
One flaw of this paper is that the writing might be clearer. For instance when presenting the technical theorem (Theorem 1), it would be useful to have an intuitive explanation for the theorem and the state-evolution-like equations. However, I believe that there are some easy fixes that would greatly improve the clarity of the exposition. Here is a list of suggestions:
- In Section 2.1, a large number of notations are introduced. It would help a lot if the authors made a graphical representation of these. For instance, a diagram where every linearity / non-linearity is a box, and the different variables $x_l$, $\hat{x}_l$ appear would help a lot.
- Section 2.2 is rather technical. The authors could try to give some more intuition of what's happening. For instance, they could spend more time after the theorem explaining what $\tau_l, \gamma_l$ and $\rho_l$ mean. They could also introduce the notation S_sig and S_var early and this section (and not in Section 3), because it helps interpreting the parameters. It would also help if they could write a heuristic derivation of the state-evolution-like equations. From the paper, the only way the reader can understand the intuition behind those complicated equations is to look at the proof of Theorem 1 (which is rather technical).
- In Section 3.1, I did not understand the difference between interpretations 1 and 2. Could the authors clarify?
- In Section 3.4, I did not understand the sentence: "In particular, near the phase transition of \gamma, S_sig/S_var = \Omega(\beta^{1.5}". If one uses the \Omega notation, it means that some parameter is converging to something. What is the parameter? As a consequence, I did not understand this paragraph.
- In Section 3.5, the authors should make clear from the beginning why they are running those specific simulations. What hypothesis are they trying to check? I finally concluded that they are running simulations to check if the hypothesis they make in the first paragraph are true. They also want to compare with some other criteria in the literature, named EOC, that also gives insights about the trainability of the network. However, they could explicitly say in the beginning of the second paragraph that this is the goal.
- In a similar spirit, the authors should end Section 3.5 with a clear conclusion on whether or not the framework enables us to predict the trainability of the autoencoder.
Minor edits / remarks:
- Typo: last but one paragraph of the introduction: "whose analysis is typically more straighforwards" -> "straightforward".
- At the end of Section 3.2: what can be proved about the behavior of \gamma / \sqrt{\rho}? It is obviously a central quantity and the authors do not say what happens in the phases where \gamma and \rho go to infinity for instance. Is it because it is hard to analyse?


The application of deep recurrent networks to audio transcription has led to impressive gains in automatic speech recognition (ASR) systems. Many have demonstrated that small adversarial perturbations can fool deep neural networks into incorrectly predicting a specified target with high confidence. Current work on fooling ASR systems have focused on white-box attacks, in which the model architecture and parameters are known. In this paper, we adopt a black-box approach to adversarial generation, combining the approaches of both genetic algorithms and gradient estimation to solve the task. We achieve a 89.25% targeted attack similarity after 3000 generations while maintaining 94.6% audio file similarity.

PAPER SUMMARY:
This paper introduces a biologically motivated black-box attack algorithm.
The target model in this case is DNN applied to the ASR context (automatic speech recognition system).
NOVELTY & SIGNIFICANCE:
The proposed approach extends the previous genetic approach of (Alzantot et al., 2018) to attack a more complicated ASR system (that handles phrases and sentences). The new contribution here is an add-on momentum mutation component on top of the existing genetic programming architecture of (Alzantot et al., 2018) as illustrated in Figure 3.
This however appears very incremental seeing that integrating the mutation component into existing system is straight-forward and that mutation is not even a new concept -- it has always been a vital component in genetic programming paradigm.
It is also unclear how this mutation component improves over the existing work (more on this in the sections below).
Another issue is this work seems to ignore the recent literature on adversarial black-box attacks to DNN model. To list a few:
Chen, P.-Y.; Zhang, H.; Sharma, Y.; Yi, J.; and Hsieh, C.-J. 2017b.
ZOO: Zeroth-order optimization-based  black-box attacks to deepneural networks without training substitute models.
In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (15-26) ACM
Cheng,  M.;  Le,  T.;  Chen,  P.-Y.;  Yi,  J.;  Zhang,  H.;  and  Hsieh,C.-J.2018.
Query-efficient hard-label black-box attack:  An optimization-based approach. arXiv preprint arXiv:1807.04457
While these works have not been used to attacking ASR system, they should be directly applicable to such system since after all, they are black-box attacks. I think the proposed method needs to be compared with these works.
TECHNICAL SOUNDNESS:
I find it surprising that even though the proposed method is claimed to be a black-box attack but in the end, it actually exploits the fact that the target model uses CTC decoder. This pertains specifically to the target model's internal architecture and a black-box attack is not supposed to know this.
CLARITY:
The paper is clearly written.
EMPIRICAL RESULTS:
I do not understand this statement:
"That 35% of random attacks were successful in this respect highlights the fact that black box
adversarial attacks are definitely possible and highly effective at the same time"
Why is 35% successful attack rate a positive result? The result tends to suggest that this is an attack with low success rate.
The 2nd paragraph in 3.2 seems to give a vague explanation: "the vast majority of failure cases are only a few edit distances away from the target.
This suggests that running the algorithm for a few more iterations could produce a higher success rate, although at the cost of correlation similarity".
Given the above statement, I do not see why the authors didn't actually "run the algorithm for a few more iterations" to verify it ...
I am also curious why is the success rate of the proposed method is significantly lower than that of the existing system -- I assume "single word black box" is the work of (Alzantot et al., 2018).
I find the empirical evaluation somewhat sloppy: why are the tested method not compared on the same benchmark? How do we interpret the results then?
REVIEW SUMMARY:
The paper misses the recent literature on black-box attack. The authors need to compare with those to demonstrate the efficiency of their proposed work. I also find the contribution of this paper too incremental & its empirical evaluation appears somewhat sloppy and not convincing (see my specific comments above).


We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

This paper shows that gradient descent mostly happens in a tiny subspace which is spanned by the top eigenvectors of the Hessian. Empirical results are shown to support the claim. This finding is interesting and provides us some insights to design more efficient optimization algorithms. Overall, this paper is interesting and easy to follow.
The experiments in section 2 do a decent job supporting the claim that gradient descent happens in a tiny subspace and the subspace is mostly preserved over long periods of training. However, I would like to add a couple more points to the discussion:
- It's not surprising that the magnitude of gradient is larger in the high curvature directions, which means that the learning would always first happen in top subspace if the learning rate is small enough. It would be interesting to tune the parameter of learning rate to see if the phenomena would occur across different learning rate (especially large learning rate).
- The argument of gradient descent happening in a tiny space is quite obvious if the Hessian has only a few large eigenvalues. Therefore, it would be interesting to discuss the spectrum of the Hessian a little bit.
- Contrary to plain gradient descent, natural gradient is able to learn low curvature directions (small eigenvalues). It would be interesting to show some experiments with natural gradient methods.
Following section 2, the authors give a toy model to further backup their claims. However, I find the example is too restrictive and may not explain why the subspace would be preserved over the training. If I understand right, the loss function of the toy model is convex and Hessian is a constant over time. For this kind of toy model, the Hessian (or equivalently the Fisher matrix) only depends on the input distribution, so it's easy to see that the Hessian would be low-rank and preserved throughout the training. However, neural networks are highly non-convex, so it's unclear to me whether the implications of the toy model would generalize. I encourage the authors to analyze more complicated models.
To summarize, I think this paper is interesting and well-written. However, it lacks convincing explanation why the subspace would preserve over the training (to me, it's more interesting than the point that gradient descent happens in tiny subspace). Anyway, it is not completely reasonable to expect all such possible discussions to take place at once.


Latent variable models have been widely applied for the analysis and visualization of large datasets. In the case of sequential data, closed-form inference is possible when the transition and observation functions are linear. However, approximate inference techniques are usually necessary when dealing with nonlinear evolution and observations. Here, we propose a novel variational inference framework for the explicit modeling of time series, Variational Inference for Nonlinear Dynamics (VIND), that is able to uncover nonlinear observation and latent dynamics from sequential data. The framework includes a structured approximate posterior, and an algorithm that relies on the fixed-point iteration method to find the best estimate for latent trajectories. We apply the method to several datasets and show that it is able to accurately infer the underlying dynamics of these systems, in some cases substantially outperforming state-of-the-art methods.

This paper discusses a algorithm for variational inference of a non-linear dynamical models. In this paper model assumption is to use single stage Markov model in latent space with every latent variable Z_t to be defined Gaussian distributed with mean depends on Z_(t-1) and time invariant variance matrix lambda. The non linearity in transition is encoded in mean of Guassian distribution. For modeling the likelihood and observation model, the Poisson or Normal distribution are used with X_t being sampled from another Gaussian or Poisson distribution with the non-linearty being encoded in the parameters of distribution with variable Z_t.  This way of modeling resembles so of many linear dynamical model with the difference of transition and observation distribution have nonlinearity term encoded in them.
The contribution of this paper can be summarized over following points:
- The authors proposed the nonlinear transition and observation model and introduced a tractable inference model using Laplace approximation in which for every given set of model parameter solves for parameters of Laplace approximation of posteriori and then model parameters get updated until converges
-the second point is to show how this model is successful to capture the non-linearity of the data while other linear models do not have that capabilities
Novelty and Quality:
The main contribution of this paper is summarized above. The paper do not contain any significant theorem or mathematical claims, except derivation steps for finding Laplace approximation of the posteriori. The main challenge here is to address effectiveness of this model in comparison to other non-linear dynamical system that we can name papers as early as Ghahramani, Zoubin, and Sam T. Roweis. "Learning nonlinear dynamical systems using an EM algorithm." Advances in neural information processing systems. 1999.
or more recent RNN paper LSTM based papers. I think authors need to distinguish what this paper can give to community beside approximate posteriori of latent variables that other competing models are not capable of. If the aim is to have that posteriori, the authors should show what type of interpretation they have drawn from that in experiments.
There are lots of literature exist on speech, language models and visual prediction which can be used as reference as well.
Clarity:
The paper is well written and some previous relevant methods have been reviewed . There are a few issues that are listed below:
1- as mentioned in Quality sections authors should be more clear about what is distinguished in this paper that other non-linear dynamical systems
2- they used short form RM for Recognition model or FPI for fixed point iteration that need need to be defined before being used
significance and experiments:
The experiments are extensive and authors have compared their algorithm with some other linear dynamical systems (LDS) competing algorithms and showed improvement in many of the cases for trajectory reconstruction.
A few points can be addressed better, it can be seen for many of experiments exhaustive search is used for finding dimension of latent variable. This issue is addressed in Kalantari, Rahi, Joydeep Ghosh, and Mingyuan Zhou. "Nonparametric Bayesian sparse graph linear dynamical systems." arXiv preprint arXiv:1802.07434 (2018). That paper can use non-parametric approaches to find best latent dimension, although the paper applied the technique on linear system, same technique could be adopted to non-linear models. Also that model is capable of finding multiple linear system that model the non linearity by switching between diffrent linear system, for switching linear system, this paper can be named as well: Linderman, Scott, et al. "Bayesian learning and inference in recurrent switching linear dynamical systems." Artificial Intelligence and Statistics. 2017.
It is shown that the model can reconstruct the spikes very well while linear model do not have that power (which is expected), but it is interesting to see how other non-linear models would compare to this model under those certain conditions
It is desired and interesting to see how the model behave one step ahead and K-step ahead prediction. Please address why it cannot be done if there is difficulties in that.


Reinforcement learning algorithms rely on carefully engineered rewards from the environment that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is difficult and not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is such intrinsic reward function which uses prediction error as a reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance as well as a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many games. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the predictionbased rewards in stochastic setups. Game-play videos and code are at https: //doubleblindsupplementary.github.io/large-curiosity/.

The authors consider the setting of a RL agent that exclusively receives intrinsic reward during training that is intended to model curiosity; technically, ‘curiosity’ is quantified by the ability of the agent to predict its own forward dynamics [Pathak, et al., ICML17]. This study primarily centers around an initially somewhat surprising result that non-trivial policies can be learned for many ’simpler’ video games (e.g., Atari, Super Mario, Pong) using just curiosity as reward. While this is primarily an empirical study, one aspect considered was the observation representation (raw pixels, random features, VAE, and inverse dynamics features [Pathak, et al., ICML17]). In examining reward curves (generally extrinsic during testing), ‘curiosity-based’ reward generally works with the representation effectiveness varying across different testbeds. They also conduct more in-depth experiments on specific testbeds to study the dynamics (e.g., Super Mario, Juggling, Ant Robot, Multi-agent Pong) — perhaps most interestingly showing representation-based transfer of different embeddings across levels in Super Mario. Finally, they consider the Unity maze testbed, combining intrinsic rewards with the end-state goal reward to generate a more dense reward space.
From a high level perspective, this is an interesting result that ostensibly will lead to a fair amount of discussion within the RL community (and already has based on earlier versions of this work). However, it isn’t entirely clear if the primary contribution is showing that ‘curiosity reward’ is a potentially promising approach or if game environments aren’t particularly good testbeds for practical RL algorithms — given the lack of significant results on more realistic domains, my intuition leans toward the later (the ant robot is interesting, but one can come up with ‘simulator artifact’ based explanations). And honestly, I think the paper reads as if leaning toward the same conclusion. Regardless, given the prevalence of these types of testbed environments, either is a useful discussion to have. Maybe the end result could minimally be a new baseline that can help quantify the ‘difficulty’ of a particular environment.
From the perspective of a purely technical contribution, there are fewer exciting results. The basic method is taken from [Parthak, et al., ICML17] (modulo some empirical choices such as using PPO). The comparison of different observation representations doesn’t include any analytical component, the empirical component is primarily inconclusive, and the position statements are fairly non-controversial (and not really conclusively supported). The testbeds all existed previously and this is mostly the effort of pulling then together. Even the ‘focused experiments’ can be explained with the intuitive narrative that in the state/action space, there is always more uncertainty the farther one goes from the starting point and this is more of a result of massive computation being applied primarily to problems that are designed to provide some level of novelly (the Roboschool examples are a bit more interesting, but also less conclusive). Finally, Figure 5 is interesting in showing that ‘curiosity + extrinsic’ improves over extrinsic rewards — although this isn’t particularly surprising for maze navigation that has such sparse rewards and can be viewed as something like ‘active exploration’. With respect to this specific setting, the authors may want to consider [Mirowski, et al., Learning to Navigate in Complex Environments, ICLR17] with respect to auxiliary loss + RL extrinsic rewards to improve performance (in this case, also in maze environments).
In just considering the empirical results, they clearly entail a fair amount of effort and just a dump of the code and experiments on the community will likely lead to new findings (even if they are that game simulators are weaker testbeds than previously thought). It is easy to ask for additional experiments (i.e., other mechanisms of uncertainty such as the count-based discussed in related work, other settings in 2.2) — but the quality seems high enough that I basically trust the settings and findings. Beyond the core findings, the other settings are less convincingly supported by seem more like work in progress and this paper is really just a scaling-up of [Pathak, et al., ICML17] without generating any strong results regarding questions around representation, what to do about stochasticity (although the discussion regarding something like ‘curiosity honeypots’ is interesting). Thus, it reads like one interesting finding around curiosity-driven RL working in games plus a bunch of preliminary findings trying to grasp at some explanations and potential future directions.
Evaluating the paper along the requested dimensions:
= Quality: The paper is well-written with a large set of experiments, making the case that exclusively using curiosity-based reward is very promising for the widely-used game RL testbeds. Modulo a few pointers, the work is well-contextualized and makes reasonable assumptions in conducting its experiments. The submitted code and videos result in a high-quality presentation and trustworthiness of the results. (7/10)
= Clarity: The paper is very clearly written. (7/10)
= Originality: The algorithmic approach is a combination of [Parthak, et al., ICML17] and [Schulman, et al. 2017] (with some experiments using [Kingma & Welling, 2013]). All of the testbeds have been used previously. Other than completely relying on curiously-based reward exclusively, there is little here. In considering combining with extrinsic rewards, I would also consider [Mirowski, et al., ICLR17], which is actually more involved in this regard. (4/10)
= Significance: Primarily, this ‘finishes’ [Parthak, et al., ICML17] to its logical conclusion for game-based environments and should spur interesting conversations and further research. In terms of actual technical contributions, I believe much less significant. (5/10)
=== Pros ===
+ demonstrates that curiosity-based reward works in simpler game environments
+ (implicitly) calls into question the value of these testbed environments
+ well written, with a large set of experiments and some interesting observations/discussions
=== Cons ===
- little methodological innovation or analytical explanations
- offers minimal (but some) evidence that curiosity-based reward works in more realistic settings
- doesn’t answer the one question regarding observation representation that it set out to evaluate
- the more interesting problem, RL + auxiliary loss isn’t evaluated in detail
- presumably, the sample complexity is ridiculous
Overall, I am ambivalent. I think that more casual ML/RL researchers will find these results controversial and surprising while more experienced researchers will see curiosity-driven learning to be explainable primarily by the intuition of the “The fact that the curiosity reward is often sufficient” paragraph of page 6, demanding more complex environments before accepting that this form of curiosity is particularly useful. The ostensible goal of learning more about observation representations is mostly preliminary — and this direction holds promise of for a stronger set of findings. Dealing with highly stochastic environments seems a potential fatal flaw of the assumptions of this method. However, as I said previously, this is probably a discussion worth having given the popularity and visibility of game-based testbeds — so, coupled with the overall quality of the paper, I lean toward a weak accept.


In this paper, we explore some basic questions on the complexity of training Neural networks with ReLU activation function. We show that it is NP-hard to train a two-hidden layer feedforward ReLU neural network. If dimension d of the data is fixed then we show that there exists a polynomial time algorithm for the same training problem. We also show that if sufficient over-parameterization is provided in the first hidden layer of ReLU neural network then there is a polynomial time algorithm which finds weights such that output of the over-parameterized ReLU neural network matches with the output of the given data.

The main result of this work is to prove that a two-layer neural network with 2 hidden neurons in the first layer and 1 hidden neuron in the second layer is NP-hard to train when all activation functions are ReLU. Similar results with the hard-thresholding activation function were proved by Blum & Rivest in 1988. The main proof consists of a reduction from the 2-hyperplane separability problem which was known to be equivalent to NNs with hard-thresholding activation functions.
Quality: Moderate. Good solid formal results but nothing really surprising.
Clarity: The manuscript is mostly well-written, and the authors gave proper credit to prior works. The only minor issue is perhaps the abuse of the variable w in introduction and the same variable w (with an entirely different meaning) in the rest of the paper. It would make more sense to change the w's in introduction to d's.
Originality: This work is largely inspired by Blum & Rivest's work, and builds heavily on some previous work including Megiddo and Edelsbrunner et al. While there is certainly some novelty in extending prior work to the ReLU activation function, it is perhaps fair to say the originality is moderate.
Significance: While the technical construction seems plausible and correct, the real impact of the obtained results is perhaps rather limited. This is one of those papers that it is certainly nice to have all details worked out but none of the obtained results is really surprising or unexpected. While I do agree there is value in formally documenting the authors' results, this conference is perhaps not the right venue.
Other comments: The discussion on page 2 (related literature) seems odd. Wouldn't the results of Livni et al and Dasgupta et al already imply the NP-hardness of fully connected ReLU networks, in a way similar to how one obtains Corollary 3.2? If this is correct, then the contribution of this work is basically a refined complexity analysis where the ReLU network is shrunken to 2 layers with 3 nuerons?
I really wish the authors had tried to make their result more general, which in my opinion would make this paper more interesting and novel: can you extend the proof to a family of activation functions? It is certainly daunting to write separate papers to prove such a complexity result for every activation function... The authors also conveniently made the realizability assumption. What about the more interesting non-realizable case?
The construction to prove Theorem 3.4 bears some similarity to a related result in Zhang et al. The hard sorting part appears to be different.


DISCRIMINATOR-ACTOR-CRITIC: ADDRESSING SAMPLE INEFFICIENCY AND REWARD BIAS IN ADVERSARIAL IMITATION LEARNING
Algorithms for imitation learning based on adversarial optimization, such as generative adversarial imitation learning (GAIL) and adversarial inverse reinforcement learning (AIRL), can effectively mimic demonstrated behaviours by employing both reward and reinforcement learning (RL). However, applications of such algorithms are challenged by the inherent instability and poor sample efficiency of on-policy RL. In particular, the inadequate handling of absorbing states in canonical implementations of RL environments causes an implicit bias in reward functions used by these algorithms. While these biases might work well for some environments, they lead to sub-optimal behaviors in others. Moreover, despite the ability of these algorithms to learn from a few demonstrations, they require a prohibitively large number of the environment interactions for many real-world applications. To address these issues, we first propose to extend the environment MDP with absorbing states which leads to task-independent, and more importantly, unbiased rewards. Secondly, we introduce an off-policy learning algorithm, which we refer to as Discriminator-Actor-Critic. We demonstrate the effectiveness of proper handling of absorbing states, while empirically improving the sample efficiency by an average factor of 10. Our implementation is available online 1 .

The paper suggests to use TD3 to compute an off-policy update instead of the TRPO/PPO updates in GAIL/AIRL in order to increase sample efficiency.
The paper further discusses the problem of implicit step penalties and survival bias caused by absorbing states, when using the upper-bounded/lower-bounded reward functions log(D) and -(1-log(D)) respectively. To tackle these problem, the paper proposes to explicit add a unique absorbing state at the end of each trajectory, such that its rewards can be learned as well.
Pro:
The paper is well written and clearly presented.
Using a more sample efficient RL method for the policy update is sensible and turned out effective in the experiments.
Properly handling simulator resets in MDPs is a well known problem in reinforcement learning that I think is insufficiently discussed in the context of IRL.
Cons:
The contributions seem rather small.
a) Replacing the policy update is trivial, since the rl methods are used as black-box modules for the discussed AIL methods.
b) Using importance weighting to reuse old trajectories for the discriminator update hardly counts as a contribution either--especially when the importance weights are simply omitted in practice. I also think that the reported problems due to the high variance have not been sufficiently investigated. There should be a better solution than just pretending that the replay buffer corresponds to roll-outs of the current policy. Would it maybe help to use self-normalized importance weights? The paper does also not analyze how such assumption/approximation affects the theoretical guarantees.
c) The problem with absorbing states is in my opinion the most interesting contribution of the paper. However, the discussion is rather shallow and I do not think that the illustrative example is very convincing. Section 4.1.1. argues that for the given policy roll-out, the discriminator reward puts more reward on the policy trajectory than the expert trajectory. However, it is neither surprising nor problematic that the discriminator reward does not produce the desired behavior during learning. By assigning more cumulative reward for s2_a1->s1 than for s2_a2->g, the policy would (after a few more updates) choose the latter action much less frequently than with probability 0.5 and the corresponding reward would grow towards infinity until at some point Q(s2,a2) > Q(s2,a1)--when the policy would match the expert exactly. The illustrative example also uses more policy-labeled transitions than agent-labeled ones for learning the classifier, which may also be problematic. The paper further argues that a strictly positive reward function always rewards a policy for avoiding absorbing states, which I think is not true in general. A strictly positive reward function can still produce arbitrary large reward for any action that reaches an absorbing state. Hence, the immediate reward for choosing such action can be made larger than the discounted future reward when not ending the episode (for any gamma < 1). Even for state-only reward functions the problem does not persist when reseting the environment after reaching the absorbing state such that the training trajectories contain states that are only reached if the simulator gets reset. Hence, I am not convinced that adding a special absorbing state to the trajectory is necessary if the simulation reset is correctly implemented. This may be different for resets due to time limits that can not be predicted by the last state-action tuple. However, issues relating to time limits are not addressed in the paper. I also think that it is strange that the direct way of computing the return for the terminal state is much less stable than recursively computing it and think that the paper should include a convincing explanation.
---------------
Update 21.11.2018
I think my initial assessment was too positive. During the rebuttal, I noticed that the discussion of reward bias was not only shallow but also wrong in some aspects and very misleading, because problems arising from hacky implementations of some RL toolboxes were discussed as theoretical shortcoming of AIL algorithms. Hence, I think the initial submission should be clearly rejected. However, the authors submitted a revised version that presents the root of the observed problem much more accurately. I think that the revised version is substantially better than the original submission. However, I think that my initial rating is still valid (better: became valid), because the main issues that I raised for the initial submission still apply to the current revision, namely:
- The technical contributions are minor.
- The theoretical discussion (in particular regarding absorbing states) is quite shallow.
The merits of the paper are:
- Good results due to off-policy learning
- Raising awareness and providing a fix for a common pitfall
I think that the problems arising from incorrectly treated absorbing states needs to be discussed more profoundly.
Some suggestions:
Section 3.1
"As we discuss in detail in Section 4.2 [...]"
I think this should refer to section 4.1. Also the discussion should in section 4.1 should be a bit more detailed. How do common implementations implicitly assign zero rewards? Which implementations are affected? Which papers published inferior results due to this bug? I think it is also important to note, that absorbing states are hidden from the algorithm and that the reward function is thus only applied to non-absorbing states.
"We will demonstrate empirically in Section 4.1 [...]"
The demonstration is currently missing. I think it would be nice to illustrate the problem on a simple example. The original example might actually work, as shown by the code example of the rebuttal, however the explanation was not convincing. Maybe it would be easier to argue with a simpler algorithm (e.g MaxEnt-IRL, potentially projecting the rewards to positive values)?
Section 3.1 seems to focus too much on resets that are caused by time limits. Such resets are inherently different from terminal states such as falling down in locomotion tasks, because they can not be modelled with the given MDP formulation unless time is considered part of the state. Indeed, I think that for infinite horizon MDPs without time-awareness, time limits can not be modelled using absorbing states (I think the RL book misses to mention that time needs to be part of the state such that the policy remains Markovian, which is a bit misleading). Instead those resets are often handled by returning an estimate of the future return (bootstrapping). This treatment of time limits is already part of the TD3 implementation and as far as I understood not the focus of the paper. Instead section 3.1. should focus on resets caused by task failure/completion, which can actually be modelled with absorbing states, because the agent will always transition to the absorbing state when a terminal state is reached which is in line with Markovian dynamics.
Section 4.2 should also add a few more details. Did I understand correctly, that when computing the return R_T the sum is indeed finite and stopped after a fixed horizon? If yes, this should be reflected in the equation, and the horizon should be mentioned in the paper. The paper should also better explain how  the proposed fix enables the algorithm to learn the reward of the absorbing state. For example, section 4.2. does not even mention that the state s_a was added as part of the solution.
-------------
Update 22.11.2018
By highlighting the difference between termination due to time-limits and termination due to task completion, and by better describing how the proposed fix addresses the problem of reward bias that is present in common AIL implementations, the newest revision further improves the submission.
I think that the submission can get accepted and I adapted my rating accordingly.
Minor:
Conclusion should also squeeze in somehow that the reward biases are caused by the implementations.
Typo in 4.2: "Thus, when sample[sic] from the replay buffer AIL algorithms will be able to see absorbing states there[sic]
were previous hidden, [...]"


Predicting the outcome of pharmacological assays based on high-resolution microscopy images of treated cells is a crucial task in drug discovery which tremendously increases discovery rates. However, end-to-end learning on these images with convolutional neural networks (CNNs) has not been ventured for this task because it has been considered infeasible and overly complex. On the largest available public dataset, we compare several state-of-the-art CNNs trained in an end-to-end fashion with models based on a cell-centric approach involving segmentation. We found that CNNs operating on full images containing hundreds of cells perform significantly better at assay prediction than networks operating on a single-cell level. Surprisingly, we could predict 29% of the 209 pharmacological assays at high predictive performance (AUC > 0.9). We compared a novel CNN architecture called "GapNet" against four competing CNN architectures and found that it performs on par with the best methods and at the same time has the lowest training time. Our results demonstrate that end-to-end learning on high-resolution imaging data is not only possible but even outperforms cell-centric and segmentation-dependent approaches. Hence, the costly cell segmentation and feature extraction steps are not necessary, in fact they even hamper predictive performance. Our work further suggests that many pharmacological assays could be replaced by high-resolution microscopy imaging together with convolutional neural networks.

The authors explore the possibility of using an end-to-end approach for predicting pharmacological assay outcome using fluorescence microscopy images from the public Cell Painting dataset. In my view, the primary contributions are the following: an interesting and relatively new application (predicting assay outcomes), enriching the CellPainting dataset with drug activity data, and a comparison of several relevant methods and architectures. The technical novelty is weak, and although the authors demonstrate that end-to-end holistic approaches outperform previous segmentation-and-feature-extraction approaches, this result is not surprising and has been previously reported in closely related contexts.
OVERVIEW
The authors evaluate the possibility of using and end-to-end deep learning approach to predict drug activity using only image data as input. The authors repurpose the CellPainting dataset for activity prediction by adding activity data from online ChEMBL databases. If made available as promised, the dataset will be a valuable resource to the community. The authors compare a number of previous approaches and state-of-the-art image classification network architectures to evaluate the use of CNNs instead of more classical image analysis pipelines. The comparison is a strong point of the paper, although some details are lacking. For example, the authors claim that GapNet is the quickest method to train, and while they report the number of hyperparameters and time per epoch, the number of epochs trained is never mentioned.
The authors propose an architecture (GapNet) for the assay prediction task. While the way Global Average Pooling is used to extract features at different stages in the network might be new, it is a straightforward combination of GAP and skip connections. Little insight into why this approach is more efficient or evidence for its effectiveness is provided. Similarly, more explanation for why dilated convolutions and SELU activations would be appreciated. A comparison between GapNet and the same network without the GAP connections could possibly provide a more interesting comparison and might also provide a more pervasive argument as to why GapNet’s should be used. Ultimately, the benefit of using GapNet over the other architectures is not strongly motivated, as training time is less of a concern in this application than predictive power.
RELATED WORK
The authors present previous work in a clear and comprehensive manner. However, the reported finding that “CNNs operating on full images containing hundreds of cells can perform significantly better at assay prediction than networks operating on a single-cell level” is not surprising, and partial evidence of this can be found in the literature. In [1], it was shown that penultimate feature activations from pre-trained CNNs applied to whole-image fluorescence microscopy data (MOA prediction) outperform the baseline segmentation-then-feature extraction method (FNN). Similarly, in [2] (the paper proposing MIL-Net), it is shown that end-to-end whole-image CNN learning for protein localization outperforms the baseline (FNN). In [3] whole image end-to-end learning outperforms whole image extracted features for a phenotyping task. All of these references use fluorescence microscopy data similar to the dataset in this work.
[1] Pawlowski, Nick, et al. "Automating morphological profiling with generic deep convolutional networks." bioRxiv (2016): 085118.
[2] Kraus, Oren Z., Jimmy Lei Ba, and Brendan J. Frey. "Classifying and segmenting microscopy images with deep multiple instance learning." Bioinformatics 32.12 (2016): i52-i59
[3] Godinez, William J., et al. "A multi-scale convolutional neural network for phenotyping high-content cellular images." Bioinformatics 33.13 (2017): 2010-2019.
APPROACH
The authors compile enrich the CellPaining dataset with activity data from various drug discovery assays. In my view, the creation of this dataset is the strongest and most valuable contribution of the paper. The method used to collect the data is described clearly and the choices made when compiling the dataset, including the thresholds and combinations of activity measures seems like a well founded approach.
The authors then identify a number of approaches that are relevant for the problem at hand, binary prediction of drug activity based on image data. These include previous approaches used for cell images and modern image classification networks.
EXPERIMENTS
The different approaches/networks mentioned above were evaluated on a testset. The results indicate that end-to-end CNN approaches outperform all non-end-to-end with no significant difference between the individual end-to-end CNNs. The results are stated clearly and the presentation of different metrics is a nice addition to properly compare the results. It would however contribute valuable information if the authors stated how the confidence intervals of the F1 score are calculated (are the experiments based on several runs of each network or how is it done).
NOVELTY/IMPACT
+ Creation of a new dataset on a new and interesting problem
+ Useful comparison of modern networks on the task
- GapNet - lacking technical novelty, insight, and performance is unconvincing
- Demonstrates that end-to-end learning outperforms cell centric approach - was this really surprising or even new information?
OTHER NOTES:
* Figure 3 is never mentioned in the main text
* Figure 3 (*’s) are confusing. Do they represent outliers? Statistical significance tests?
* Figure 5 which panel is which?
* Be clear what you mean when you refer to “upper layers” of a network
* An important point not mentioned: in practice, many assays use stains that are closely tied to the readout, unlike the dataset here which provides only landmark stains. The results found here do not necessarily apply in other cases.


We show that dropout training is best understood as performing MAP estimation concurrently for a family of conditional models whose objectives are themselves lower bounded by the original dropout objective. This discovery allows us to pick any model from this family after training, which leads to a substantial improvement on regularisation-heavy language modelling. The family includes models that compute a power mean over the sampled dropout masks, and their less stochastic subvariants with tighter and higher lower bounds than the fully stochastic dropout objective. We argue that since the deterministic subvariant's bound is equal to its objective, and the highest amongst these models, the predominant view of it as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective.

This paper studies the problem of making predictions with a model trained using dropout. Authors try to provide a theoretical foundation for using dropout when making predictions. For this purpose, they show that when using dropout training we are maximizing a common lower bound on the objectives of a family of models, including most of the previously used methods for prediction with dropout.
I find that the paper addresses a relevant problem and try to apply a novel approach. But, in general, I find the paper is not easy to follow and to grasp the main ideas.
Here I detail my main concerns:
1. This is one of my main concerns. The contraposition between the geometric and the average model. I don't like this contraposition. The average model is just the standard marginalization operation over the weights, $p(y|x) = \int p(y|x,w)p(w|\Theta)dw$. This is the natural solution for the prediction problem to the problem if we accept the generative model given in Eq (3).
In the case of the variational dropout, we depart from the same generative model, but we employ an approximation. It is the variational approximation the one that induces the geometric mean provided in eq (6). I.e. if we want to compute the posterior over the label y* for a sample x*, after training, we should compute the associated lower bound
$\ln p(y*|x*) >= E_q[\ln p(y*|x*,w)] - KL(q|p)$
In this case, q(w) = p(w|\Theta), as stated Eq (3) and in the corresponding equation provided in page 2 (the q(w) is not learnt because it only depends on the dropout rate, while the $\Theta$ are learnt by maximum log-likelihood and do not have a $q$ associated).  This gives rise to the geometric mean approximation provided in Eq (6).  I.e. the geometric mean prediction is simply the result of using a variational approximation at prediction time.
My problem here is that authors employ convoluted arguments to introduce this geometric mean prediction and the average prediction, without making the connection discussed above.
3. Section 3.3 and 3.4 introduces new arguments for modifying the dropout rate (and the alpha) parameter at test time. But, again, I find the arguments convoluted. We consider the dropout rate a hyper-parameter of the model, the standard learning theory tells us to fix the parameters with the training data and evaluate them later when making predictions. Why should we use different dropout rates at training and testing? Authors arguments about the tightness of the bound of Eq (8) and Eq(9). are not convincing to me.
So, I don't find authors provide convincing answers to the raised questions at the beginning of the paper about the use of dropout when making predictions.
Minor comments:
1. The generative model for Variational dropout is the same than the generative model for the "conditional model", eq. (3).
2. In Eq. (7) authors are defining the weighted power mean. I think it would be clearer to directly introduce the weighted power mean instead of the standard power mean in Section 3.2.
3. Section 3.3. I find some parts are difficult to understand. "suppose we pick a base model from the power mean family and have a continuum of subvariants with gradually reduced variance in their predictions but the same expectation." Later, I can understand authors are referring to the possibility of reducing the dropout rate.


LANCZOSNET: MULTI-SCALE DEEP GRAPH CONVO- LUTIONAL NETWORKS
We propose the Lanczos network (LanczosNet), which uses the Lanczos algorithm to construct low rank approximations of the graph Laplacian for graph convolution. Relying on the tridiagonal decomposition of the Lanczos algorithm, we not only efficiently exploit multi-scale information via fast approximated computation of matrix power but also design learnable spectral filters. Being fully differentiable, LanczosNet facilitates both graph kernel learning as well as learning node embeddings. We show the connection between our LanczosNet and graph based manifold learning methods, especially the diffusion maps. We benchmark our model against several recent deep graph networks on citation networks and QM8 quantum chemistry dataset. Experimental results show that our model achieves the state-of-the-art performance in most tasks.

This paper proposes to use a Lanczos alogrithm, to get approximate decompositions of the graph Laplacian, which would facilitate the computation and learning of spectral features in graph convnets. It further proposes an extension with back propagation through the Lanczos algorithm, in order to train end to end models.
Overall, the idea of using Lanczos algorithm to bypass the computation of the eigendecomposition, and thus simplify filtering operations in graph signal processing is not new [e.g., 35]. However, using this algorithm in the framework of graph convents is new, and certainly interesting. The authors seem to claim that their method permits to learn spectral filters, what other methods could not do - this is not completely true and should probably be rephrased more clearly: many graph convnets, actually learn features.
The general construction and presentation of the algorithms are generally clear, and pretty complete. A few things that could be clarified are the following:
- in the spectral filters of Eq (4), what gets fundamentally different from polynomial filters proposed in other graph convnets architectures?
- what happens when the graph change? Do the learned features make sense on different graphs? And if yes, why? If not, the authors should be more explicit in their presentation
- what is the complexity of the proposed methods? that should be minimally discussed (at least), as it is part of the key motivations for the proposed algorithms
- how is the learning done in 3.2? If there is any learning at all? (btw, S below Eq (6) is a poor notation choice, as S is used earlier for something else)
- the results are not very impressive - they are good, but not stellar, and could benefit from showing an explicit tradeoff in terms of complexity too?
The discussion in the related work, and the analogy with manifold learning are interesting. However, that brings probably to one of the main issues with the papers - the authors are obviously very knowledgeable in graph convnets, graph signal processing, and optimisation. However, there are really too many things in this paper, which leads to numerous shortcuts, and some time confusion. Given the page limits, not everything can be treated with the level of details that it would deserve. It might be good to consider trimming down the paper to its main and core aspects for the next version.


KNOCKOFFGAN: GENERATING KNOCKOFFS FOR FEATURE SELECTION USING GENERATIVE ADVER- SARIAL NETWORKS
Feature selection is a pervasive problem. The discovery of relevant features can be as important for performing a particular task (such as to avoid overfitting in prediction) as it can be for understanding the underlying processes governing the true label (such as discovering relevant genetic factors for a disease). Machine learning driven feature selection can enable discovery from large, high-dimensional, nonlinear observational datasets by creating a subset of features for experts to focus on. In order to use expert time most efficiently, we need a principled methodology capable of controlling the False Discovery Rate. In this work, we build on the promising Knockoff framework by developing a flexible knockoff generation model. We adapt the Generative Adversarial Networks framework to allow us to generate knockoffs with no assumptions on the feature distribution. Our model consists of 4 networks, a generator, a discriminator, a stability network and a power network. We demonstrate the capability of our model to perform feature selection, showing that it performs as well as the originally proposed knockoff generation model in the Gaussian setting and that it outperforms the original model in non-Gaussian settings, including on a real-world dataset.

This manuscript describes an extension of the knockoff framework, which is designed to carry out feature selection while controlling the FDR among selected features, to settings in which the generative distribution of the features is not Gaussian. Specifically, the authors employ a GAN (with several modifications and additions) in which the generator produces knockoffs and the discriminator attempts to identify which features have been swapped between the original and the knockoff.
The method works as follows: (1) A conditional generator takes random noise and the real features as input, and outputs knockoff features. (2) A modified discriminator is used in such a way that the generator learns to generate knockoffs satisfying the necessary swap condition, so as to control the FDR of the knockoff procedure. (3) A power network uses Mutual Information Neural Estimation (MINE) to estimate the mutual information between each feature and its knockoff counterpart, so as to maximize the power of the knockoff procedure.
Results are provided on synthetic and real data. As for the synthetic data, when the underlying feature distribution is Gaussian, the proposed method, KnockoffGAN, performs almost as well as the original knockoff and outperforms the BHq method; when the underlying feature distribution is non-Gaussian, KnockoffGAN dominates both the original knockoff and BHq methods. As for the real data, the authors claim to identify nine relevant features for cardiovascular disease and eight relevant features for diabetes, whereas the original knockoff procedure identifies zero features from the same data.
General comments:
This is an extremely impressive piece of work. The manuscript itself is a pleasure to read, and the results clearly demonstrate that the proposed KnockoffGAN both controls FDR and achieves power comparable to the original knockoff procedure in the Gaussian setting and much better than the original knockoff when the underlying distribution is not Gaussian.
Strengths:
The combination of GANs and knockoff filter is a very promising and intriguing idea.
The use of the modified discriminator to ensure that the generated knockoffs satisfy the necessary swap condition is novel and intuitively sound.
The use of MINE to maximize power by maximizing the mutual information between each feature and its knockoff counterpart is also interesting.
The paper is well written, reads smoothly and the ideas are well exposed.
The illustrative figure is straightforward.
Weaknesses:
Intuitively, the modified discriminator and the power network should conflict with each other. I expect it was tricky to achieve a good tradeoff between two, but the authors failed to elaborate on these details.
The authors do not provide the design details of the neural networks. How dependent on the specific parametrization of the network architecture is the performance? How does the training order of four networks matter to the performance?
The manuscript should cite [[ Jaime Roquero Gimenez, Amirata Ghorbani, and James Zou. "Knockoffs   for the mass: new feature importance statistics with false discovery   guarantees." arXiv:1807.06214, 2018. ]] which proposes a way to generate knockoffs for a Gaussian mixture model, and this method should be included in the relevant supplementary figure.
In Section 5.1.4, I would like to know, for a fixed data set, how the regularization affects the final values of the other loss terms.
The analysis of real data in Section 5.2 is unsatisfying in several respects.
First, there is an unfortunate oversight in Table 1: the text refers to three features that are "trivial," but only one of these is marked with an asterisk.  This leaves open the question of whether there are other trivial features beyond the three mentioned in the text. In addition, it is not clear exactly what it means for a feature to be "trivial" in this context.
This point gets to a deeper problem with the evaluation, which is that we are told, with no evidence, that these features are supported by literature in PubMed.  I would like to see two things here.  First, it seems obvious to me that if you are going to say that there is support in PubMed, you are obliged to actually report the citations that supposedly give this support. This could be done in the appendix. Equally importantly, there is a potential here for ascertainment bias which should be combatted in some fashion.  Presumably, some human expert had to do the PubMed searches to make this assessment.  I would like to know how "permissive" this assessor is.  To assess this, one could give the assessor a collection of terms, some of which were selected by KnockoffGAN and some at random, and then report the results. Obviously, some features that are significant may not be in the list of selected terms (because KnockoffGAN does not achieve 100% power) and so may appear as false negatives. But without some assessment like this, I have trouble believing this assessment.
A related point is that it seems quite unfortunate that the authors chose a data set that cannot be described at all due to the anonymity constraint.  At the very least, it seems that we should be told the dimensionality of the data set. The Knockoff literature contains real data sets that could have been used here.
Minor comments:
On the first page, the sentence beginning "On the other hand," should clarify that this is only in expectation.
p. 3: Missing right paren after [7].
p. 5: Write out “Gaussian process.”
p. 5: "as little" -> "as little as possible"
p. 6: "to show that in" -> "to show, in"
In Figures 2-5, add a horizontal line at 10% FDR for reference.
p. 10: "features ones" -> "features"
Note to program committee:
I did not review the technical details of the proof in the appendix.


POST SELECTION INFERENCE WITH INCOMPLETE MAXIMUM MEAN DISCREPANCY ESTIMATOR
Measuring divergence between two distributions is essential in machine learning and statistics and has various applications including binary classification, change point detection, and two-sample test. Furthermore, in the era of big data, designing divergence measure that is interpretable and can handle high-dimensional and complex data becomes extremely important. In this paper, we propose a post selection inference (PSI) framework for divergence measure, which can select a set of statistically significant features that discriminate two distributions. Specifically, we employ an additive variant of maximum mean discrepancy (MMD) for features and introduce a general hypothesis test for PSI. A novel MMD estimator using the incomplete U-statistics, which has an asymptotically normal distribution (under mild assumptions) and gives high detection power in PSI, is also proposed and analyzed theoretically. Through synthetic and real-world feature selection experiments, we show that the proposed framework can successfully detect statistically significant features. Last, we propose a sample selection framework for analyzing different members in the Generative Adversarial Networks (GANs) family.

The authors focus on the selection problem of k statistically significant features discriminating 2 probability distributions accessible via samples. They propose a non-parametric approach under the PSI (post selection inference) umbrella using MMD (maximum mean discrepancy) as a discrepancy measure between probability distributions. The idea is to apply (asymptotically) normal MMD estimators, rephrase the top-k selection problem as a linear constraint, and reduce the problem to Lee et al., 2016. The efficiency of the approach is illustrated on toy examples and in GAN (generative adversarial network) context. The technique complements the PSI-based independence testing approach recently proposed by Yamada et al., 2018.
The submission is a well-organized, clearly written, nice contribution; it can be relevant to the machine learning community.
Below I enlist a few suggestions to improve the manuscript:
-Section 1: The notion of characteristic kernel (kernel when MMD is metric) has not been defined, but it was referred to. 'Due to the mean embeddings in RKHS, all moment information is stored.': This sentence is somewhat vague.
-Section 1: 'MMD can be computed in closed form'. This is rarely the case (except for e.g. Gaussian distributions with Gaussian or polynomial kernels). I assume that the authors wanted refer to the estimation of MMD.
-Section 1: 'K nearest neighbor approaches (Poczos & Schneider, 2011)'. The citation to this specific estimator can go under alpha-divergences. The Wasserstein metric could also be mentioned.
-Section 3.1: k is used to denote the number of selected features and also the kernel used in MMD. I suggest using different notations.
-Theorem 1: '\Phi is the CDF...'. There is no \Phi in the theorem.
-Section 3.2: The existence of MMD (mean embedding) requires certain assumptions: E_{x\sim p}\sqrt{k(x,x)} < \infty, E_{x\sim q}\sqrt{k(x,x)} < \infty.
-Section 3.2.: block estimator: 'B_1 and B_2 are finite'. 'fixed'?
-Section 3.2.: MMD_{inc}:
   i) 'S_{n,k}': k looks superfluous.
   ii) 'l': it has not been introduced (cardinality of D).
-Section 3.3: typo: 'covraiance' (2x)
-Section 3.3: Fan et al. 2013: The citation can go to \citep{}.
-Theorem 2:
   i)'c' is left undefined.
   ii)Comma is missing before 'where'.
   iii)\xrightarrow{d} (Theorem 2, Corollary 3-4): Given that 'd' also denotes dimension in the submission, I suggest using a different notation for convergence in distribution.
-At the introduction of block-MMD the block size (B) was fixed, while in the experiments (e.g. Figure 3) it is growing with the sample size (B=\sqrt{n}). The assumption on B should be clearly stated.
-Section 5.1: (b) mean shift: comma is missing before 'where'.
-References:
   i) Abbreviations and names in the titles should be capitalized (such as cramer, wasserstein, hilbert-schmidt, gan, nash).
   ii) Scholkopf should be Sch\{"o}lkopf (in the ALT 2005 work).
   iii) 'Exact post-selection inference, with application to the lasso': All the authors are listed; 'et al.' is not needed.


Online and stochastic optimization methods such as SGD, ADAGRAD and ADAM are key algorithms in solving large-scale machine learning problems including deep learning. A number of schemes that are based on communications of nodes with a central server have been recently proposed in the literature to parallelize them. A bottleneck of such centralized algorithms lies on the high communication cost incurred by the central node. In this paper, we present a new consensus-based distributed adaptive moment estimation method (DADAM) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation. Such a framework not only can be extremely useful for learning agents with access to only local data in a communication constrained environment, but as shown in this work also outperform centralized adaptive algorithms such as ADAM for certain realistic classes of loss functions. We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings. Empirical results demonstrate that DADAM works well in practice and compares favorably to competing online optimization methods.

This paper presents a consensus-based decentralized version of the Adam algorithm for online optimization. The authors consider an empirical risk minimization objective, which they split into different components, and propose running a separate online optimization algorithm for each component, with a consensus synchronization step that involves taking a linear combination of the parameters from each component before applying each component's individual parameter update. The final output is a simple average of the parameters from each component.
The authors study the important problem of distributed optimization and focus on adapting existing state-of-the-art methods to this setting. The algorithm is clearly presented, and to the best of my knowledge, original. The fact that this work includes both theoretical guarantees for the convex and non-convex settings as well as numerical experiments strengthens the contribution.
On the other hand, I didn't find the actual method presented by the authors to be motivated very well. The main innovation with respect to the standard Adam/AMSGrad algorithm is the use of a mixing matrix W, but the authors do not discuss how the choice of this matrix influences the performance of the algorithm or how one should specify this input in practice. This seems like an important issue, especially since all of the bounds depend on the second singular value of this matrix. Moreover, arguments such as Corollary 10 do not actually imply that DADAM outperforms ADAM when this singular value is large, making it difficult to assess the impact of this work. The numerical experiments also do not test for the statistical significance of the results.
There are also many typos that make the submission seem relatively unpolished.
Specific comments:
1. page 1: "note only". Typo.
2. page 2: "decentalized". Typo.
3. page 2: "\Pi_X[x]. If \Pi_X(x)...." Inconsistent notation.
4. page 3: "largest singular of matrix". Typo.
5. page 3: "x_t* = arg min_{x \in X} f_t(x)". f_t isn't defined in up to this point.
6. page 4: "network cost is then given by f_t(x) = \frac{1}{n} \sum_{i=1}^n f_{i,t}(x)" Should the cost be  \frac{1}{n} \sum_{i=1}^n f_{i,t}(x_{i,t})? That would be more consistent with the definition of regret presented in Reg_T^C.
7. page 4: "assdessed". Typo.
8. page 4: " Reg_T^C := \frac{1}{n} \sum_{i=1}^n \sum)_{t=1}^T f_t(x_{i,t})..." Why is this f_t and not f_{i,t}?
9. page 4: "\hat{v}_{i,t} = v_3 ..." You should reference how this assignment in the algorithm relates to the AMSGrad algorithm. Moreover, you should explain why you chose to use a convex combination in the assignment instead of just the max.
10. page 5: Definition 1. This calculation should be derived and presented somewhere (e.g. in the appendix).
11. page 5: Assumption 3. The notation for the stochastic gradient is not very clear and easily distinguishable from the notation for the deterministic gradient.
12. page 5: Theorem 4. D_T can be very large in the bound, which would make the upper bound meaningless. Can you set hyperparameters in such a way to minimize it? Also, what is the typical size of \sigma_2(W) that one would incur?
13. page 6: Remark 6. This remark seems misleading. It ignores the log(T) and D_T terms, both of which may dominate the data dependent arguments.
14. page 6: "The update rules \tilde{v}_{i,t}...". \tilde{v}_{i,t} is introduced but never defined.
15. page 6: Last display equation. The first inequality seems like it can be an equality.
16. page 7: Equation (14). Doesn't the presence of \sigma_2(W) imply that the O(1/T) term may not be negligible? It would also be helpful to give some examples of how large T needs to be in (15a) and (15b) in order for this statement to take effect.
17. page8: "distributed federated averaging SGD (FedAvg)". What is the reference for this? It should be included here. It should probably also be mentioned in the introduction as related work.
18. page 9: Figure 1. Without error bars, it is impossible to tell the statistical significance of these results. Moreover, how sensitive are these results to different choices of hyperparameters?
19. page 9: "obtain p coefficients". What is p in these experiments?
20. page 9: Metropolis constant edge weight matrix W". What is \sigma_2(W) in this case?
21. page 10: Acknowledgements. This shouldn't be included in the submission.


Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for two different tasks: learning to follow navigational instructions and embodied question answering. In this paper, we aim to learn a multitask model capable of jointly learning both tasks, and transferring knowledge of words and their grounding in visual objects across tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual objects in the visual representations, and align them with each other. This disentangled task-invariant alignment of representations facilitates grounding and knowledge transfer across both tasks. We show that the proposed model outperforms a range of baselines on both tasks in simulated 3D environments. We also show that this disentanglement of representations makes our model modular, interpretable, and allows for transfer to instructions containing new words by leveraging object detectors.

The authors propose a multitask model using a novel “dual-attention” unit for embodied question answering (EQA) and semantic goal navigation (SGN) in the virtual game environment ViZDoom. They outperform a number of baseline models originally developed for EQA and SGN (but trained and evaluated in this multitask paradigm).
Comments and questions on the model and evaluation follow.
1. Zero-shot transfer claim:
1a. This is not really zero-shot transfer, is it? You need to derive object detectors for the meanings of the novel words (“red” and “pillar” from the example in the paper). It seems like this behavior is supported directly in the structure of the model, which is great — but I don’t think it can be called “zero-shot” inference. Let me know if I’ve misunderstood!
1b. Why is this evaluated only for SGN and not for EQA?
2. Dual attention module:
2a. The gated attention model only makes sense for inputs in which objects or properties (the things picked out by convolutional filters) are cued by single words. Are there examples in the dataset where this constraint hold (e.g. negated properties like “not red”)? How does the model do? (How do you expect to scale this model to more naturalistic datasets with this strong constraint?)
2b. A critical claim of the paper is that the model learns to “align the words in both the tasks and transfer knowledge across tasks.” (Earlier in the paper, the claim is that “This forces the convolutional network to encode all the information required with respect to a certain word in the corresponding output channel.”) I was expecting you would show some gated-attention visualizations (not spatial-attention visualizations, which are downstream) to back up this claim. Can you show me visualizations of the gated-attention weights (especially when trained on the No-Aux task) which demonstrate that words and objects/properties in the images have been properly aligned? Show that e.g. the filter at index i only picks out objects/properties cued by word i?
3. Auxiliary objective: it seems like this objective solves most of the language understanding problem relevant in this task. Can you motivate why it is necessary? What is missing in the No-Aux condition, exactly? Is it just an issue with PPO optimization? Can you do error analysis on No-Aux to motivate the use of the Aux task?
4. Minor notes:
4a. In appendix A, the action is labeled “Turn Left” but the frames seem to suggest that the agent is turning right.
4b. How are the shaded regions estimated in figs. 7, 8? They are barely visible — are your models indeed that consistent across training runs? (This isn’t what I’d expect from an RL model! This is true even for No-Aux..?)
4c. Can you make it clear (via bolding or coloring, perhaps) which words are out-of-vocabulary in Table 3? (I assume “largest” and “smallest” aren’t OOV, for example?)


Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging-the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior works in these aspects.

The paper introduces a generative model for video prediction. The originality stems from a new training criterion which combines a VAE and a GAN criteria. At training time, the GAN and the VAE are trained simultaneously with a shared generator; at test time, prediction conditioned on initial frames is performed by sampling from a latent distribution and generating the next frames via an enhanced conv LST . Evaluations are performed on two movement video datasets classically used for benchmarking  this task - several quantitative evaluation criteria are considered.
The paper clearly states the objective and provides a nice general description of the method.  The proposed model extends previous work by adding an adversarial loss to a VAE video prediction model.  The evaluation compares different variants of this model to two recent VAE baselines. A special emphasis is put on the quantitative evaluation: several criteria are introduced for characterizing different properties of the models with a focus on diversity. w.r.t. the baselines, the model behaves well for the “realistic” and “diversity” measures. The results are more mitigated for measures of accuracy. As for the qualitative evaluation, the model corrects the blurring effect of the reference SV2P baseline, and produces quite realistic predictions on these datasets. The difference with the other reference model (SVG) is less clear.
While the general description of the model is clear, details are lacking. It would probably help to position the VAE component more precisely w.r.t. one of the two baselines, by indicating the differences. This would also help to explain the difference of performance/ behavior  w.r.t. these models (Fig. 5).
It seems that the discriminator takes a whole sequence as input, but some precision on how this done could be added.  Similarly, you did not indicate what the deterministic version of your model is.
The generator model with its warping component makes a strong hypothesis on the nature of the videos: it seems especially well suited for translations or for other simple geometric transformations characteristics of the benchmarking videos .  Could you comment on the importance of this component? Did you test the model on other types of videos where this hypothesis is less relevant? It seems that the baseline SVG makes use of simpler ConLSTM for example.
The description of the generator in the appendix is difficult to follow. I missed the point in the following sentence: “For each one-step prediction, the network has the freedom to choose to copy pixels from the previous frame, used transformed versions of the previous frame, or to synthesize pixels from scratch” .
Also, it is not clear from the discussion on z, whether sampling is performed once for each video of for each frame.
Overall, the paper proposes an extension of VAE based video prediction models and produces an extensive evaluation. While the model seems to perform well, the originality and the improvement w.r.t. baselines are somewhat limited.