The standard variational lower bounds used to train latent variable models produce biased estimates of most quantities of interest. We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series. If parameterized by an encoder-decoder architecture, the parameters of the encoder can be optimized to minimize its variance of this estimator. We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost. This estimator also allows use of latent variable models for tasks where unbiased estimators, rather than marginal likelihood lower bounds, are preferred, such as minimizing reverse KL divergences and estimating score functions.

This paper proposes an unbiased estimator of $\log p_\theta(x)$. Many unbiased estimators of $p_\theta$ exist, but $\log p_\theta$ is needed in many other settings, some of which are not well-served by standard estimators of $p_\theta$. The SUMO estimator is essentially a Russian roulette-based extension of IWAE; it is exactly unbiased, but takes a random and unbounded number of samples.
This allows marginally better optimization of certain models than IWAE with a much smaller average number of samples, and (more importantly) opens new possibilities such as entropy maximization which are not well-served by lower bounds like IWAE. This is a very nice advantage of this estimator.
One complaint about this class of estimators in general is: yes, the exact SUMO procedure is technically unbiased. But in practice, if SUMO takes more than, say, a day of compute time – something that will happen with extremely small but nonzero probability – then the user will kill it. And the SUMO estimator conditioned on taking less than a day of compute time is actually biased. This also likely means that, though unbiased, these estimators can be potentially skewed or otherwise "unpleasant." For SUMO in particular, the estimator with $K$ truncated probably has bias bounded based on the bias of IWAE with batch size equal to the truncation point, which is likely quite small. But it would be nice to understand this a little more. (Perhaps it's been studied by some of the recent cited work on these types of estimators.)
Relatedly, you don't prove that this estimator has a finite variance, and in fact it seems plausible theoretically that it might be infinite. Like the "notorious" harmonic mean estimator, this is troubling. It seems that things are okay in practice, but how can we tell whether the variance is really finite or not? I don't know if there's a good answer, but one diagnostic might be something like the one suggested at https://stats.stackexchange.com/a/9143 . (Your comments about occasional "bounded but very large" gradient estimates are troubling in this respect, depending on what exactly you mean by "bounded".) When you do gradient clipping, the estimator of course then has a finite variance, but can we get some sense of how much bias that introduces?
Overall, though, I think this is a very nice new estimator that is both well-founded – despite leaving some questions open – and likely to be practically useful. Given that it is also extremely on-topic for ICLR and novel, I'm rating the paper as "accept."
(For slightly more detail on the "thoroughness assessment": I did not really check the proofs in the appendix, but did pay attention to the derivations in the main body.)
Smaller notes:
- Top of page 3: you comment that SGD "requires unbiased estimates of" gradients of the log-density. In fact, SGD can be shown to work with biased gradient estimators, with suboptimality in the results depending on the bias; see e.g. Chen and Luss, http://arxiv.org/abs/1807.11880 .
- In the definition of $\tilde{Y}$, above (7): it might make more sense to define $\tilde{Y}$ with some recursive scheme, rather than as an estimator that either computes one of the $\Delta$ values or infinitely many of them.
- Start of 3.1: presumably $\Delta_k$ is what converges absolutely, not IWAE?
- Start of 3.2: as you note, it is clearly not true that the $\Delta_k^g$ are independent. But you don't really "assume independence" – the Russian roulette estimator is still a valid estimator, just perhaps not the optimal among that class. It would be better to say something like that since it seems that the $\Delta_k^g$ are *nearly* independent (or at least nearly uncorrelated), the Russian roulette estimator is probably at least a reasonable choice.
- Re: the discussion after (9) and in (12), as well as a few other places: I think you show in Appendix A.3 that $\mathbb E[ \nabla \operatorname{SUMO}(x) ]$ exists, but you don't show that it equals $\nabla \mathbb E[ \operatorname{SUMO}(x) ]$. This is likely true, particularly if $q$ and $p$ are each everywhere-differentiable, and it's totally fine if you don't want to prove it out formally, but it would be worth at least a footnote that this is a thing that requires proof. (See e.g. https://arxiv.org/abs/1801.01401 Theorem 5, for a formal result of this type supporting ReLU activations, which you may be able to just use directly.)


Designing energy-efficient networks is of critical importance for enabling stateof-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently,  framed the search of efficient neural architectures into a continuous splitting process: it iteratively splits existing neurons into multiple off-springs to achieve progressive loss minimization, thus finding novel architectures by gradually growing the neural network. However, this method was not specifically tailored for designing energy-efficient networks, and is computationally expensive on large-scale benchmarks. In this work, we substantially improve  in two significant ways: 1) we incorporate the energy cost of splitting different neurons to better guide the splitting process, thereby discovering more energy-efficient network architectures; 2) we substantially speed up the splitting process of Wu et al. (2019), which requires expensive eigen-decomposition, by proposing a highly scalable Rayleigh-quotient stochastic gradient algorithm. Our fast algorithm allows us to reduce the computational cost of splitting to the same level of typical back-propagation updates and enables efficient implementation on GPU. Extensive empirical results show that our method can train highly accurate and energy-efficient networks on challenging datasets such as ImageNet, improving a variety of baselines, including the pruning-based methods and expert-designed architectures.

The paper is well written and flows very well. The idea is straightforward and easy to understand. Here are my feedbacks:
--Method--
Methodology-wise, the novelty is somewhat limited. The main technical contributions are: 1) formulate linear programming to reduce energy costs, and the formulation of linear programming is straightforward. 2) the claimed main contribution is an application of Rayleigh-Quotient gradient descent to approximate the eigenvalue calculations in the model proposed by Wu et al (2019).
I believe computing the eigenvalue shall not be the only way to solve Eq.(3), e.g. using gradient-based method or Lagrange. My main question is whether computing the global optimum really matters? Since your method is still essentially an approximation algorithm, which may break the optimality condition here. From the experimental results, it seems that your approximation is quite close to the analytical solution. Therefore, it is unclear computing a global optimum really matters here.
I tried to buy the idea of escaping the saddle point with splitting (it indeed sounds straightforward and reasonable). It will be great if the author can add some empirical experiments to verify it.
--Content--
The entire section2 is at reviewing prior works, and I believe you should cut down the content here.
--Experiments--
1. Diversity of your tests: the author main uses MobileNet in testing their ideas. It seems that their method starts at MobileNet, then tries to improve it. I suggest authors adding other recent works, e.g. FBNet, MobileNetV3, into their tests to diversify the types of networks.
2. Results variations: all figures, e.g. fig.3, fig.4, in the paper lack plotting their results variations. Since the optimization may converge to different local optimum, it is more persuasive to show their performance variations.
3. The final results are not surprising: in table.1 and table.2, as far as I'm aware, the mainstream accuracy for ImageNet under the mobile setting should be 75% top-1. And the SOTA top-1 accuracy on ImageNet is around 85.5%. Table.1 and table.2 do somewhat show the effectiveness of your method, but its significance is limited especially considering the limited novelty of methodology.
Minors:
In Fig.3 k = 6, it seems vanilla splitting is better than accuracy and parameters, except for 0.2 log higher flops. I don't think this makes a compelling case here.
In Fig.4, from flops 7 ~ 9, your results are similar to Bn especially accuracy > 0.6. When accuracy < 0.6, it is less interesting, and I believe the improvement should be huge.
Fig.5, could you please compare the time using MAGMA from NVIDIA? I had some experiences in implementing the LAPACK and BLAS on GPUs, and it should not be that slow.
Overall, this paper has some interesting results, which shows the eigenvalue can be approximated by Rayleigh-Quotient gradient descent, and show positive improvement. However, the methodological and experimental results can definitely be strengthened. The author may consider proving that achieving the global optimum on Eq.(3) really matters, then motivate the methodology. Thank you.


We present a novel method for photo-realistic re-rendering of reconstructed objects. The digital reproduction of object appearances is of paramount importance nowadays. Augmented and virtual reality relies on such 3D content. It enables virtual showrooms, virtual tours & sightseeing, the digital inspection of historical artifacts and many other applications. Classical approaches use methods to reconstruct the geometry of an object and textures to capture the appearance properties. Instead, we propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object. As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on "remembering" object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data. Results are best seen in the supplemental video: https://www.dropbox. com/sh/8pi37qlfqoukthd/AAB0sdQT79noBRtKBKWCywtJa?dl=0.

Authors propose a novel neural image-guided rendering method, a hybrid between classical image-based rendering and machine learning. They learn EffectsNet in a self-supervised manner to capture the view-dependent component using a reprojection loss. To render a novel view they subtract a view-dependent component from K closest train views, reproject the diffuse component to the novel view, add view-dependent component, use a CompositionNet to render a final image. They use both synthetic and real data for the experiments and compare to state-of-the-art image-based rendering methods of Hedman et al.
I think the method discussed in the paper is a nice contribution to the neural rendering area. The paper is written clearly and the provided amount of technical details is enough to reproduce the method.
I only have some concerns about the experiments.
1) The method relies on the depth maps and the main contribution of the paper is related to the rendering of view-dependent effects. However, in realistic scenarios, the presence of the view-dependent effects leads to severe degradation of photogrammetry pipeline: imprecise camera poses and geometry. To my understating, imprecise geometry and camera poses should affect the method heavily. I think, the paper lacks a thorough evaluation on more realistic data, having only a single example in Fig. 14.
2) Authors compare to DeepBlending of Hedman et al. and although the proposed model achieves the best quantitative result, to my mind, the qualitative difference is not very noticeable, while it is known that MSE error does not correlate with human perception good enough. To justify the presented method it would be great to see experiments with a more clear difference to Hedman et al. baseline.
3) Authors of Pix2Pix (Isola et al., 2016) do not provide experimental results on novel view-point synthesis. Although it is completely fine to refer to Pix2Pix as to the closest and simplest neural-based baseline I do not find it convincing to compare the proposed method to Pix2Pix without any other baselines as in the figures 5, 11, 12. Pix2pix had never been claimed to be developed or validated for this particular problem so it is a kind of "artificial" baseline. Moreover, I believe a stronger "artificial" baseline can be easily constructed. In particular, it is hard to believe that for the "Vase" scene, which has almost no view-dependent effects, a network that takes a world space position map cannot learn a direct mapping from (x,y,z) to (r, g, b). That is why, I believe, a stronger "artificial" baseline could be easily constructed by some kind of tweaking of Pix2Pix, say, turning off GAN or changing neural networks' architecture slightly.
Considering the issues mentioned above my I tend to recommend this paper for rejection.
Questions:
1) Video shows examples of novel view synthesis for a predefined camera trajectory. How this trajectory is created for both synthetic and real objects? How does it differ from the train trajectory?
Some minor comments:
1) In Fig. 2 it reads "Input color", whereas the RGB color image is not used as input to EffectsNet. It confuses on the first read. In "... based on the current view and the respective depth map", it is also easy to think that "current view" refers to RGB image.
2) The model from Figure 8 is never used.


A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).

Review:
This paper considers the problem of overfitting in RL through a specific model of overfitting, namely one where noise is introduced in the observation space, independently of the controllable dynamics.  The paper provides both theoretical and empirical insights into the various manifestations of overfitting in this class of domains.
A strength of this paper is to deepen our understanding of the phenomenon of overfitting in RL.  To get a deep understanding of hard problems it is worthwhile to look at sub-classes of problems, and as such the identification of the observational overfitting setting is interesting.  The paper provides insightful findings such as:
-	Explicit separation of f and g_theta.
-	Theoretical properties of generalization for the specific case of LQR and linear policies.
There are also several empirical results, on three contrasting domains which illustrates various aspects of the interaction between parameterization vs generalization.
But overall, though I read the paper in detail, and am knowledgeable of the topic, I am still struggling with extracting very specific new findings from this work.  Here are a few examples:
p.6:  “We observe empirically that the underlying state dynamics has a significant effect on generalization performance as the policy nontrivially increased test performance” ->  I see where you draw this analysis from the results.  But is this finding new or surprising?  I would have been very comfortable saying (without your results) that the underlying state dynamics can have a significant effect on generalization performance.
p.6: “This suggests that the Rademacher complexity and the weight perturbation bound for rewards vary highly for different environments.”  -> Again, how is this new knowledge?
p.7: “this also suggests that the RL generalization quality of a convolutional architecture is not limited to real world data” -> Same question, what new knowledge have we gained?
Perhaps the key message is that “there are generalization benefits to overparameterization” and that “implicit generalization” plays a key role in controlling this?   If that is the main message, than it is somewhat obfuscated by all the material on LQR, which doesn’t really go in this direction (p.4: “We being with a theorem which implies that a high dimensional observational space directly contributes to overfitting”).  I must say I found most of the material in 3.1 to distrct from the main message – but perhaps it is because I am not very interested in the LQR setting, and did not understand how the findings there supported those in latter sections.
Overall, I would say the paper suffers from some clarity issues.  There are some minor typos, then some key terms of are not sufficiently explained/defined (e.g. g_\theta in Sec.2.2 is said to project “the latent data to unimportant features”, but then there is discussion of “in settings where $g$ does matter” – I’m confused.)   It’s not clear whether Thm 3.1 is new, or adaptation of existing results.  More broadly, I found Sec.3.1 difficult to follow. The last paragraph of Sec.4 seems superfluous.   I would recommend some good editing throughout.
Another concern with the work is the fact that several aspects of RL are ignored (e.g. exploration, entropy, \gamma, noise, stochastic gradients, etc. – as per p.4), yet are known to have an effect on overfitting, perhaps much more than the depth and width of the neural network.  If that’s the case, it is perhaps dangerous to ignore them in the analysis; it is not described in the paper, for the domains studied in the empirical results, whether this is the case or not.  As a result, it’s not clear how far the current analysis carries.  I’m also not clear on the analysis at bottom p.7 “IMPALA-LARGE (…) memorizes less than IMPA due its inherent inductive bias” -> Seems to me IMPALA-LARGE might simply be in underfitting regime.
======
Update post-rebuttal:
Thank you for the thoughtful responses.   Given the other strong reviews, and your careful characterization of various aspects of the work, I am favorable to accepting.  Please try to incorporate elements of this discussion into the paper, which I think will give a richer and more nuanced understanding of the work to future readers.


Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations to more complex domains and practical applications, it is important to enable hyperparameter tuning and model selection of existing unsupervised approaches without requiring access to ground truth attribute labels, which are not available for most datasets. This paper addresses this problem by introducing a simple yet robust and reliable method for unsupervised disentangled model selection. Our approach, Unsupervised Disentanglement Ranking (UDR), leverages the recent theoretical results that explain why variational autoencoders disentangle (Rolinek et al., 2019), to quantify the quality of disentanglement by performing pairwise comparisons between trained model representations. We show that our approach performs comparably to the existing supervised alternatives across 5400 models from six state of the art unsupervised disentangled representation learning model classes. Furthermore, we show that the ranking produced by our approach correlates well with the final task performance on two different domains.

This paper addresses the problem of unsupervised model selection for disentangled representation learning. Based on the understanding of “why VAEs disentangle” [Burgess et al. 2017, Locatello et al. 2018, Mathieu et al. 2019, Rolinek et al. 2019], the authors adopt the assumption that disentangled representations are all alike (up to permutation and sign inverse) while entangled representations are different, and propose UDR method and its variants. Experimental results clearly show that UDR is a good approach for hyperparameter/model selection.
Overall, I think a reliable metric for model selection/evaluation is needed for the VAE-based disentangled representation learning. According to comprehensive experimental studies performed in this paper, UDR seems to be a potentially good choice.
However, I am not sure if very good disentangled representations must benefit (general) subsequent tasks, though the authors provide experimental evidence on fairness classification and data efficiency tasks. Actually, the data generation process in the real-world may consist of different generative factors that are not independent of each other. Though good disentangled representation provides good interpretability, it needs not to be better than entangled representation for concrete tasks. Specifically, for concrete supervised classification tasks, VAE with beta smaller than 1 (not towards disentanglement) might be the best (Alexander A. Alemi et al. 2017, Deep VIB).
Another concern is about the choice of some key “hyperparameters”.
For the KL divergence threshold in equation 3, you set it to be 0.01. It looks like the choice would control how much the UDR favors a “sparse representation map”. The larger the value, the few “informative dimensions” would be considered.
In supplementary material, you say that “uninformative latents typically have KL<<0.01 while informative latents have KL >>0.01”. Is this judgment based on “qualitative feeling”? For me, as you are contributing a ``quantitative measurement”, it is interesting and important to see how this threshold would generally affect UDR’s behavior in one (or more) datasets you have tried.
Another hyperparameter I cared is P (number of models for pairwise comparison). In the paper, you validate the effect of P in the range [5,45]. How would P smaller than 5 affect UDR? According to Table 1, if I was using UDR, I’d rather using P>=20 (or at least 10) rather than 5.
Also, it seems to me P would grow up due to the size of factors that generate the data. Thus, I also have a little concern about the computation cost of the proposed metric (as also mentioned by the authors).
Others concerns:
-- As a heavy experimental paper, most experimental results are in supplementary material, while the authors spent a lot of time in the main text explaining the conclusions found in other papers.
-- To validate the fundamental assumption of UDR, the authors might consider to quantitatively validate that, disentangled representations learned by those approaches you used in the paper are almost the same (up to permutation and sign inverse).


A patient's health information is generally fragmented across silos. Though it is technically feasible to unite data for analysis in a manner that underpins a rapid learning healthcare system, privacy concerns and regulatory barriers limit data centralization. Machine learning can be conducted in a federated manner on patient datasets with the same set of variables, but separated across sites of care. But federated learning cannot handle the situation where different data types for a given patient are separated vertically across different organizations. We call methods that enable machine learning model training on data separated by two or more degrees "confederated machine learning." We built and evaluated a confederated machine learning model to stratify the risk of accidental falls among the elderly.

This paper considers the problem of learning from medical data that is separated both horiontally (across different practices and centers) and vertically (by data type). The contribution is a "confederated" machine learning method that learns across these divides. The particular application considered here as means of illustration is that of fall prediction in the elderly. Specifically the authors investigate an ML approach to risk-stratifying elderly patients with respect their likelihood of falling in the next two years.
The basic challenge addressed here is learning in the setting in which different data elements are available only at specific sites, and it is assumed that they do not share data. In addition, it is assumed there are multiple distinct sites that have data corresponding to the respective elementes. However, it is assumed that labels (target vectors) are shared across all sites. This setup is simulated using available data. A simple distributed training scheme is outlined.
This is a potentially important problem worthy of study. I some major concerns with the present work, however. First, I do not think that ICLR is really the best venue for this work. The machine learning component of this is quite straightforward; basically SGD is performed iteratively on parameters associated with the data types "owned" by the respective (simulated) sites. Updates are then averaged over these parameter subsets. This is perfectly reasonable, but not terribly novel. The presentation of this is also much longer than it needs to be for the ICLR audience. I think this paper, in its current form, would be better suited for an audience more interested in clinical applications specifically (and I say this as someone quite appreciative of work on applied ML; it's just that the audience here will be more interested in methodological innovations.)
With respect to clinical utility: Do we really need ML to tell us about risk of falls? I mean, if we were to ask the MD who had seen these patients to perform a simple stratification (perhaps on an ordinal scale), would they not likely be able to do so reasonably well? The authors mention something like this, discussing the 'clinical screening process' which involves asking about prior falls. This seems like a really strong baseline. The authors argue that this is time-consuming,
In any case, is AUCPR an appropriate or useful metric here? In practice one would need to pick a threshold on which to act; perhaps a simulation that investigated doing so would provide a more meaningful evaluation. Although again a strong baseline here would probably be to ask physicians to risk stratify patients for interventions direclty (I appreciate that this would be a non-trivial experiment to run, but still).
I also have a question regarding the simulation. I *think* the authors have randomly assigned patients to the respective simulated sites; is that right? This seems problematic because in practice patients would not be IID distributed in this way; sites would have their own patient populations which would affect the losses. This should be somehow taken into account in the simulation.
Other comments
---
- I think I am missing something in the notation here. $Y_{si}$ is a 'binary label' but seems to vary across 'states' for an individual, is that right? Shouldn't this be constant for an individual? The paper states below that "The output of the classifier is a binary variable indicating whether the beneficiary had a fall during the follow up period."
- Labels were derived from ICD codes; was there any effort to spot check these? I am always a bit concerned about deriving labels from ICD and trusting them.
- As far as I understand from 2.1, the authors have not included features extracted from notes in the patient history; is that right? Why not?
Smaller issues
---
- I would strongly suggest numbering your equations. Also, suggest using \text while in mathmode for superscripts like `diag'.
- "Step 1.." --> "Step 1." (p3)"
- "The parameter Θof f is randomly initialized" --> missing space before "of"
- On page 4: L(Xdiag, Xmed, Xlab, Θ) is written incorrectly.
- page 4: "Tstands"  missing space.
- page 5: " fallst." --> "falls."
- Appendix Tables 2 and 3 both contain the typo "varaible" (should be "variable")
- In Appendex Table 2, I suggest reporting results with a consistent amount of precision, e.g., 0.002 --> 0.0020 here.


Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. However, catastrophic forgetting poses a grand challenge for neural networks performing such learning process. Thus, neural networks that are deployed in the real world often struggle in scenarios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.e., rare edge cases. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a rapid learning plastic component (compressed episodic memory) to the fixed (slow changing) parameters of the softmax output layer; enabling learned representations to be retained for a longer timescale. We demonstrate the flexibility of our method by integrating well-known task-specific synaptic consolidation methods to penalize changes in the slow weights that are important for each target task. We evaluate our approach on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmarks, and introduce an imbalanced variant of Permuted MNIST -a dataset that combines the challenges of class imbalance and concept drift. Our proposed model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.

The authors introduce DIFFERENTIABLE HEBBIAN CONSOLIDATION,a new framework for continual learning that can be implemented in the usual differentiable programming setups. This framework is motivated in terms of complementary learning system (CLS) theory which features an episodic memory module. The method is shown to be easily implemented as seen in their pytorch pseudocode (authors also suggest code will be released). Additionally, authors show the method leads to significant  improvements over simple baselines, and can complement other task-specific hebbian-based learning paradigms
As an alien to the continual learning literature, my judgment is mostly positive. First, the paper is well-written, and the method is well-motivated in terms of cognitive principles with a nice commentary about underlying neurobiological substrates. The description of alternative methods seems exhaustive (but see below). The resulting algorithm is simple and doesn't seem to entail a significant computational burden. Importantly, it is differentially programing-friendly. Results show improvements over a baseline. However, I have two significant criticisms regarding the experimental section that I hope the authors will address.
1)Authors comment on two paradigms to hebbian-based continual learning: the task-specific and theirs, based on CLS. Authors show their method complement improvements due to task-specific approaches (which is great), but proper comparisons to other CLS-based approaches seem missing. In other words, I would hope a much better statement than "
However, all of these methods were focused on rapid learning on simple tasks or metalearning over a distribution of tasks or datasets. Furthermore, they did not examine learning a large
number of new tasks while, alleviating catastrophic forgetting in continual learning" Perhaps the authors may try find experimental cases where comparison to such approaches are possible, and show the empirical results? I am concerned the finetuned MLP baseline could be a bit weak.
2)The authors should try more experiments. They showed many MNIST variants and a 5-vision dataset. However, results in other papers in the literature are shown in much more sophisticated contexts (e.g. learning to play games). I understand this could exceed the computational capabilities of authors, but  why, for example, not to try on a different dataset, as CIFAR? (as the Zenke et al paper did).
MInor: On section 3, page 4 authors mention the number of labels d but that number appears undefined


Coreference resolution is an important task for gaining more complete understanding about texts by artificial intelligence.The state-of-the-art end-to-end neural coreference model considers all spans in a document as potential mentions and learns to link an antecedent with each possible mention. However, for the verbatim same mentions, the model tends to get similar or even identical representations based on the features, and this leads to wrongful predictions. In this paper, we propose to improve the end-to-end system by building an attention model to reweigh features around different contexts. The proposed model substantially outperforms the state-of-the-art on the English dataset of the CoNLL 2012 Shared Task with 73.45% F1 score on development data and 72.84% F1 score on test data.

This paper extends the neural coreference resolution model in Lee et al. (2018) by 1) introducing an additional mention-level feature (grammatical numbers), and 2) letting the mention/pair scoring functions attend over multiple mention-level features. The proposed model achieves marginal improvement (0.2 avg. F1 points) over Lee et al., 2018, on the CoNLL 2012 English test set.
I recommend rejection for this paper due to the following reasons:
- The technical contribution is very incremental (introducing one more features, and adding an attention layer over the feature vectors).
- The experiment results aren't strong enough. And the experiments are done on only one dataset.
- I am not convinced that adding the grammatical numbers features and the attention mechanism makes the model more context-aware.
Other suggestions:
- The citation format seems weird through out the paper.
- Table 1 and 3 look like ablation results. It might be less confusing if it's presented as "Full system: xx%; without pairwise FA: yy%; without grammatical numbers zz% ...".
- Equation 8 - 10 are quite confusing. What is f(x)? How large is V? What is u? etc.
- Please define/explain the "grammatical numbers" feature when it's introduced in Section 2.2.


In this paper we present a deep Multiple Instance Learning (MIL) method that can be trained end-to-end to perform classification from weak supervision. Our MIL method is implemented as a two stream neural network, specialized in tasks of instance classification and weighting. Our instance weighting stream makes use of Gaussian radial basis function to normalize the instance weights by comparing instances locally within the bag and globally across bags. The final classification score of the bag is an aggregate of all instance classification scores. The instance representation is shared by both instance classification and weighting streams. The Gaussian instance weighting allows us to regularize the representation learning of instances such that all positive instances to be closer to each other w.r.t. the instance weighting function. We evaluate our method on five standard MIL datasets and show that our method outperforms other MIL methods. We also evaluate our model on two datasets where all models are trained end-to-end. Our method obtain better bag-classification and instance classification results on these datasets. We conduct extensive experiments to investigate the robustness of the proposed model and obtain interesting insights.

This paper described an approach of performing multiple instance learning (MIL) by using a network branch to weight instances and then using a Gaussian normalization layer on top of it, where the weightings are predicted based on in-bag variances. The instance weighting scheme, a classic in MIL, has been proposed in deep networks by [3]. Hence, I don't see much novelty here except that there is a Gaussian normalization layer after the instance weighting in the MIL framework.
I'm a bit worried that sigma seems to be unnormalized in the first case -- what would happen if the bag score distribution is non-Gaussian? GP1T1 seems more sensible by learning all these weights.
However, I see significant issues in terms of evaluation which makes hard to accept this paper.
I firmly believe that 22 years after the (Dietterich 1997) paper, it's no longer enough to only use the original MIL datasets and classification datasets such as the CIFAR-10 bags to evaluate MIL. I may be alone here which is open to discussions, but the original motivation for MIL is for the problem to be weakly-supervised where we only know a high-level label but no low-level labels. There exist many realistic image problems that are similar to this (e.g. weakly-supervised detection and semantic segmentation) and have received a plethora of work, hence it's unclear to me whether still using these arbitrarily generated CIFAR-10 bags and the 5 old datasets would still apply to MIL as an approach. After all, MIL is almost dying and "weakly-supervised learning" has risen in popularity with almost being the same problem as MIL. I think for MIL to work its way back, it should first start by using the right datasets to test (e.g. the datasets in [3] would be a great starting point) and comparing with other weakly-supervised detection approaches that do not use additional information.
Besides the philosophical point, a practical issue is that the numbers seem to be bold arbitrarily according to the authors' whim. In table 3, MI-Net DS is better than every other method in the last 2 columns, but not bold, and also it seems that no t-test was performed to determine the significance of differences. This also happens in Table 1, where I think WSDN should be equivalent with the proposed method in the last 2 rows.
Finally, the choice of excluding MI-Net in the CIFAR experiments is dubious as well, given its performance in the simple datasets. Why is MI-Net not tested on the CIFAR experiments?


Adversarial example generation becomes a viable method for evaluating the robustness of a machine learning model. In this paper, we consider hard-label blackbox attacks (a.k.a. decision-based attacks), which is a challenging setting that generates adversarial examples based on only a series of black-box hard-label queries. This type of attacks can be used to attack discrete and complex models, such as Gradient Boosting Decision Tree (GBDT) and detection-based defense models. Existing decision-based attacks based on iterative local updates often get stuck in a local minimum and fail to generate the optimal adversarial example with the smallest perturbation. To remedy this issue, we propose an efficient meta algorithm called BOSH-attack, which tremendously improves existing algorithms through Bayesian Optimization (BO) and Successive Halving (SH). In particular, instead of traversing a single solution path when searching an adversarial example, we maintain a pool of solution paths to explore important regions. We show empirically that the proposed algorithm converges to a better solution than existing approaches, while the query count is smaller than applying multiple random initializations by a factor of 10.

This paper proposes a meta-algorithm for the so-called "decision-based attack" problem, where a model that can be accessed only via label queries for a given input is attacked by a minimal perturbation to the input that changes the predicted label. The algorithm, BOSH, augments any iterative algorithm for this problem with a diversification strategy based on bayesian optimization and throwing away bad solutions. Empirically, it is shown that BOSH can improve the performance of recently developed algorithms for this problem, by exploring more solutions and refining them intelligently.
Overall, the decision-based attack problem is very practically relevant as it assumes minimal access to the classifier. I also really like that the authors looked into tree-based models in addition to neural networks. The algorithmic ideas that are proposed are simple and effective, as supported by the experimental results.
However, I have some serious comments about the experimental evaluation that I believe can substantially improve the quality of the paper, if addressed. Whether I raise my score or not will depend on how well the authors address these questions. I also have concerns about related work in heuristic algorithms.
Questions:
- Related work: the ideas of diversifying solution paths and throwing away bad solutions are very popular in combinatorial heuristics. You should do a thorough review of work in that area and in Genetic Algorithms (GA). You could look into the following classical/survey papers as starting points, in particular the first survey's chapter 4.
Glover, Fred, and Manuel Laguna. "Tabu search." Handbook of combinatorial optimization. Springer, Boston, MA, 1998. 2093-2229.
Feo, Thomas A., and Mauricio GC Resende. "Greedy randomized adaptive search procedures." Journal of global optimization 6.2 (1995): 109-133.
- Other Black-Box algorithms: you should compare against well-established black-box optimization algorithms such as NOMAD and RBF-OPT. Both are based on very solid mathematical foundations and have high-quality open source implementations:
NOMAD: https://www.gerad.ca/nomad/
rbf-opt: https://github.com/coin-or/rbfopt
- Runtime comparison: your analysis with respect to number of queries is very good and insightful. In addition, we should get a sense of the runtime performance. If you run each of the approaches with the same time limit, how do they fare?
- Comparing to "optimal" attacks: we need to know how well the solutions are compared to the best possible, or a close-enough approximation. You could run white-box attacks and compare the relative error to the quality of the white-box attack. Otherwise, it is hard to tell what gap remains to be closed algorithmically and it is difficult for other researchers to know whether it's worth trying to improve what you propose here in the future.
- Time complexity: please give a time complexity analysis of BOSH as a function of all its hyperparameters.
Clarity:
- Figure 1: I don't understand what this figure shows. Where are the 2 minima? Please clarify further.
- You minimize l(.) / g(.) in Algorithm 2, but maximize it in Appendix B.2. Maximizing makes more sense. Which one is it?
- Theorem 1: Is that your result or Bergstra et al.'s?
Minor comments:
- "Adversarial example generation becomes a viable method for evaluating the robustness of a machine learning model." --> "Adversarial example generation has become a viable method for evaluating the robustness of a machine learning model."
- "when searching an adversarial" --> "when searching for an adversarial"
- "Distortion" is used in the literature much less than "Perturbation"; consider switching them.
- "our mega algorithm" --> "our meta-algorithm"
- Please use consistent notation: SignOPT or Sign-OPT.
- Appendix B.1: "undifferentiable" --> "non-differentiable"
- "are the t − x1 samples" --> "are the t − 1 samples"


The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks. However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive. In this work, we develop BlockSwap: a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential. These networks can then be used as students and distilled with the original large network as a teacher. We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classification, and COCO for detection, and provide a comprehensive ablation study of our approach. BlockSwap quickly explores possible block configurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. 8 minutes on a single CPU for CIFAR-10).

This paper introduces an approach to compressing a deep network so as to satisfy a given budget. In contrast to methods that replace all convolutional blocks with the same cheaper alternative, the authors argue that mixing different types of such cheap alternatives should be effective. To achieve this in a fast manner, they propose to randomly sample architectures with mixed blocktypes that satisfy the given budget and rank these architectures using Fisher information.
Originality:
- The approach is simple, but seems effective. Although the different components that make up the final method are known, they are put together so as to address a different task, in a way that I find interesting.
Methodology:
- From the paper, it is not entirely clear how sampling under budget constraints is achieved. I imagine that one could do it in a naive way, by choosing the first block uniformly randomly, and then removing the candidates that do not fit the budget for the subsequent blocks. This, however, would seem to give a different importance to the early blocks than to the later ones, since the former would essentially not have any budget constraints. I would appreciate it is the authors could explain their strategy in more detail.
- The assumption that the number of blocks in the student is the same as in the teacher seems constraining. A better architecture might be obtained by having fewer, wider blocks, or more, thinner blocks. Do the authors see a potential way to address this limitation?
- In principle, it seems that the proposed Fisher Information-based strategy could be used for general NAS, not just for compact architectures. Have the authors investigated this direction?
Related work:
- It seems to me that the literature review on compression/pruning is a bit shallow. I acknowledge, however, that most works do not tackle the scenario where a budget is given. However, Chen et al., "Constraint-aware Deep Neural Network Compression", ECCV 2018, do, and it would be interesting to discuss and provide comparisons with this work.
Experiments:
- I appreciate the ablation study, which answered several of my questions.
- In Table 1, it seems counterintuitive that the (negative) correlation becomes smaller as the number of mini-batches increases. Do the authors have an explanation for this?
- In Section 5, are the baseline compact networks all trained using the same Attention Transfer algorithm as for the BlockSwap ones?
- In Table 2, the budget values (P. (K)) seem fairly arbitrary? How were they obtained? They seem to match those of SNIP. Is this because the authors ran SNIP, and then set their budget accordingly?
- Below Fig. 3, the authors mention that they are generous with sparsity-based methods because they count the number of non-zero parameters. Note that several structured-sparsity compression methods have been proposed (Alvarez & Salzmann, NIPS 2016, 2017; Wen et al., NIPS 2016), which, by contrast with regular sparsity ones, would cancel out entire filters.
Summary:
Overall, I like the simple idea proposed in this paper. I would however appreciate it if the authors could clarify the way they sample the architectures and address my questions about their experiments.


Reinforcement learning (RL) has become an increasingly active area of research in recent years. Although there are many algorithms that allow an agent to solve tasks efficiently, they often ignore the possibility that prior experience related to the task at hand might be available. For many practical applications, it might be unfeasible for an agent to learn how to solve a task from scratch, given that it is generally a computationally expensive process; however, prior experience could be leveraged to make these problems tractable in practice. In this paper, we propose a framework for exploiting existing experience by learning reusable options. We show that after an agent learns policies for solving a small number of problems, we are able to use the trajectories generated from those policies to learn reusable options that allow an agent to quickly learn how to solve novel and related problems.

Summary:
The authors propose to learn reusable options to make use of prior information and claim to do so with minimal information from the user (such as # of options needed to solve the task, which options etc). The claim is that the agent is first able to learn a near-optimal policy for a small # of problems and then is able to solve a large # of tasks by such a learned policy.  The authors build on the idea that minimizing the number of decisions made by the agent results in discovering reusable options. The options are learned offline by learning to solve a small number of tasks. Their algorithm introduces one option at a time until introducing a new option doesn’t improve the objective further. The ideas are interesting, However, the paper as it stands is lacking in thorough evaluation.
Detailed comments:
The proposed approach offers two key contributions:
-an objective function to minimize the decision states
-incrementally constructing an option set that can be reused later, without the a priori specification of the # of options needed.
The introduction is well written, however, given the intuitions behind the objective function; in some sense, the idea here is to minimize the decisions or terminations intuitively relates to terminating only at critical or bottleneck states. It would be useful to provide such motivation in the introduction.
Intuitively the objective criterion is interesting. With a cursory look at the proofs, they seem fine, although I have to admit I have not looked in detail into the proofs.
Paper writing could be significantly improved. Several points are not clear and need further clarification:
-The term near-optimal is mentioned several times, but it is not clear the policies are near-optimal with respect to what? The task or a set of tasks?
-How does the proposed approach ensures that they are near-optimal? Please clarify.
-“We can obtain the estimate for equation 1 by averaging over a set of near-optimal trajectories” The aim as states is to learn options that are capable of generating near-optimal trajectories (by using a small # of terminations). The authors then say that “given a set of options, a policy over options, a near-optimal sample trajectory, we can calculate..” Where does the near-optimal sample trajectory come from? Please provide clarifications.
In experiments: FR rooms experiments are interesting, in the visualization of the option policies, do the figures here show the flattened policy of the options? What do the termination functions look like?
Atari experiments are limited in nature in that they show only two games. Moreover, It is a bit confusing as to what is multi-task in the ATARI experiments. The authors mention the training of options and then talk about the results in the plots (4) show the training curves. However, they do not mention what are “novel tasks for Breakout/Amidar” in this context.
Considering the proposed approach is closely related to the idea of selective terminations of options, it is natural to expect a comparison with Harb, 2018 and Hartyuanm 2019. The work could benefit by comparing with the aforementioned baselines. In particular, the visualization in 4b showing options learned in Amidar does not show much improvement from what was observed before in Harb, 2018.
With the motivation of this paper, I am unable to convince myself about options being “reusable” for multi-task here. It would be very useful for the reader to clarify what “novel tasks” are here to appreciate what is learned. Looking deeper into the appendix, I understand that the authors “first learned a good performing policy with A3C for each game and sample 12 trajectories for training.” This is not at all clear in the main paper. Besides, what does it mean by a "good" policy? If we already have that, it is unclear what gains do we get from the proposed method.
One obvious limitation here is that they also have a hard imposed constraint here is that the options cannot run for more than 20 time-steps in total, to make the objective function a suitable choice.
Overall:
An interesting objective function, Learn not only option set but also the number of options needed and incrementally learn new options.
Paper writing does not convey clearly what are novel tasks and could be significantly improved.
Since the paper claims multi-task and mentions several lifelong learning works like [1], I was expecting rigorous baselines showing performance over multiple tasks. The experiments are lacking in that evidence except for four rooms domain, which is much simpler a domain.
Near-optimal property is very much lacking the clarity to the best of my knowledge.
[1]Ammar, Haitham Bou, et al. "Online multi-task learning for policy gradient methods." International Conference on Machine Learning. 2014.


To simultaneously capture syntax and global semantics from a text corpus, we propose a new larger-context recurrent neural network (RNN) based language model, which extracts recurrent hierarchical semantic structure via a dynamic deep topic model to guide natural language generation. Moving beyond a conventional RNN based language model that ignores long-range word dependencies and sentence order, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence topic dependences. For inference, we develop a hybrid of stochastic-gradient MCMC and recurrent autoencoding variational Bayes. Experimental results on a variety of real-world text corpora demonstrate that the proposed model not only outperforms state-ofthe-art larger-context RNN-based language models, but also learns interpretable recurrent multilayer topics and generates diverse sentences and paragraphs that are syntactically correct and semantically coherent.

[Additional review]
This paper proposes a technique to incorporate document-level topic model information into language models.
While the underlying idea is interesting, my biggest issue is with the misleading assertions at the very beginning of the paper. In the second paragraph of Section 1, the paper claims that RNN-based LMs often make independence assumptions between sentences, hence why they develop a topic modelling approach to model document-level information. Some issues with this claim:
1. Pretty much every LM paper that evaluates on language modelling benchmark (PTB, WT-103, Wikitext-2) uses LSTMs/Transformers incorporate cross-sentential, document-level information as context, through a very simple approach of just concatenating all the sentences and adding a unique token to mark sentence boundaries.
2. Prior work has shown that LSTMs/Transformers with cross-sentential context can, and in fact do, make use of information from previous sentences.
a. Evidence 1: Khandelwal et al. (2018) showed that LSTMs memorise word orders from the past ~50 tokens, and retain semantic information from the past ~200 tokens; both of which extend far beyond the length of an average sentence, suggesting that information from the previous sentences is used in the predictions of the current sentence.
b. Evidence 2: Language models that operate on single sentences typically do worse than language models that take into account cross-sentential context, e.g. the language model of Kim et al. (2019) that operates on single sentences gets ~90 ppl. on PTB test set, while LSTMs that condition on multiple sentences get a much better ~50-something ppl. on Mikolov PTB.
Crucially, these prior works defeat the paper’s motivation of why it claims to need topic models in the first place (i.e. to model cross-sentential context), while just concatenating multiple sentences as context would do, and in fact has been done many times.
2. Prior work (mostly in Transformer-land) has come up with ways to make use of very long-range context, from Transformer-XL to the more recent compressive Transformer (https://openreview.net/forum?id=SylKikSYDH) that can condition on entire books. While these are done for Transformers, in principle one can also apply similar techniques to LSTMs.
3. While Transformer-XL has the potential to make use of word orders in the preceding sentences, it seems that this paper’s approach cannot do that, since they only take the bag-of-words from the preceding sentences. It thus seems that their bag-of-word approach is less expressive, and hence less powerful, than the simpler alternative of concatenating sentences.
4. The perplexity results (Table 1) are not done on very standard datasets (no PTB evaluation for instance). It is thus hard to evaluate the strength of the baseline models. In the paper's defense, it seems that they were following the experimental setup of Wang et al. (2019), but the paper should elaborate more on the choice of evaluation datasets.
5. The inference part is not particularly self-contained. The paper simply refers the TLASGR-MCMC method (which is an important part to make inference scalable) to prior work (Cong et al., 2017; Zhang et al., 2018), yet does not explain (even briefly) how the approach works, and how it can be combined with their recurrent topic model formulation.
6. Evaluation of the induced topic hierarchy (Figure 4) is only done through qualitative samples, and the paper does not really explain how to pick the samples (i.e. possible cherry-picking). I am not very familiar with the topic modelling literature, but it would be nice if the induced hierarchy can be evaluated quantitatively.
References:
1. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away. In Proc. of ACL 2018.
2. Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. Unsupervised recurrent neural network grammars. In Proc. of NAACL 2019.
3. Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational autoencoders for text generation. In Proc. of NAACL 2019.
4. Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian MCMC. In Proc. of ICML 2017
5. Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In Proc. of ICLR 2018


Noise injection is a fundamental tool for data augmentation, and yet there is no widely accepted procedure to incorporate it with learning frameworks. This study analyzes the effects of adding or applying different noise models of varying magnitudes to Convolutional Neural Network (CNN) architectures. Noise models that are distributed with different density functions are given common magnitude levels via Structural Similarity (SSIM) metric in order to create an appropriate ground for comparison. The basic results are conforming with the most of the common notions in machine learning, and also introduces some novel heuristics and recommendations on noise injection. The new approaches will provide better understanding on optimal learning procedures for image classification.

This paper studies the different types of noises that could be added to the training image dataset while training an CNN model for classification. They study 5 different types of noise functions: Gaussian, Speckle, Salt and Pepper, Poisson, Occlusion.

Pros:
1. Rigorously studying how to augment training data for CNN is important.
Cons:
1. The primary question is novelty . - what is the research contribution of this dataset? They are running experiments of 5 different known noise functions, on two image datasets, on a single deep learning models. There are no fundamental research questions or hypothesis. This is a mere running of few experiments - known methods and known approaches.
2. Are the results generalizable? Answer is no! The results on shown on two subsamples of Imagenet datasets for only ResNet 18 model. Maybe for this combination speckle noise (and not Gaussian, as pointed out in the comments by the authors) is better. How can the assure that for a different dataset, model, task combination the same speckle noise would perform better ?
3. Improvement suggestion: What I would ideally look in this topic, is a method to automatically study the properties of the training data images (study the distribution) and conditional on this distribution recommend the best noise type and noise intensity. Thus, the whole story of noise injection could be made dynamic for a dataset, model, task combination
4. Writing of the paper could be improved: 1. The need for noise based augmentation of is well known (Section 1). 2. The different kinds of noise functions are mostly text book knowledge (Section 2). 3. The different image quality metrics written here - MSE, PSNR, SSIM are also text book knowledge (Section 3). Overall, the first 4 pages of the paper are redundant and could be compressed into 1 page. Would like to read more on the experiments, analysis, and maybe automation of noise selection techniques in different kinds of tasks - segmentation, text classification, seq2seq etc.
5. Choose very naive noise (or augmentation) functions: In general, the approach of augmentation of training data has evolved so much in the literature, that adding noise is hardly in practice.
1. "The Effectiveness of Data Augmentation in Image Classification using Deep Learning" - Style Transformation
2. "Improving Deep Learning using Generic Data Augmentation" - Geometric and Affine Transformation
Thus, the study on data augmentation should be performed across all these different transformation functions on training data and using only noise function is naive and is incomplete.


We consider the problem of knowledge transfer when an agent is facing a series of Reinforcement Learning (RL) tasks. We introduce a novel metric between Markov Decision Processes and establish that close MDPs have close optimal value functions, that is that optimal value functions are Lipschitz continuous with respect to tasks. These theoretical results lead us to a value transfer method for Lifelong RL, which we use to build a PAC-MDP algorithm with improved convergence rate. We illustrate the benefits of the method in Lifelong RL experiments.

Updated review: Thanks to the authors for response. I think it clarifies a few points for me except one, which is that I still dont understand how the ATARI games share the same state space? Under what model is this true? And if the paper really wants to use this as an example, why are there no results on even a single ATARI game.
------------------------------------------------------------------------------------------------------------------------------------------------------------
Summary: The paper presents a method for lifelong reinforcement learning problems. By lifelong reinforcement learning the paper means that a set of tasks, specified by MDPs, specified by reward and transition function (R and T) are presented to the agent sequentially and the agent must solve them. The idea of the paper is to assume that the tasks that live in the space <R, T> are Lipschitz continuous and if that is so the experience from the previous task can be used to upper bound the Q values for the current task. The paper suggests using the RMax algorithm that can take as input an upper bound (optimistic initialization) on the Q values of the MDP it is trying to solve and solve the MDP efficiently. For the proposed algorithm called LRMax, the paper presents theoretical results on the sample and computational complexity. The experiments on an 11x11 grid world show that LRMax is more efficient compared to other baselines.
I think this is an interesting paper that takes a principled approach towards a lifelong RL problem. However, I find the paper hard to read and I think that the connection of what paper presents to a real-life problem is missing.
The assumption of Lipschitz continuity of tasks makes little sense to me. Does this mean that the state and action space of the MDPs that specify these tasks have to be the same always? Is that not a very restrictive set of lifelong RL problems? And if it is tried to remove the restriction just by assuming a very big state and action space and how do we verify the Lipschitz continuity and will the method scale to large state, action spaces?
I also found it difficult to understand if the paper is considering an online setting or an offline setting. It seems to me it is considering an online setting, but then proposition 6 and 7 are called sample and computational complexity. In general, in online setting sample complexity can be expressed in terms of T - time step, or the number of rounds. But the paper presents computational complexity in terms of timesteps. Is it just wrong naming or Is the computational complexity the cost of computation at every time step? But then it includes a T term. For quite a lot of time, I thought that this is computational complexity and the T denotes the transition function until I realized, T in this theorem denotes time step.
(Using T for transition function and the time step is confusing).
But then how can computational complexity (that measures the time required) be expressed in terms of time?
The paper claims that it proposes an upper bound on Q values of an MDP that can be computed analytically. I doubt that. First, I think that if this is true then it would have been useful to keep this algorithm in the main body of the paper. I particularly feel that such a bound is a big and important contribution. In any case, it is in the appendix. In the appendix, the paper basically proposes to use dynamic programming to compute the upper bound. So we do assume a model of the transition probabilities of the world that is explicitly maintained and probably learned from the experience of the agent. Does this not restrict the approach of this paper to very small problems for which such tables can be maintained? (This restriction is in addition to the restriction imposed by the assumption of Lipschitz continuity).
Experiments:
The experiments shows that LRMax is more efficient compared to other baselines when it is presented with sequential MDP to solve. This is nice and expected. I think one of the experiments that would have been interesting is what happens when the tasks are not Lipschitz continuous. I would not expect the algorithm the perform well in such cases, but it would be good to know how poorly does the algorithm performs compared to other baselines. So how robust the proposed algorithm is to the assumption it is specifically designed for?
Minor comments:
- the paper keeps using the phrase ‘agent explores greedily wrt to Q function.’ I found this confusing, I think the paper meant agent acts greedily wrt to Q function. The world explore seems to indicate that there is a separate exploration phase.
- The paper claims that distance between two games in arcade learning environment is smaller than the maximum distance between any two MDPs defined on a common state-action space of ALE. Can the paper clarify what this means. Is there a way to verify this claim. What are the common state action space of all games in ALE?
 If it is so, then the LRMax will do great in terms of transferring knowledge in between games on ALE. Why did the paper choose not to show results on ALE then?


Reinforcement learning algorithms can acquire policies for complex tasks automatically, however the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning has enabled agents to leverage prior experience to adapt quickly to new tasks, the performance of these methods depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data due to on-policy training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, even if policies and value functions cannot. These dynamics models can then be used to continue training policies for out-ofdistribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task.

###  Summary
1. The paper proposes an algorithm capable of off-policy meta-training (Similar to PEARL) as well as off-policy policy adaptation (By relabelling previous data using the adapted model and reward function).
2. The basic idea is to meta-learn a model that can adapt to different MDPs using a small amount of data. Moreover, the adaptation is done by only changing the latent context vector (Similar to CAVIA or CAML). The remaining parameters of the model (theta) are fixed after meta-training.
3. The paper also proposes learning a universal policy that, when given the context vector of a task, can maximize the reward for that task. This means that for with-in distribution meta-testing tasks, the policy can be used as it is (by giving it the right context vector which can be computed by adapting the model). For out-of-distribution tasks, however, it is important to update this policy.
4. To update the policy, the paper proposes combining previously stored data (for example data used in meta-training) with the adapted model to do off-policy learning (Using SAC).
### Decision with reasons
I vote for rejecting the paper in its current form for the following reasons:
1- The paper assumes that it is possible to learn models for out-of-distribution tasks with a few samples that are accurate on all the previously stored data. This is fundamentally incorrect. If the MDP changes in a significant way, it is not reasonable to expect that we can adapt a model from a few samples. Moreover, even if we can adapt the model using a lot of new experience, it is not reasonable to expect that we can use this model to accurately label all previous data. The authors do acknowledge this when describing results in Figure 3, however they seem to underplay this limitation.
2- Turning the meta-RL problem into a supervised learning problem has already been explored. For instance, Nagabandi et al. (2018)[1] showed that it is possible to quickly adapt models to changes using meta-learning. They, however, used decision time planning for the control policy (By random shooting method). This paper, on the other hand, uses Dyna style planning with an off-policy learning algorithm on previously stored data. The only difference is the choice of off-the-shelf planning algorithm which is not a significant contribution (There are some other small differences, such as learning a context vector and not model initialization, learning a universal policy etc, however, I don't see how they are essential for the proposed approach; maybe the authors can clarify why those choices are essential)
3- The paper assumes a context vector alone is sufficient to capture changes in MDPs (It keeps the rest of the model fixed at adaptation). This might be reasonable if the context vector is sufficiently large, but the paper does not even mention the size of the context vector. It also skips other important details. For example, it does not mention any details about hyper-parameter selection, how the context-vector used in the model, etc. It's hard to judge the importance of the experimental results because of this.
### Questions
1- "Effective model training requires the validation batch to contain data corresponding to optimal behavior for the tasks, which we obtain by training a universal policy conditioned on the context descriptor"
It's not clear to me why the validation batch must contain data corresponding to the optimal behavior.
2- Is the proposed framework really consistent? At adaptation, only the context vector is being updated whereas model parameters (theta) are fixed. Why is a context vector alone sufficient to adapt the model to drastic changes in the MDP?
[1] https://arxiv.org/abs/1812.07671
### UPDATE
The authors gave a detailed response to the reviews and answered some of my main concerns. However, I'm still not convinced that the paper, in its current form, can be accepted. My issues are:
The paper combines some existing ideas in a new way but falls short of justifying the choices it made. The proposed contribution is that it is consistent (meta-learning methods that learn a network initialization are also consistent), can do off-line meta-training (So can PEARL) and can use old meta-training data at meta-test time (This is novel to this paper). However,  the proposed methodology also has some downfalls. For example:
It does not allow continual adaptation. This is an important limitation of existing consistent meta-learning methods and this paper does not address it. Nagabandi et al 18 [1], on the other hand, propose a similar solution that is also capable of continual adaptation.
MOST IMPORTANTLY, the empirical evaluation in the paper is very unsatisfactory. Even though the authors have included hyper-parameters in the appendix in the updated version of the paper, they still do no specify how these parameters were selected. Were the parameters selected to maximize the performance of their method and then copied for the baselines? This would not be a fair comparison.
Given the above-mentioned issues, I don't think the paper in its current form can be accepted and I'm maintaining my initial score. I think the authors should do a more thorough empirical investigation and tune the baselines and their method separately (using comparable compute budget). They should also report results on multiple environments using the same parameters (i.e. tune hyper-parameters on one or a few environments and reports results on some other environments as commonly done in Atari)
[1] Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL


We address the problem of Any-Code-to-Code Generation (AnyC2C) -generating code given its surrounding code without any restriction on the vocabulary or structure. The state-of-the-art in this problem is the sequence-to-sequence (seq2seq) approach, which treats code as a sequence and does not leverage any structural information. We introduce a new approach to AnyC2C that leverages the strict syntax of programming languages to model a code snippet as a tree -structural language modeling (SLM). SLM estimates the probability of the program's abstract syntax tree (AST) by decomposing it into a product of conditional probabilities over its nodes. We present a neural model that computes these conditional probabilities by considering all AST paths leading to a target node. Unlike previous structural techniques that have severely restricted the kinds of expressions that can be generated in this task , our approach can generate arbitrary expressions in any programming language. Our model significantly outperforms both seq2seq and a variety of existing structured approaches in generating Java and C# code. We make our code, datasets, and models available online.

This paper presents a grammar-based generation approach for "slot-filling" style code generation tasks. Given a context AST with opening non-terminal node, the model completes the opening node by predicting a sequence of child nodes, which forms a sub-AST rooted at the original opening node. The proposed model encodes context ASTs using a path-based approach (Alon et al., 2019a), essentially generalizing the previous model of Alon et al., 2019a from a code-to-sequence setting (e.g., generating natural language comments from code) to a "code-to-code" setting (i.e., code completion given contextual snippets).
Strong Points:
* The paper is very well written. The idea of formalizing code completion as structured language modeling and extending Alon et al., 2019a for the task is natural and well executed, with strong models and significantly improved results on two code completion benchmarks for both Java and C#.
* The authors attempted to establish comparisons with most existing code generation models.
Detailed Review:
*Technical Contribution* I have a very mixed feeling with this paper, while the model registers high empirical performance, the technical contribution is a bit limited, as detailed below:
    - *Path-based Context Encoding* The most important contribution in this submission is the application of path-based AST encoding model of Alon et al., 2019a to encode context (the given contextual and partially generated ASTs) for code generation. While the path-based encoding scheme is indeed a powerful model that intuitively encapsulates and generalizes over most previous approaches (Section 7), applying the model to a different task without significant task-specific adaptation or in-depth analysis might not sound technically novel. Meanwhile, the core idea of modeling/encoding the information flow in both the given context AST and partially generated programs for opening node expansion has already been explored in Brockschmidt et al. (2019a), albeit using a different encoding approach (GNNs) and in a relatively restricted setting of generating arithmetic expressions.
    - *Node-based Tree Generation Model* Apart from the path-based context encoding model, the node-based generation model presented in Section 2 also seems interesting. However, it might take longer time-steps to generate the node sequence instead of the sequence of production rules (composed of multiple child nodes), which could make optimization and inference more difficult. On the other hand, to control arity, the node-based approach need to inject synthetic "EOS" nodes to signal end of generating an argument sequence, while existing production rule-based systems could easily generate arbitrary number of argument nodes using either a transition system (e.g., Yin and Neubig, 2018) or a special neural component to compute end-of-argument-list probability (e.g., Rabinovich et al. (2017)), without using separate production rules of different arity.
    - *Syntactic Copy Mechanism* While the proposed syntactic terminal token copy mechanism (Section 3.3) could be better than the vanilla sequential one, there have already been syntactic copying models capable of copying both terminal tokens and partial ASTs from the context (Yin et al., 2019).
    How to Improve: to better understand the different technical contributions outlined above and their relative impacts, the following ablation studies would be helpful:
        - Importance of Path-based Context Encoding: the Seq→Path ablation in Table 3 alone might not be adequate to demonstrate the importance of path-based encoding of AST contexts for code generation tasks. The authors should compare with the GNN-based context encoding approach in Brockschmidt et al. (2019a) as this is the most relevant work. The original GNN→NAG model cited in Table 2 used a much simpler copying mechanism and a vanilla production-based tree generation model, and therefore not directly comparable with a tuned SLM.
        - Importance of Node-based Tree Generation Model: If possible, the authors might consider swapping their node-based tree generation model with a state-of-the-art production-based approach (e.g., Yin et al., 2019) to demonstrate its effectiveness.
*Claims* The authors claimed in the beginning of the paper that previous program synthesis approaches are either restricted in domains (e.g., DSLs like SQL) or grammars (e.g., restricted grammar of the full language), therefore coining the proposed approach as "any-code generation". However, there are indeed code generation systems (some of them cited in this paper) that synthesize open-domain code snippets in general-porpuse programming languages without restriction on vocabulary or grammar. To give a few examples, Iyer et al. (2018) generate open-domain Java class member functions; Rabinovich et al. (2017) and Zhao et al. (2019) predict full Python classes or partial snippets, while Yin et al. (2019) synthesize open-domain short C# code diffs observed in GitHub commits. In fact, the the proposed "any-code generation" benchmark is limited to sub-expressions defined within a function, whose scope is more restricted than other benchmarks like CONCODE.
    How to improve: the authors might present more evidence to substantiate the their claim on the novelty of the AnyGen benchmark compared with existing open-domain, general purpose code synthesis benchmarks, or consider revising the claim and the title.
References:
* Zhao et al., Neural Networks for Modeling Source Code Edits. 2019
* Yin et al., Learning to Represent Edits. 2019


We propose a novel high-performance interpretable deep tabular data learning network, TabNet. TabNet utilizes a sequential attention mechanism that softly selects features to reason from at each decision step and then aggregates the processed information to make a final prediction decision. By explicitly selecting sparse features, TabNet learns very efficiently as the model capacity at each decision step is fully utilized for the most relevant features, resulting in a high performance model. This sparsity also enables more interpretable decision making through the visualization of feature selection masks. We demonstrate that TabNet outperforms other neural network and decision tree variants on a wide range of tabular data learning datasets, especially those are not saturated in performance, and yields interpretable feature attributions and insights into the global model behavior.

This paper proposes a new neural network for tabular data which uses sequential attention to perform instance-wise feature selection which can help learn superior decision rules, in addition to facilitating interpretability of the resulting model.  Overall, the paper is well-written and fairly easy to follow, and the idea appears conceptually well-grounded and of high practical value. However, I have concerns about the experiments.
As the proposed model is primarily empirically motivated (ie. there is no theoretical justification for why it should be better), it is imperative the authors conduct thorough experiments to convincingly demonstrate its utility.  However, the current experiments are too sparse to be fully convincing, given how easy it is nowadays to evaluate ML models across diverse collections of tabular datasets (eg. OpenML, UCI, CatBoost). While other papers on tabular-data models study many real datasets (eg. Klambauer et al), this paper only studies 4 real datasets (and does not even quantify the variability in results across multiple training/test splits).
Also, why did the authors not compare against xgboost/lightGBM/catboost, some of the most popular tabular models, in all datasets?
It seems different hyperparameter values were also used for each dataset, what was the search-space / hyperparameter optimization method used here? Or were hyperparameters for each dataset found through manual tuning.
Klambauer et al (2017). Self-Normalizing Neural Networks.
https://arxiv.org/abs/1706.02515
OpenML: https://www.openml.org
UCI: https://archive.ics.uci.edu/ml/datasets.php
CatBoost: https://catboost.ai/#benchmark
Other comments:
- Why don't the authors quantitatively compare the quality of the selected features from their model vs other feature selection methods for the Syn datasets?
- What is "softmax-training" in Appendix D? If the authors just mean standard training, I recommend they use this name instead.
- Since there are many components to this model, it would help clarify their purpose to the reader if the authors could provide one concrete example of the feedforward pass of a hypothetical instantiation of their model, say for a simple datapoint with just 2 features.
- the subscript i in h_i should match the i used in M[i], P[i] to avoid confusion
- P[0] (the base case) needs to be explained.
- a[i-1] should be more clearly defined, and the base case explained in more detail (I assume it is just the input features).

- I think it is a bit confusing to present the model parameters in terms of batch-size B, and think it would be better if you introduced them assuming batch-size = 1 (the generalization to batch-size B is obvious and should be familiar to readers as it's the same for all neural models).


While generative neural networks can learn to transform a specific input dataset into a specific target dataset, they require having just such a paired set of input/output datasets. For instance, to fool the discriminator, a generative adversarial network (GAN) exclusively trained to transform images of black-haired men to blond-haired men would need to change gender-related characteristics as well as hair color when given images of black-haired women as input. This is problematic, as often it is possible to obtain a pair of (source, target) distributions but then have a second source distribution where the target distribution is unknown. The computational challenge is that generative models are good at generation within the manifold of the data that they are trained on. However, generating new samples outside of the manifold or extrapolating "out-of-sample" is a much harder problem that has been less well studied. To address this, we introduce a technique called neuron editing that learns how neurons encode an edit for a particular transformation in a latent space. We use an autoencoder to decompose the variation within the dataset into activations of different neurons and generate transformed data by defining an editing transformation on those neurons. By performing the transformation in a latent trained space, we encode fairly complex and non-linear transformations to the data with much simpler distribution shifts to the neuron's activations. Our technique has the advantage of being generally applicable to a wide variety of data domains, modalities, and applications. We first demonstrate it on image transformations and then move to our two main applications in biology: removal of batch artifacts representing unwanted noise and modeling the effect of drug treatments to predict synergy between drugs.

Overview: This paper presents a new generative modeling approach to transform between data distributions via a technique the authors dub “neuron editing”. Their approach for “neuron editing” can be used to learn how neurons in a DNN encode particular transformations in the latent space, with the hope of using it to be able to generate data that are out-of-sample and/or do not lie on the same manifold as the data used to train the generative model. Experiments were performed on the CelebA dataset. The authors further demonstrate the usefulness of their proposed idea by using it to remove noise/batch effects from data and also predicting synergy between drugs by using their idea to model the effect of drug treatments.
While the idea behind this paper is interesting and their method does seem to provide improvements over other generative models, the paper is a bit difficult to follow and the applications to biology are not explained well enough to understand the implications of their results. The fact that the authors did not compare to StyleGANs also seems a bit suspicious because these models are sota for the types of problems that this paper addresses. Additionally, the authors should consider rewriting the motivation for this problem to be more general. This is a convincing new method to transform distributions in any setting, but the introduction would lead one to believe its applicability is mostly to computational biology.
Detailed comments:
- The writing and motivation needs to be reworked in order to ensure that the introduction, results, and abstract match in their tone and content. The abstract uses a face recognition/generation application to motivate the work, the introduction focuses solely on computational biology, the results switch between biology and face recognition.
- Section 2: Three desirable properties for a transformation are listed, but what makes these properties desirable?
Equation (1): Not obvious why the piecewise function definition is necessary, since the second equation seems to hold for j = 0 and j = 99 as well. This should either be corrected or simplified.
- “Mode collapse refers to the discriminator being unable to detect differences in variability between real and fake examples.” Not true. Mode collapse is when a large region of the model’s input space maps onto the small region around a single (often bad) sample.
- Section 3: It would be helpful to see gradual interpolation from the original latent representation of a sample along the direction the neuron edit will be performed, and continued beyond the proposed final latent activations, to demonstrate how this process actually affects the model’s output.
- Doesn’t really seem like the authors tried very hard with the other methods (especially the vanilla GAN) and the omission of StyleGAN, the current state-of-the-art in this kind of transformation learning, is conspicuously omitted. It is highly likely that a well-trained StyleGAN would do better than the other GAN/AE techniques compared against.
- A PCA-based transformation is applied directly to the image data, but it would be interesting (and perhaps more informative) to perform an alignment of the PCs of activations in the latent space.
- Section 4: Again, the motivation distinctly focuses on computational biology when one could easily imagine this approach being applicable to a variety of problems.
- Comparisons with StyleGANs would’ve been appreciated, especially given the fact that they’re now considered state-of-the-art when it comes to modifying the latent space and creating out of sample images as proposed in this work.
- The idea is interesting, though one wonders if there isn’t any other work where the neurons have been “edited” to accommodate different transformations, given that the idea is itself rather intuitive. A more thorough literature review in that regard would be helpful.
- The CelebA experiments help lay establish an intuitive understanding of the proposed technique and were helpful. However, the ideas are a little disconnected with the biological applications of the technique. Better motivation/bridging of the two sets of experiments would be nice.
- A more detailed explanation and analysis of the combinatorial drug application would’ve been helpful to understand the results.


This work addresses the long-standing problem of robust event localization in the presence of temporally misaligned labels in the training data. We propose a novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervised learning. Unlike classical models which are constrained to strictly fit the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead. Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences. We demonstrate state-of-the-art performance against standard benchmarks in a number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance.

This paper deals with temporal label noise, or label misalignment problem in localization, which is the long-standing problem of robust event localization regarding the temporally misaligned labels in sequential data.  Some existing works are constrained to hardly fit the annotations of data during training. The work models the temporal label misalignment with perturbations to the ground-truth timestamps, and presents a soft localization learning scheme which relaxes the reliance on the exact position of labels. The idea introduced to temporal data is well-motivated since the label for temporal data is often dispersed. Technically speaking, the proposed SoftLoc loss comprises two terms: 1). $\mathscr{L}_{SLL}$: a soft learning loss that relaxes the prediction mass concentration, by symmetrically filtering the labels and predictions. This term helps to relax the model’s reliance on exact label locations. 2). $\mathscr{L}_{MC}$: a mass convergence loss that acts as a regularizer to facilitate the model with precise impulse-like localizations. With a trade of factor to balance two terms, the SoftLoc model can achieve precise impulse-like localization performance without weakening the model robustness. Various applications, such as PIANO ONSET, DRUM DETECTION and TIME SERIES DETECTION are performed to verify the effectiveness of the proposed method. And state-of-the-art performance is achieved.
Despite the achievement indicated by the experiments, I still have some concerns about this paper.
(1)	The authors propose the relaxed loss L_{SLL} for the soft learning of the location. And the label smoothing idea (applying a ˜S^2-Gaussian filter to the labels) has been introduced to increase the robustness to temporal misalignment of annotations [2]. Although the authors discuss the several inherent drawbacks of it in the 2nd paragraph of Related Work, they still follow the label smoothing idea in their model (Eq.2) in Section 4.1 in a two-side smoothed process. The reason should be clarified.
(2)	The authors claim that their method has the advantage of generalizing some regimes of weakly-supervised learning. And the adopted L_{MC} is a weakly-supervised loss. Does the advantage come from L_{MC}?
(3)	The authors propose the two-side relaxed loss L_{SLL} for the soft learning of temporal localization problem. However, in the ablation study, the authors do not give the results with different level of label noise for one-side variant. Adding these results could help to demonstrate the necessity of the two-side relaxed loss L_{SLL}.
References:
[1] Improved musical onset detection with convolutional neural networks
[2] Onsets and frames: Dual-objective piano transcription