Neural networks achieve state of the art accuracy on many standard datasets used in image classification. One would like to design neural network models whose performance is not affected by natural (or non-malicious) geometric transformations as well as adversarial (or malicious) perturbations. Previous work has studied generalization to natural geometric transformations (e.g., rotations) as invariance, and generalization to adversarial perturbations as robustness. In this paper, we examine the interplay between invariance and robustness. We empirically study the following two cases: (a) change in adversarial robustness as we improve only the invariance using equivariant models and training augmentation, (b) change in invariance as we improve only the adversarial robustness using adversarial training. We observe that the rotation invariance of equivariant models (StdCNNs and GCNNs) improves by training augmentation with progressively larger rotations but while doing so, their adversarial robustness drops progressively and this drop is very significant on MNIST. As a plausible explanation for this phenomenon we observe that the average perturbation distance of the test points to the decision boundary decreases as the model learns larger and larger rotations. We take adversarially trained LeNet and ResNet models which have good ∞ adversarial robustness on MNIST and CIFAR-10, respectively, and observe that adversarially training them with progressively larger norms results in a progressive drop in their rotation invariance profiles. On these adversarially trained models we observe that the change in the rate of invariance is marginal for small rotations. As a plausible explanation for this phenomenon, we show empirically that the principal components of adversarial perturbations and the principal components of perturbations given by small rotations are nearly orthogonal.

This paper examines the interplay between the related ideas of invariance and robustness in deep neural network models. Invariance is the notion that small perturbations to an input image (such as rotations or translations) should not change the classification of that image. Robustness is usually taken to be the idea that small perturbations to input images (e.g. noise, whether white or adversarial) should not significantly affect the model's performance. In the context of this paper, robustness is mostly considered in terms of adversarial perturbations that are imperceptible to humans and created to intentionally disrupt a model's accuracy. The results of this investigation suggests that these ideas are mostly unrelated: equivariant models (with architectures designed to encourage the learning of invariances) that are trained with data augmentation whereby input images are given random rotations do not seem to offer any additional adversarial robustness, and similarly using adversarial training to combat adversarial noise does not seem to confer any additional help for learning rotational invariance. (In some cases, these types of training on the one hand seem to make invariance to the other type of perturbations even worse.)
This paper is mostly clear and reasonably written. However, I do not think that the results of this investigation are significant enough to warrant publication at ICLR. In particular, I'm not sure that I really understand the motivation of this research question. I suppose a full notion of robustness would include invariance to perturbations of all types -- whether adversarial or otherwise -- and one might hope that techniques for encouraging such resilience mutually reinforce each other. However, since they are human perceptible, the perturbations associated with invariance are exactly the features that are not associated with adversarial noise, so I don't see why they should be related at all.
From a different perspective, invariance is a property of the true data distribution -- a rotated version of an image of a cat is another valid sample from the underlying distribution -- the invariance property is a type of constraint tying together different elements of the data generating distribution. On the other hand, adversarially perturbed images are often thought to be "off the data manifold" -- i.e. not valid samples from the true underlying distribution. Given this perspective, I am confused about why I should expect to see any interplay between these two ideas. The fact that the authors do not find any interplay is reasonable, but I remain confused about why they were investigating this question in the first place.
Unfortunately, this means that I don't think this work meets the significance criterion for being published at ICLR.


Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-ofthe-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

Paper 1872
Paper proposes a method for CL. The method is based on hypernetworks. These networks are a metamodel, which produce the parameters (from a task-conditioned embedding) which will be used in the main network. Preventing forgetting in the main network is now, replaced by preventing forgetting in the hypernetwork. This is done by imposing a regularization on the hypernetwork outcome, imposing that the generated weights should be similar for previous tasks (similar to Li & Hoiem who impose this on the network outputs). In addition, the paper proposes chunking, which refers to using an additional set of chunk, embeddings which are shared for all tasks, which allow compressing the hypernetwork. Furthermore, they propose an extension that allows for image replay (this is not an easy extension and an impressive contribution on itself, but maybe confusing for the current paper).
CONCLUSION
Overall, I like the idea of the paper and it is well explained. However, I found that the experiments of the paper where not well designed to verify the main contribution (hypernetworks), nor where they compared to the most relevant methods. I am borderline with this paper, and recommend borderline accept (borderline not being an option).
QUESTIONS
1. I think the motivation of why hypernetworks are expected to have less forgetting (than addressing forgetting directly in a network) should be discussed early in the paper.
2.I do not understand why the training is performed in two steps. First computing a candidate Delta THETA_H and then o ptimizing Eq 2. Why not directly optimizing Eq 2, replacing the second factor with|| f_h(e^t, THETA^*_h)-f_h(e^t, THETA_h) ||. This is how this regularization is normally applied (e.g. Li & Hoiem). If the authors insist in using Eq 2, I would like to see it compared with the proposed version.
3. The experiments should show that hypernets better address CL then addressing this directly in the network (and preferably provide reasons for this). Comparison with the closest methods like HAT and PackNet should be included. Especially, HAT is interesting since it is also based on an embedding.
4. Also, more experiments on CIFAR would be welcome. The MNIST variations already provide very high accuracies. For CIFAR-10/100 groups of 20 classes are added in 5 steps ? Scenario CL3 would be interesting for CIFAR as well.
5. I would like to see more analysis and results for the chunking. (As said before the replay is also a nice addition, but it seems an add-on of the main-text, shrinking the space to analyze the main contributions of the paper in the experiments.)
I guess HNET+ENT for CL1 scenario does not use ENT and is just HNET?


Active learning typically focuses on training a model on few labeled examples alone, while unlabeled ones are only used for acquisition. In this work we depart from this setting by using both labeled and unlabeled data during model training across active learning cycles. We do so by using unsupervised feature learning at the beginning of the active learning pipeline and semi-supervised learning at every active learning cycle, on all available data. The former has not been investigated before in active learning, while the study of latter in the context of deep learning is scarce and recent findings are not conclusive with respect to its benefit. Our idea is orthogonal to acquisition strategies by using more data, much like ensemble methods use more models. By systematically evaluating on a number of popular acquisition strategies and datasets, we find that the use of unlabeled data during model training brings a spectacular accuracy improvement in image classification, compared to the differences between acquisition strategies. We thus explore smaller label budgets, even one label per class.

The authors study the problem of incorporating unsupervised (representation pre-training) learning and semi-supervised learning into active learning for image classification; specifically, performing pre-training before active learning starts [Caron, et al., 2018] and then applying inductive label propagation [Issen, et al., 2019] (slightly modification in the cost function to look more like importance sampling) before active learning querying occurs for each round (Algorithm 1).  The most novel technical innovation of this submission is the joint label propagation (jLP) querying function (which is a method of ‘spanning’ the learned manifold space).  Experiments are conducted on four (multi-class) image classification datasets (MNIST, SVHN, CIFAR-10, CIFAR-100), showing that unsupervised learning and semi-supervised learning can improve active learning on these datasets — although random selection often works better (as best as I can tell) implying that negative results are also a contribution of this paper. Finally, some active learning experiments are conducted using a per-round label budget of one example per class — also demonstrating mixed results with random sampling performing better in general.
In my mind, this paper has two primary components: (1) taking the position that semi-supervised and unsupervised learning can improve overall performance and, in principle, help with active learning and (2) propose jLP, which is a learning algorithm agnostic approach to spanning the manifold space. However, jLP doesn’t really seem to work in general. Thus, the main result is the first point — updating previous (pre-deep learning) results on SS/US AL to deep learning. Honestly, I think the primary conclusion is that semi-supervised and unsupervised learning has improved over the past decade (especially semi-supervised learning for image classification). The second result is that active learning in deep learning (at least for this application) hasn’t kept up. Wrt to (1), as the authors have pointed out, many others have applied semi-supervised learning to AL (including more that the authors didn’t include). Additionally, many have used unsupervised learning for AL (which the authors seem less aware of) from pre-clustering (e.g., [Nguyen & Smeulders, Active Learning using Pre-clustering; ICML04]) to one/few-shot learning (e.g., [Woodward & Finn, Active One-Shot Learning; NeurIPS16 workshop]) to using pre-trained embeddings for many ‘real-world tasks’ (e.g., NER [Shen, et al., Deep Active Learning for Named Entity Recognition; ICLR18] using word2vec). Thus, the interesting question would be to compare multiple pre-training techniques and ideally the relative effect on the active learning component (assuming this is the focus of the paper). With respect to semi-supervised learning, they have validated that inductive label propagation [Issen, et al., 2019] works for this task, but haven’t shown that this helps with active learning. Since this is a negative results without a theoretical contribution, I would again expect trying several semi-supervised algorithm and evaluating their relative performance in general and wrt the active learning querying strategy. Accordingly, I don’t think the contribution of this work in its current state is sufficiently well-developed — and would lean toward rejecting in its current form.
Below are some additional detailed comments (some also covered above):
— Given that this points toward a negative result, a more convincing direction to take would be to consider more combinations of unsupervised and semi-supervised approaches — specifically emphasizing how they affect the active learning component. This might point to more general findings and maybe toward a theory (maybe even consider a second application).
— The empirical emphasis is more around overall performance rather than the interaction between unsupervised representation learning and active learning, which is more toward the stated goal of the paper.
— Wouldn’t the right way to do (deep) representation learning in multiple rounds be to fine-tune at least some fraction of the time?  If the only claim is pre-training or pre-clustering, people certainly do this — just often not as a point of emphasis.
— The ‘first semi-supervised’ claim really only holds in the context of deep learning; however, scope is really more like semi-supervised applied to image classification, which would be a pretty narrow contribution in scope.
— Overall, there is a general overstatement of contributions and results: this is certainly not the first SSAL or USAL and the statement relative to deep learning is subtle; some of the empirical results are interesting, but I am not sure about ‘spectacular gains’ (and these gains aren’t seemingly due to the contribution of the paper).
— I don’t understand the ensemble model analogy in the abstract; is it because it is a ‘meta-algorithm’?
Some more positive notes:
+ It is interesting that there is some contradictory evidence relative to [Wang, et al., 2017; Gal, et al., 2017]; this is probably worth digging into a bit deeper.
+ The experimental details well-described given space constraints.
In summary, there are some interesting observations that are probably worth pursuing. However, the current contribution is basically that: (1) active learning doesn’t seem to really help, (2) semi-supervised learning and unsupervised learning improve performance for this task. Since (1) was really the point of the paper (as stated) in the title, I don’t think there is enough here to accept in its current form.


Large transformer-based generative models trained on huge corpora have shown unparalleled language generation ability. While these models are powerful, finegrained control of attributes of the generated language (e.g. gradually switching topic or sentiment) is difficult without modifying the model architecture to allow extra attribute inputs or fine-tuning with attribute-specific data. Both entirely change the original generative function -which, if done poorly, cannot be undone -and also entail the significant cost of retraining. We instead propose the simple Plug and Play Language Model (PPLM) approach for controlled language generation. PPLM consists of plugging in simple attribute classifiers (which may be single layer models or even a bag-of-words), and making updates in the activation space, without changing any model parameters. Such a control scheme provides vast flexibility and allows full recovery of the original generative function. The results demonstrate fine-grained control over a range of topics and sentiment styles, as well as the ability to detoxify generated texts. Our experiments, including human evaluation studies, show that text generated via this control scheme is aligned with desired attributes, while retaining fluency.

The authors describe a method for training plug and play language models, a way to incorporate control elements into pre-trained LMs. In contrast to existing work, which often trains conditioned upon the control element, the authors emphasize that their method does not require re-training the initial LM. This is exciting and a great research direction. It is evaluated in a number of different settings.
1. The authors claim that this method is a baseline for controlled text generation (see e.g. the title). However, there does not appear to be any evaluation with any existing work that performs controlled text generation. I don't see how this can be proposed as a baseline for controlled text generation is there is no comparison to other methods. I imagine the authors will emphasize that that's not fair - because their method doesn't require retraining the language model - but it is relevant to demonstrate if there is a gap in performance or not. As is, there is only one baseline- unconditional language model - and to me this is mostly a way to calibrate the evaluators and not a way to compare their model against other models.
2. Can the authors make a point or discuss the relationship of this work to neural style transfer? Compared to unsupervised style transfer approaches, which also use lists of words or attributes to learn to dis-entangle content and style, what are the benefits of the proposed approach and how would it compare?
3. Can the authors discuss the effectiveness of their control mechanism for less logical control settings? For example, what if there was "religion" for "the potato" prompt? Does the model still respect these settings, or no?
4. Can the authors add analysis on how much the model respects the control variables? This is quite common in existing controlled generation papers. If the model is updated to have the control variables and then is not provided with one at test time, what happens? Can you also control very easy to measure attributes, such as length?
This question ties in with a general point I am ambivalent to in this paper- that it is very long, but there is very little analysis done on what makes the method work, why it is better than other control methods or control baselines, where the proposed control mechanism is not effective, how the model scales if there are large quantities of topics rather than just a few of them, if the BoW and discriminator attribute models work well together or if certain attributes are easier to learn than others, so the model focuses more on those when there are conflicts, etc
5. Missing citations:
Previous work has investigated controlling various attributes of text generation. Several of these works have also controlled multiple attributes simultaneously. For example, here's a list of a few of the works that were missed:
Kikuchi et al 2016
Ficler and Goldberg, 2017
Wang et al, 2017
Fan et al, 2018
Baheti et al, 2018
See et al, 2019
Martin et al, 2019
The related work section only focuses on very recent work, e.g. only one paper is discussed amongst a large body of existing work. I feel this is not an accurate reflection of how much previous work has investigated these techniques and analyzed how models deal with control variables.
Please also cite:
- which dataset was used for story generation, appears to be missing
- top-k sampling
I have read the author response. Thanks for the details and additional analysis in the paper.


The Information Bottleneck (IB) framework is a general characterization of optimal representations in large scale learning that suggests an intriguing interpretation of efficient representations in deep learning in particular. It is based on the optimal trade off between the representation complexity and accuracy, both of which are quantified by mutual information. The problem is solved by alternating projections between the encoder and decoder of the representation, which can be performed locally at each representation level. The non-parametric IB framework, however, has several drawbacks: (i) the difficult estimation of information measures in high dimensions; (ii) the inability to extend the encoder beyond the training data -the need to to know the full joint distribution of the inputs and labels; (iii) the IB does not necessarily optimize the actual prediction of unseen labels, which can be critical when training the encoder from finite samples. Here we present a new framework, the Dual Information Bottleneck (dualIB), which improves all the above drawbacks. By switching the order in the KL-divergence between the representation decoder and data, formally the dual distortion, the optimal decoder becomes the geometric rather than the arithmetic mean of the input points. This conversion preserves exponential forms of the original distribution and consequently conserves the original features of the data. We show how to solve this new representation learning formulation and illustrate it's favorable properties over the original IB when trained from a small sample. We also analyze the critical points of the dualIB and discuss their importance for the quality of this approach.

Description:
The information bottleneck (IB) is an information theoretic principle for optimizing a mapping (encoding, e.g. clustering) of an input, to trade off two kinds of mutual information: minimize mutual information between the original input and the mapped version (to compress the input), and maximize mutual information between the mapped input and an output variable. It is related to a minimization of an (expected) Kullback-Leibler divergence betwen conditional distributions of an output variable.
In this paper, instead of the original IB, authors consider a previously presented dual problem of Felice and Ay, where the Kullback-Leibler divergence is minimized in the reverse direction: from the conditional distribution of output given encoding, p(y|hat x), to the conditional distribution given the original input, p(y|x).
The "dual problem" itself has a more complicated form than the original IB, authors claim this is "a good approximation" of the original bottleneck formulation, and aim to prove various "interesting properties" of it.
- An iterative algorithm (Algorithm 1) similar to the original IB algorithm but with a few more steps is provided.
- A theorem about critical points where cardinality of the representation changes is given, similar to the IB critical points, and another theorem about difference of curves on an information plane between the IB and dual-IB solutions.
- Authors also show that if the true conditional distribution of outputs given inputs has an exponential-family form, the dual-IB decoder also has a form in the same family, which is said to reduce computational complexity of the algorithm.
Evaluation:
This is an entirely theory-based paper; although an algorithm is given, it is not instantiated for any concrete representation learning task, and no experiments at all are demonstrated.
Overall, I feel the motivation is not clear and strong enough. The abstract does not illustrate the importance of the mentioned "interesting properties" well enough for concrete tasks. Reading the paper, the clearest motivations seem to be improving computational complexity, and having a clearer connection to output prediction in cases where the predictor may be sub-optimal. However, authors do not quantify these well:
- The computational complexity improvement is not made clear (quantified) in a concrete IB optimization task: it seems it is only for exponential families, and even for them only affects one part of the algorithm, reducing its complexity from dim(X) to d; the impact of this is not tried out in any experiment.
- For output prediction, authors motivate that dual-IB could have a more direct connection e.g. "due to finite sample, in which it can be very different from the one obtained from the full distribution"). Authors further claim that the dual-IB formulation can "improve the generalization error when trained on small samples since the predicted label is the one used in practice". However, this is not tested at all: no prediction experiments, no quantification of generalization error, and no comparisons are done, thus the impact of the clearer connection to output prediction is not tested at all, and no clear theorems are given about it either.
The property that the algorithm "preserves exponential form of the original data distribution, if one exists" is interesting in principle, but it is unclear if any real data would anyway precisely have such a distribution; what happens if the data is not exactly in an exponential family?
In its current state the paper, although based on an interesting direction, in my opinion does not make a sufficient impact to be accepted to ICLR.
Other comments:
"Application to deep learning" mentioned in Section 1.5 is only a sincle sentence in the conclusions.
There have been some other suggested alternative IB formulations, for example the Deterministic IB of Strouse and Schwab (Neural Computation 2017) which also claim improved computational efficiency. How does the method compare to those?
Section 1.5 claims the algorithm "preserves the low dimensional sufficient statistics of the data": it is not clear what "preserves" means here, certainly it seems the decoder in Theorem 7 uses the same kinds of sufficient statistics as in the original data, but it is not clear that hat(x) would somehow preserve the same values of the sufficient statistics.


Exploration is a key component of successful reinforcement learning, but optimal approaches are computationally intractable, so researchers have focused on hand-designing mechanisms based on exploration bonuses and intrinsic reward, some inspired by curious behavior in natural systems. In this work, we propose a strategy for encoding curiosity algorithms as programs in a domain-specific language and searching, during a meta-learning phase, for algorithms that enable RL agents to perform well in new domains. Our rich language of programs, which can combine neural networks with other building blocks including nearest-neighbor modules and can choose its own loss functions, enables the expression of highly generalizable programs that perform well in domains as disparate as grid navigation with image input, acrobot, lunar lander, ant and hopper. To make this approach feasible, we develop several pruning techniques, including learning to predict a program's success based on its syntactic properties. We demonstrate the effectiveness of the approach empirically, finding curiosity strategies that are similar to those in published literature, as well as novel strategies that are competitive with them and generalize well.

This paper proposes to meta-learn a curiosity module via neural architecture search. The curiosity module, which outputs a meta-reward derived from the agent’s history of transitions, is optimized via black box search in order to optimize the agent’s lifetime reward over a (very) long horizon. The agent in contrast is trained to maximize the episodic meta-reward and acts greedily wrt. this intrinsic reward function. Optimization of the curiosity module takes the form of an epsilon-greedy search, guided by a nearest-neighbor regressor which learns to predict the performance of a given curiosity program based on hand-crafted program features. The program space itself composes standard building blocks such as neural networks, non-differentiable memory modules, nearest neighbor regresses, losses, etc. The method is evaluated by learning a curiosity module on the MiniGrid environment (with the true reward being linked to discovering new states in the environment) and evaluating it on Lunar Lander and Acrobot. A reward combination module (which combines intrinsic and extrinsic rewards) is further evaluated on continuous-control tasks (Ant, Hopper) after having been meta-trained on Lunar Lander. The resulting agents are shown to match the performance of some recent published work based on curiosity and outperforms simple baselines.
This is an interesting, clear and well written paper which covers an important area of research, namely how to find tractable solutions to the exploration-exploitation trade-off. In particular, I appreciated that the method was clearly positioned with respect to recent work on neural architecture search, meta-learning approaches to curiosity as well as forthcoming about the method’s limitations (outlining many hand-designed curiosity objectives which fall outside of their search space). There are also some interesting results in the appendix which show the efficacy of their predictive approach to program performance.
My main reservation is with respect to the empirical validation. Very few existing approaches to meta-learning curiosity scale to long temporal horizons and “extreme” transfer (where meta-training and validation environments are completely different). As such, there is very little in the way of baselines. The paper would greatly benefit from scaled down experiments, which would allow them to compare their architecture search approach to recent approaches [R1, R2], black-box optimization methods in the family of evolution strategies (ES, NES, CMA-ES), Thompson Sampling [R3] or even bandits tasks for which Bayes-optimal policies are tractable (Gittins indices). These may very well represent optimistic baselines but would help better interpret the pros and cons of using neural architecture search for meta-learning reward functions versus other existing methods. Conversely, the paper claims to “search over algorithms which [...] generalize more broadly and to consider the effect of exploration on up to 10^5, 10^6 timesteps” but at the same time does not attempt to show this was required in achieving the reported result. Pushing e.g. RL2 or Learning to RL baselines to their limits would help make this claim.
Along the same line, it is regrettable that the authors chose not to employ or adapt an off-the-shelf architecture search algorithm such as NAS [R4] or DARTS [R5]. I believe the main point of the paper is to validate the use of program search for meta-learning curiosity, and not the details of the proposed search procedure (which shares many components with recent architecture search / black-box optimization algorithms). Using a state-of-the-art architecture search algorithm would have made this point more readily.
Another important point I would like to see discussed in the rebuttal, is the potential for cherry-picking result. How were the “lunar lander” and “acrobot” environments (same question for “ant” and “hopper”) selected? From my understanding, it is cheap to evaluate learnt curiosity programs on downstream / validation tasks. A more comprehensive evaluation across environments from the OpenAI gym would help dispel this doubt. Another important note: top-16 results reported in Figure 4 and Table 1 are biased estimates of generalization performance (as they serve to pick the optimal pre-trained curiosity program). Could the authors provide some estimate of test performance, by e.g. evaluating the performance of the top-1 program (on say lunar lander) on a held-out test environment? Alternatively, could you comment on the degree of overlap between the top 16 programs for acrobot vs lunar lander? Thanks in advance.
[R1] Learning to reinforcement learn. JX Wang et al..
[R2] RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. Yan Duan et al.
[R3] Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search. Guez et al.
[R4] Neural Architecture Search with Reinforcement Learning. Barrett et al.
[R5] DARTS: Differentiable Architecture Search. Liu et al.


Multivariate spatial point process models can describe heterotopic data over space. However, highly multivariate intensities are computationally challenging due to the curse of dimensionality. To bridge this gap, we introduce a declustering based hidden variable model that leads to an efficient inference procedure via a variational autoencoder (VAE). We also prove that this model is a generalization of the VAE-based model for collaborative filtering. This leads to an interesting application of spatial point process models to recommender systems. Experimental results show the method's utility on both synthetic data and real-world data sets.

The work describes an application of a spatial point process for solving problems with missing data. The authors introduce a novel method based on a non-parametric definition of point intensities for the multivariate case. The method incorporates VAE framework to effectively handle missing points via smooth intensity estimation and enjoys amortized inference for efficient computations and quick prediction generation. Using a sequence of mild assumptions, the authors show connection to a popular VAE-based collaborative filtering model, which turns out to be a special case of their approach.

This is a rigorous study providing theoretically justified evidence on the effectiveness of the proposed approach. Apart from the issues in the last part of experiments with classical collaborative filtering task (which will be detailed below), the work presents a solid research. I would therefore vote for accepting it.
The text is well structured, and all key points are clearly explained. The problem solved by the authors is well described, and the motivation for this work is convincing. The way point process theory is applied constitutes a rigorous probabilistic approach. The authors convincingly justify the need for all approximations and simplifications made in the model. One of key results making the entire model feasible is supported by the corresponding theorem proved by the authors. I haven’t carefully verified all the derivations, though.

My major concern is related to the last part with experiments on the Movielens data. As the authors state, “applications without explicit spatial information, we embed each event into a latent space as a vector.” “No spatial information” is exactly the case with the standard collaborative filtering task, which the authors attempt to solve. This leads to an introduction of an additional model like GNN, which is unrelated to the main approach. As GNN is involved it’s not immediately obvious that the improvement over standard VAE architecture, observed in the experiments on ML-100K and ML-1M, is due to a better point process modelling.  No evidence is provided to argue that this is not simply due to a good compression or a good data preprocessing achieved by a GNN architecture itself. Therefore, the results on a pure recommender systems part are not convincing. What would happen if GNN was trained and fed into another (simpler) algorithm? Maybe a simple KNN based algorithm would produce comparable or even better results? As indicated by the work of [Dacrema, Cremonesi, Jannach 2019] on “A Worrying Analysis of Recent Neural Recommendation Approaches”, VAE-CF (along with several other recently proposed neural network-based methods) is inferior to even properly tuned kNN-based models. I would not be surprised, if a kNN model trained on GNN output would produce even better results than the proposed VAE-SPP.
Another related question is how incorporating GNN affects the training time? Is it comparable to that of VAE-CF or is it much worse? Computational performance is an important part in making practical decisions and should be also considered.
Furthermore, both ML datasets used for tests are too small and not very representative to make any generalized conclusions. Even on a larger ML-20M dataset an optimal SVD-based model can be trained within several minutes on a standard CPU on a laptop (according to my experiments, VAE-CF would take at least twice longer on Tesla K80). Therefore, it can hardly be considered a realistic example. In practice, there could be millions and hundreds of millions of items. The authors even mention it in the in the introduction, using it as a vehicle to motivate their approach. However, computing similarities between that many entities can be a laborious task on its own, which adds an extra layer of complexity and again is not directly related to the main approach. It can easily become a bottleneck or make further computations inefficient. More efficient similarity computations may in contrast reduce the resulting accuracy.
The issue can get even worse, because, unlike classical MF methods, there’s still no proper support for sparse operations in NN frameworks. In the VAE framework it means that, during the training, user batches will be converted into dense arrays and may become inefficient to work with in terms of memory and CPU utilization (a few non-zero entries vs. hundreds of millions of explicitly stored zeroes).
In spite of all this, I’d also suggest rephrasing “We validate these beneﬁts through extensive experiments” as it sounds a bit exaggerated (if we are considering real recommender systems applications). I agree that the proposed approach is potentially applicable in real cases for recommender systems, however, there’s still not enough evidence for this. In fact, I don’t even think that completely removing the part with ML-100K and ML-1M datasets would make the whole work any worse. Clearly stating the region of applicability of the proposed approach would be enough. Right now some statements in this section in contrast are raising concerns rather than convincing the reader. The wording should be at least changed, so that readers do not get an impression that the case with classical CF task is solved purely by the proposed VAE-SPP approach.
Other remarks to help improve the text:
1) “… points are more likely to … form clusters than the simple Poisson process …” the sentence seems to be inconsistent.
2) “The generative process of our model can be described as follow:” -> … as follows:
3) Page 4, last paragraph, line 6 – shouldn’t the upper bound for summation be N_u instead of just N?
References:
Dacrema, Maurizio Ferrari, Paolo Cremonesi, and Dietmar Jannach. "Are we really making much progress? A worrying analysis of recent neural recommendation approaches." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109. ACM, 2019.


Recent studies show that machine learning models are vulnerable to adversarial examples. In 2D image domain, these examples are obtained by adding imperceptible noises to natural images. This paper studies adversarial generation of point clouds by learning to deform those approximating object surfaces of certain categories. As 2D manifolds embedded in the 3D Euclidean space, object surfaces enjoy the general properties of smoothness and fairness. We thus argue that in order to achieve imperceptible surface shape deformations, adversarial point clouds should have the same properties with similar degrees of smoothness/fairness to the benign ones, while being close to the benign ones as well when measured under certain distance metrics of point clouds. To this end, we propose a novel loss function to account for imperceptible, geometry-aware deformations of point clouds, and use the proposed loss in an adversarial objective to attack representative models of point set classifiers. Experiments show that our proposed method achieves stronger attacks than existing methods, without introduction of noticeable outliers and surface irregularities. In this work, we also investigate an opposite direction that learns to deform point clouds of object surfaces in the same geometry-aware, but cooperative manner. Cooperatively generated point clouds are more favored by machine learning models in terms of improved classification confidence or accuracy. We present experiments verifying that our proposed objective succeeds in learning cooperative shape deformations.

This paper looks at the task of (adversarially or cooperatively) perturbing a point cloud in context of a classification task. It follows the prevailing paradigm of changing the original input in the direction of a high positive/negative gradient while staying ‘close’ to the original input, and the key contribution here is to define a different notion of ‘closeness’.
While previous work (Xiang et. al., at least for ‘point addition attack’) used a combination of chamfer and Hausdorff distance as the notion of closeness, this paper additionally includes change in curvature (which is an intuitive term to include) when computing the adversarial/cooperative updates to the point cloud. The obtained results do visually look less perturbed compared to the previous approach, and the obtained adversarial shapes are more robust against two defenses studied.
Concerns/Questions:
1) If a point cloud P’ is only a (small) perturbation of a point cloud P, then the Chamfer distance / Hausdorff distance is essentially the  L_2 / L_infinity norm of their difference. While the use of curvature terms is different, I feel the claims of importance of using ‘distance metric of point clouds’ is not very well justified (as I’d expect essentially same results if these two terms were replaced by L2 and L_infinity norms instead). I think the use of these terms was more necessary in the work of Xiang et. al., as they allowed point addition, so the ‘norm of difference of point clouds’ is not well defined.
2) This paper uses a different (more aggressive) adversarial term (in Eqn. 8) compared to Xiang et. al., so it is not surprising that the results in Table 1 indicate more robustness to defenses.
3) In addition to the above comments about specifics, I feel this work’s contribution over prior work is not significant. While Xiang et. al. did use L2 norm for their perturbation case, they did investigate the Chamfer/Hausdorff distances for another scenario, and therefore the main contribution here is an additional loss term.
4) This is perhaps a hard concern to address, but simply showing some qualitative results to highlight that the changes are ‘imperceptible’ is not sufficient. Ideally, one should report a curve on ‘change perceptibility’ vs ‘attack success rate’ (though this would require some notion of perceptibility that was not used in optimization). Alternately, one could compare methods via A/B testing on mechanical turk, asking ‘Which are these two shapes are closer to the original one?’, and ablate for a certain level of confidence on the wrong class, which approach led to less changes. The current results simply show some examples, but provide no empirical way of judging which approach actually leads to more imperceptible changes.
Overall, though the results are perceptually encouraging, I have slight concerns the empirical results reported. However, the primary issue is that the contribution regarding the additional term, while intuitive, is not a significant one in its own right.
While the rating here only allows me to give a ‘3’ as a weak reject, I am perhaps a bit more towards borderline (though leaning towards reject) than that indicates.


Inferring causal effects of a treatment, intervention or policy from observational data is central to many applications. However, state-of-the-art methods for causal inference seldom consider the possibility that covariates have missing values, which is ubiquitous in many real-world analyses. Missing data greatly complicate causal inference procedures as they require an adapted unconfoundedness hypothesis which can be difficult to justify in practice. We circumvent this issue by considering latent confounders whose distribution is learned through variational autoencoders adapted to missing values. They can be used either as a pre-processing step prior to causal inference but we also suggest to embed them in a multiple imputation strategy to take into account the variability due to missing values. Numerical experiments demonstrate the effectiveness of the proposed methodology especially for non-linear models compared to competitors.

Summary:
       The paper considers average treatment effect estimation treatment T and an unobserved confounder Z causes the outcome Y with an added constraint that the observed X is a noisy measurement of the underlying Z and some of the entries of observed X are missing at random (MAR). Previous work (Kallus et al. 2018) on similar settings assumed a low rank model connecting Z and X along with some entries missing at random which we do not observe. Further, Y (outcome) is related to the treatment and Z with a linear model. They actually show that matrix factorization techniques with these assumptions form an unbiased estimator for the Average Treatment Effect. There is prior work on doubly robust estimators under ignorability assumptions.
In this paper, the authors want to consider general non-linear relationships between Z and X with the same MAR (missing at random assumption) for missing entries.  So they fit a latent variable model in the form of VAE to find the P(Z| observed part of X) using a slightly modified version of the ELBO Lower Bound. For missing entries, they just replace those entries by a constant and do the usual VAE fit. After the VAE fit, multiple Z's are samples from the optimized P(Z| observed X) and then used in the doubly robust formula on each Z for estimating the average treatment effect and then finally the estimates averaged over the different Z's.
There is an alternative where the conditional mean of the latent variable is estimated from the VAE and used in the doubly robust computation.
In many synthetic examples, authors compare this with existing baselines and show that their method improves.
Pros:
 - Baselines compares are comprehensive enough from my perspective.
 - The paper is well written with clear pointer to existing work on doubly robust estimators with standard ignorability assumptions and the work for the linear, low rank model case by Kallus et al. 2018.
Cons:
   Major Issues
     - There is no reason to believe that even for the synthetic experiments, that the VAE posteriors would asymptotically yield unbiased ATE's which is provably the case in (Kallus et al. 2018) (of course for their restricted linear model/low rank assumptions). There is no reason to suppose Z's used in eq (9) from the VAE satisfy ignorability even in the asymptotic limit. In this light, the paper in essence just estimates Z's from some latent model that is fit and then use those latents to regress Y and then computes ATE. This seems a natural heuristic to try given such a problem. So I don't find a big methodological novelty. I would be willing to increase my scores if the authors could convince me on this point.
  - For the LRMF model (Fig 4) the MF approach seems to do as well as the authors proposal and we have a guarantee for the MF case under those linear/low rank modeling assumptions. So the only demonstrated benefit is for the synthetic experiments for Fig 5 and Fig 3 (I agree that it is considerable particularly with large fraction of missing values in Fig 3) in whose settings we dont know about how unbiased it is in the limit. More synthetic experiments with different kind of generative models could be more convincing.
- Some real world data set would have been more convincing - although I agree ground truth is hard to come by.
Minor Issues:
   - You have set B=200 for all the experiments for MDC-MI. Do the results change when B is increased or decreased ?? Does variance go down or the bias itself changes with B -  This would be a useful insight to have.
  -  There is typo in the ELBO lower bound equation in page 11. There are other minor typos. Please correct for it.
 - Since the paper is about estimate treatment effect from measurements of an unobserved confounder - it is important to cite - https://ftp.cs.ucla.edu/pub/stat_ser/r366-reprint.pdf from the causal DAG literature.
 - The covariance of X given Z for the DLVM model is not clear - It seems to say exp ( or some matrix vector products) * Identity. What does this mean ?
- The feature dimensions seems to be set at 10 - so would we expect the same results in much higher dimensions - like say 100s for few tens of thousands of samples??
********UPDATE after reading the rebutall,changes and the new experiment**********
I appreciate the authors actually accepting that identifiability issues cannot be easily resolved even if one knows P*(Z|X).
I recommend the authors to elaborate on this point in the camera ready version. However, showing that the proposed methods work on a real benchmark (semi-synthetic one used in Shalit et. al 2017) is commendable.  However, I find that the MF method is competitive (almost all the time) with their method when Doubly robust estimators are used.
But having matched an existing baseline (the MF method) that deals with confounders on real data and showing superior synthetic results and authors clarifying and toning down their theoretical claims, I am inclined to increase the score to Weak accept.


Automatic Piano Fingering is a hard task which computers can learn using data. As data collection is hard and expensive, we propose to automate this process by automatically extracting fingerings from public videos and MIDI files, using computer-vision techniques. Running this process on 90 videos results in the largest dataset for PIANO-FINGERING with more than 150K notes. We show that when running a previously proposed model for automatic PIANO-FINGERING on our dataset and then fine-tuning it on manually labeled piano fingering data, we achieve state-of-the-art results. In addition to the fingering extraction method, we also introduce a novel method for transferring deep-learning computer-vision models to work on out-of-domain data, by fine-tuning it on out-of-domain augmentation proposed by a Generative Adversarial Network (GAN). For demonstration, we anonymously release a visualization of the output of our process for a single video on https://youtu.be/Gfs1UWQhr5Q

In this paper, the authors proposed an automatic piano fingering algorithm, that accepts YouTube videos and corresponding MIDI files and outputs fingering prediction for each note. The claimed contribution is two-fold: First, they proposed the algorithm, and second, they claim that the algorithm can be used to automatically generate large datasets for piano fingering problems. The motivation is clearly stated and convincing. The overall algorithm is mainly described.
However, I would like to reject this paper. Major issues:
* Some key information is missing in Section 3.6, which is the only section that shows technical details: What is X_{n_k}? How is that related to the estimated finger poses? What is the function f in the definition of function g? (Also, it would be helpful to label the equations for clarification.) Are you doing Bayesian inference? With the key information missing, it is hard to fully understand the remaining technical details in this section.
* Their experimental results cannot properly support their claims. In Section 4.2, the authors try to show the strength of their proposed piano fingering algorithm by comparing their automatically annotated dataset APFD with an existing manually annotated dataset PIG. The authors showed the evaluation results of models trained and fine-tuned with different datasets. However, this is not an acceptable comparison for me, due to several reasons.
First, in order to show the strength of automatic piano fingering prediction, it is much better to directly run the prediction algorithm on datasets with known labels. According to the related work section, there is at least one existing work by Takegawa et al. that uses videos and MIDI files to detect piano fingering. Can you compare your algorithm with theirs?
Second, it is essentially unreliable to compare two datasets by comparing the performance of two prediction models, as there are too many implementation details that are almost impossible to control.
Third, it is not clear how we should compare the testing errors in Table 2. Yes, a model initially trained on PIG and fine-tuned on APFD may perform better than a model trained merely on APFD, but does that suggest anything (and the advantage is just 0.4%)? Similarly, the experimental result that an MLP model initially trained on APFD then fine-tuned with PIG works better than an HMM model that is trained with PIG data alone cannot prove anything. There are too many possible reasons that may lead to this experimental result.
* How is this method more attractable than the existing ones? There are neither experimental comparisons nor high-level justifications of why the existing algorithms are not applicable to the given scenario. In Section 2, although the authors described a good number of existing work on piano-fingering and their drawbacks, they failed to point out the strength of their paper as a comparison. As a result, the strength of this paper is still unclear after reading this section. How does this paper avoid the drawbacks of these previous papers?
* The writing of this paper needs to be greatly improved. It takes a lot of effort to literally understand this paper: There may be missing parts, misplaced clauses, and broken logic between sentences. I have listed several examples in the minor issues part.
Minor issues:
* In the first paragraph of Section 1: The sentence before 'In practice ...' is incomplete.
* In the last paragraph of Section 1: Missing brackets for \textsection 3.3 and \textsection 3.4. Also, 'on A new dataset we introduce' should be 'on THE new dataset we introduce'.
* On page 3, the sentence 'In this work, we continue the transition of search-based methods that optimize a set of constraints with learning methods that ...' is not making sense to me. Do you mean that your work is an extension of search-based methods, or do you mean that your work is not a search-based method? Also, are you optimizing a set of constraints, or optimizing with a set of constraints?
* On page 3, the last sentence in Section 2: '... and adapt their model to compare TO our ...' should be '... and adapt their model to compare WITH our ...'. The last part of this sentence is also a bit confusing: How do you compare a model with a dataset?
* On page 4, the paragraph starting with 'MIDI files': The first two sentences are almost the same; the period between them is missing. I guess one of them should be deleted. The following sentences in this paragraph are also subject to grammatical errors. For example, the sentence 'It consists of a sequence of events ... to carry out, WHEN, and allows for ...' is not a complete sentence. 'We only use videos that come along with a MIDI file' -> 'We only use videos that come along with MIDI files'.
* On page 5, last paragraph in Section 3.3: 'highest probability defections' -> 'highest probability detections'.
* The last paragraph on Page 5: 'Using off-the-shelve ...' -> 'Use off-the-shelf ...'.
* In Section 4.2.1, the corresponding result is Table 2, instead of Table 1.


Autoencoders are powerful generative models for complex data, such as images. However, standard models like the variational autoencoder (VAE) typically have unimodal Gaussian decoders, which cannot effectively represent the possible semantic variations in the space of images. To address this problem, we present a new probabilistic generative model called the Hierarchical Bayes Autoencoder (HBAE). The HBAE contains a multimodal decoder in the form of an energybased model (EBM), instead of the commonly adopted unimodal Gaussian distribution. The HBAE can be trained using variational inference, similar to a VAE, to recover latent codes conditioned on inputs. For the decoder, we use an adversarial approximation where a conditional generator is trained to match the EBM distribution. During inference time, the HBAE consists of two sampling steps: first a latent code for the input is sampled, and then this code is passed to the conditional generator to output a stochastic reconstruction. The HBAE is also capable of modeling sets, by inferring a latent code for a set of examples, and sampling set members through the multimodal decoder. In both single image and set cases, the decoder generates plausible variations consistent with the input data, and generates realistic unconditional samples. To the best our knowledge, Set-HBAE is the first model that is able to generate complex image sets.

The article proposes to replace the conditional distribution p(x|z) in an VAE by a general graphical model parameterized by a deep neural network. Using a series of approximations it arrives at a triplet of models, where the graphical model is split into a model representing the likelihood and a model that generates samples from the distribution. the third model is the usual autoencoder. The core idea is that the latent space of the VAE serves as bottleneck so that the graphical model develops a multi-modal distribution. At the same time, the use of a neural network allows for a much better modeling than using the typical local Gaussian distribution as is used in most VAE-approaches. Experimental results show that reconstructions and markov-chains starting from reconstructions produce decent results. An extension to sets is proposed as well as a task showing performance for semi-supervised learning.
-------------------------------------------
The text above serves mainly as a  summary of what the paper sets out to do. However, we have to check whether the paper is actually achieving this goal. In my view, it does not, and it suffers from severe problems on the theoretical side. Furthermore, important evaluations that are standard within the VAE framework are missing: samples from the true underlying distribution. In the remainder of this review, I will try to substantiate the claims regarding theory which lead me to an overall decision to reject the article in its current form.
Let us start from a birds-eye perspective: the model consists of three parts:
the Encoder E, which i will refer to as q(z|x)
the Energy-function, also called E(x|z) and with -E referred to as D later on.
a generator function G(z,z') that is supposed to sample from the distribution p(x|z) = 1/Z(z) exp(-E(x|z)) using entropy generated by z'~N(0,I). I will refer to this distribution as r(x|z)
This split is smart: using a general graphical model to model p(x|z) makes it very hard to sample from the distribution and it is difficult to estimate the normalization constant Z. The paper refers to Zhai et.al (2016) to find eq(3). I really like (3) because the inequality is tight when G is producing samples from p(x|z). To be more exact, we have that the estimator for the normalization constant is log(Z) - KL(r(x|z) || p(x|z) ). This reveals the first issue: the generator must have an r(x|z) > 0 for all x except a countable infinite subset. This is difficult to achieve with dimensionality of z and z' smaller than dimensionality of x and G being deterministic. Therefore, KL(r(x|z) || p(x|z) ) is already infinite. This is a technicality that i will ignore, since we can always add a minimal amount of noise on the output of G, even though this paper does not.
The real problem starts with eq(4). Since r(x|z) is implicit, there is no way to compute its entropy, which is then just discarded. The paper argues that this is no problem, because they intend to somehow bound the entropy. We will look at this point, later.
The issue with bounding is that for this to be meaningful, we also need a bound on the entropy of p(x|z), because otherwise there is no way for r(x|z) to achieve a small KL-divergence. If we now look at (4), we can compute the optimal r without bounded entropy. The optimal r is the distribution that by (5) maximizes -E(x|z) for a given z. this result is not a multi-modal distribution but a delta-peak on the optimum and certainly not p(x|z). so the bound is super important and technically we need a bound on log|det(d/dz' G(z,z'))|.
This is difficult from a theoretical view-point as well as from a modeling view-point because we know that we have to have nearly discontinuous transformations to create sharp edges. Further, to transform z'~N(0,I), into a multi-modal distribution, there must be a function g(z')->w so that w is multi-modal. This is difficult to achieve with bounded determinant.
Looking at the experiments, let's see whether the determinant is actually bounded. The paper claims that bounding can be achieved using batch-normalization. First this requires a reference on why this should be the case and show that this holds as implemented for the resnet block and second, this requires an estimate for how large the entropy can still be. if the entropy can be very large, we can not claim to train a VAE, because the error going from (3) to (4) is large. Secondly, we have to ensure that both E and G have bounded entropy, because otherwise KL(r(x|z) || p(x|z) ) will be large and there is no way to that the approximation in (3) is good, at which point E can not be claimed as doing maximum-likelihood on the data.
As shown in Figure 3, the Generator has a Conv-Block and the initial layer is a linear layer. Similarly the discriminator (or E) has a linear output layer. Therefore, the entropy is not bounded as the paper does not claim to have any entropy normalization here. The paper does not say what the activation functions are, but if the final layer of E is linear or Relu, we can add yet another problem to the list (next to the unbounded Jacobian): the VAE framework stops making sense. If we look at eq (6) or (7), the final divergence term (multiplied by beta) hinges on the fact that it provides a suitable error contribution. However, if the final layer of D is linear, we can just scale that layer arbitrarily large until the error contribution of the divergence vanishes.
Lastly, the Energy-function does not even necessarily model a probability function. Remember, we need that Z(z)=\int exp(-E(x|z)) dx <\infty. However, the definition by Figure (3) is E(x|z)=-h(x)^T(z) and the output layer of h is linear. so we can find weights such that h(x)=c \forall x. In which case
Z(z)=\int exp(-c^Tz) dx = exp(-c^Tz) \int 1 dx
which is only bounded if x is bounded, which I can not see since the generator is not bounded as well. This is only one of many counter-examples.
A few other small things:
- For z from a normal distribution the bottleneck discussion in 2.1 last paragraph does not apply, because for a dataset with N samples it is always possible to partition the real-space of Z into N partitions with probability 1/N each. This even works with 1-dimensional normal distributions, independently of the dimensionality of x. So a multi-modal distribution for p(x|z) is really not necessary if encoder and decoder are strong enough to memorize the dataset.
- below (5) it is claimed that this is similar to a WGAN, but WGAN have a difficult constraint to be 1-Lipschitz to make it work (which proves to be extremely difficult in practice as it translates to a similar inequality constraint on the jacobian)
- In Fig(5) it is unclear to me how a markov-chain is defined using the model. I assume this is done by taking samples from z', generating a new x and encoding this to obtain a new z. If z' models different features from z (assuming z' is multi-modal) we would assume that the z would remain the same in each point, so no exploration of p(z) takes place. However, we can already see in the lat rows that the pictures look more strange, so some mixing is taking place but it is not clear how that relates to p(z). Therefore i would really like to see images generated by sampling z~N(0,I) and z'~N(0,I) to see whether the generated images look reasonable.
- Moreover, to show that the Generator actually produces images that are likely under p(x|z), we would also need samples generated from the true underlying distribution p(x,z), e.g. using hamilton monte-carlo.
- Finally, i have to notice that the model, while called hierarchival VAE, does not provide a way to compute z' from the input. is that than really an autoencoder?


Formal verification techniques that compute provable guarantees on properties of machine learning models, like robustness to norm-bounded adversarial perturbations, have yielded impressive results. Although most techniques developed so far require knowledge of the architecture of the machine learning model and remains hard to scale to complex prediction pipelines, the method of randomized smoothing has been shown to overcome many of these obstacles. By requiring only black-box access to the underlying model, randomized smoothing scales to large architectures and is agnostic to the internals of the network. However, past work on randomized smoothing has focused on restricted classes of smoothing measures or perturbations (like Gaussian or discrete) and has only been able to prove robustness with respect to simple norm bounds. In this paper we introduce a general framework for proving robustness properties of smoothed machine learning models in the black-box setting. Specifically, we extend randomized smoothing procedures to handle arbitrary smoothing measures and prove robustness of the smoothed classifier by using f -divergences. Our methodology improves upon the state of the art in terms of computation time or certified robustness on several image classification tasks and an audio classification task, with respect to several classes of adversarial perturbations.

The paper introduces a generalization of the randomized smoothing approach for certifying robustness of black-box classifiers, allowing the smoothing measure to be an arbitrary distribution (whereas previous work almost exclusively focused on Gaussian noise), and facilitating the certification with respect to different metrics under the same framework.
Given the wide interest in certified robustness based on randomized smoothing, the generalizations considered in this paper could have a high potential. I found the motivation, the definition of the framework and the statement of the main theoretical results, up to Section 3.2, very clear. The following sections were not as well organized, in my opinion, and need improvement. In particular, I found it not very clear how the framework can applied to recover previous results, in particular the results from Cohen et al. (2019). All the ingredients seem to be there, among Theorem 3, the result from Balle & Wang (2018), and Corollary 5, but the arguments could be presented in a better organized way.
What I also found is missing is a clear descriptions of practical algorithms for applying the framework. Again, most of the ingredients seem to be there (e.g. Table 1 in the main body, Lemma 6 as well as the discussion about sampling mechanism in the Appendix), but they are not well organized. Presumably, the same statistical methodology as in Cohen et al. is used to obtain estimates of theta_a and theta_b, but this doesn't seem to be clearly stated anywhere. How is certification practically performed for intersections of contraint sets as introduced at the end of section 2? How the full-information certification can be applied based on empirical estimates is unclear to me, too.
Finally, the description of the experiments need improvements: For instance, in Section 5.1, which smoothing measure was used? Was the ResNet classifier trained using samples from this measure? Which sets of f-divergences were used? What does Figure 3 (a) exactly show? Do the blue points correspond to the 50 data samples? The number of random samples for computing empirical estimates and the confidence bounds are missing.
In Figure 4 (a), as the l_0 norm counts the number of altered pixels I don't understand why the certified accuracy varies e.g. for epsilon *between* 0 and 1.
The experiment on the Librispeech model seems interesting, but the paper does not contain sufficient information to understand and assess the experimental set-up or the results.
In summary, while I believe that the generalizations proposed in this paper have potential, in its present form the manuscript doesn't describe clearly and accurately enough how the framework can be applied to recover previous theoretical results or perform certification in practice.
----------
I acknowledge I have read the authors' response. As it addresses my main concerns I have changed my rating from "weak reject" to "weak accept".


The leading approaches to tensor completion and link prediction are based on the canonical polyadic (CP) decomposition of tensors. While these approaches were originally motivated by low rank approximations, the best performances are usually obtained for ranks as high as permitted by computation constraints. For large scale factorization problems where the factor dimensions have to be kept small, the performances of these approaches tend to drop drastically. The other main tensor factorization model, Tucker decomposition, is more flexible than CP for fixed factor dimensions, so we expect Tucker-based approaches to yield better performance under strong constraints on the number of parameters. However, as we show in this paper through experiments on standard benchmarks of link prediction in knowledge bases, ComplEx, (Trouillon et al., 2016), a variant of CP, achieves similar performances to recent approaches based on Tucker decomposition on all operating points in terms of number of parameters. In a control experiment, we show that one problem in the practical application of Tucker decomposition to large-scale tensor completion comes from the adaptive optimization algorithms based on diagonal rescaling, such as Adagrad. We present a new algorithm for a constrained version of Tucker which implicitly applies Adagrad to a CP-based model with an additional projection of the embeddings onto a fixed lower dimensional subspace. The resulting Tucker-style extension of ComplEx obtains similar best performances as ComplEx, with substantial gains on some datasets under constraints on the number of parameters.

The authors present a new way of decomposing 3-order tensors which uses interpolation
between the Tucker and CP decompositions, called CPT. The main idea is to present the components of the CP model
with an additional low-rank structure.
The authors also provide a new optimization algorithm called ADA-imp, for learning this decomposition,
which is a variant of Adagrad adapted to their settings.
The paper is overall interesting, clearly written and well-motivated.
The mathematical derivations are, as far as I could follow, correct and non-trivial.  (I did not read all the details in the Appendix).
The authors also show favorable experimental results on two knowledge-base datasets, with improved loss vs. #parameter used tradeoff.
A few unclear issues and suggestions for improvements are below.
The authors present the problem as completion of a binary 3-order tensor, i.e. predicting for triplets (subject, predicate, ?) if '?' refers to 0 or 1.
But they also write 'we formulate this problem as a multi-class classification problem, where the classes are the entities of the knowledge base'  - so this is not a binary problem? does this mean there is some structure that must be present in the tensor? (e.g. there is exactly one '1' in each column of length N? This should be clarified.
It would be good to make the description of Algorithms 1 and 2 more precise and detailed.
For example, the operation/algorithm AdaGrad(\eta;w_k; g_k;G_k) is not defined. AdaGrad is described in the Appendix but it is hard to match it to get the precise operation used in Algorithm 1.
Algorithm 1 shows one step of PComplEx, and it would be good to add the entire PComplEx algorithm, with input,output&parameters.
The authors present their method in the context of knowledge base completion, thus for tensors of order 3, but it is not clear if any of the components they proposed indeed specialized for this problem, or is it a contribution to general tensor decomposition. Some remarks regarding the (in?)applicability of the method more generally would be helpful.
Figure 3 describing the experimental results should be explained better. There are few methods shown only in some of the graphs and only for some parameter values - why?
The complexity measure 'parameters-per-entity' should be clearly defined (I didn't find it in the text). Similarly, the performance measures 'mean reciprocal rank' and 'hits at 5%'
should be defined in terms of the tensor.
The authors should also add running times of the different experiments and methods.
Minor:
--------
In the main paper, the authors define an (N,L,N) tensor, but in the appendix Section 9.9 they list N and P. Does P refer to L here?
The authors mention a few times usage of 'deep-learning techniques' - but I believe that in at least some of the contexts, they refer to optimization methods which are typically used in deep learning, and  are applied here to train other models presented in the text, and not to the usage of actual deep learning architectures - this is confusing and should be clarified.
Page 7, top: what are the matrices M^(1), M^(2), M^(3)? they seem to be different for different decompositions


Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.

This paper describes the "OmniNet" architecture, which is essentially a transformer to convert any 2-dimensional (time and spatial) input into a sequence of output tokens. The idea is that a single model (the "Central Neural Processor") would learn to perform multiple tasks on multiple inputs at the same time. This seems like a reasonable approach, and would allow a single model to be able to process text, image, as well as video or potentially speech input just the same. Dedicated modality-specific "input peripherals" would "normalize" the data appropriately, and a "start token" seems to provide the model with information on what type of input data is to follow.
I am a bit confused by the problem formulation and implementation:
- Multi-task learning works best if there are clear dependencies and shared characteristics between the task - I am not sure if the tasks used here (POS tagging, image captioning, VQA, video activity recognition) share sufficient characteristics
- Successful multi-task learning often leads to shared representations developing across the different tasks. Is there any indication of this happening in the proposed work? Would it help to make the modality specific encoders (BPE and ResNet) trainable, so that the transformer model has an easier time learning multi- and cross-modal representations? Or is this entirely the CNP's role?
- It is probably possible to use HogWild as a way to train a multi-modal and multi-task system in the described manner, but will this lead to shared representations developing? Is there any way to include regularization or other tricks to encourage the formation of shared cross-modal representations?
Here are a few ideas:
- Would it be possible to show how well the model works when trained and optimized (including hyper parameter optimization) on a single task only? it would then be possible to show that the multi-task model can outperform single-modality single-task models
- What about tasks like speech input? There is other, published work on multi-task training for CTC models w.g. by Ramon Sanabria ("Hierarchical Multi Task Learning With CTC", SLT 2018), in addition to the reference provided
- Would it be possible to show how the model learns to develop cross-modal representations during training? this would be a convincing visualization and rationalization of the proposed approach


The efficacy of the width of the basin of attraction surrounding a minimum in parameter space as an indicator for the generalizability of a model parametrization is a point of contention surrounding the training of artificial neural networks, with the dominant view being that wider areas in the landscape reflect better generalizability by the trained model. In this work, however, we aim to show that this is only true for a noiseless system and in general the trend of the model towards wide areas in the landscape reflect the propensity of the model to overfit the training data. Utilizing the objective Bayesian (Jeffreys) prior we instead propose a different determinant of the optimal width within the parameter landscape determined solely by the curvature of the landscape. In doing so we utilize the decomposition of the landscape into the dimensions of principal curvature and find the first principal curvature dimension of the parameter space to be independent of noise within the training data.

This paper targets at a deep learning theory contribution based on information geometry. This contribution is tightly based on Zhang et al. (2018) and explains the generalization of deep learning from a Bayesian perspective. The main contribution the authors claimed is an optimal degree of curvature exist which gives the best generalization guarantees, which is in contrast to the commonly perceived "the wider the better".
First of all, the writing (including language etc) is of poor quality, to the extent that the submission is very difficult to read and can be rejected merely based on this, with unusual expressions, missing  punctuations, super long sentenses, and wongly used words. The reviewer won't list example here because they are everywhere.
What is even worse is the conceptral errors and defected derivations. For example, in eq.(1), the authors equate the Fisher information matrix (which is an expected Hessian) to the Hessian matrix, this is subject to conditions which must be clearly given right before/after the equation. As their results are largely based on the correctness of eq.(2), let's examine the derivations in appendix A.1.  In the first equation in A.1, what is the subindex "j"?  "Utilizing Laplace Approximation of the integral": such approximations have conditions that must be clearly stated. It is not clear how one can get the last approximation in page 12 from the previous equations. In summary, their eq.(2) is a loose approximation which is subject to a set of conditions (that are not given), and the derivation is of poor quality.
As a theoreiritical contirbution, the authors did not manage to converge to some simple and clear statements (theorems or equvalent). Instead, the contribution is largely *explanatory*. It is hard to observe anything new, given the poor writing and organization. The first 4 pages are mainly introductions of previous works.
The authors used information geometry and minimum description length to explain the generalization of deep learning. This is a small area. It is hard to miss closely related works by simple searching. Instead, the authors only cited Rissanen (1978). On the other hand, as the authors used the spectrum properties of the Fisher information matrix, there are some recent works by Amari which can be cited.


We consider the learning to learn problem, where the goal is to leverage deep learning models to automatically learn (iterative) optimization algorithms for training machine learning models. A natural way to tackle this problem is to replace the human-designed optimizer by an LSTM network and train the parameters on some simple optimization problems (Andrychowicz et al., 2016). Despite their success compared to traditional optimizers such as SGD on a short horizon, these learnt (meta-) optimizers suffer from two key deficiencies: they fail to converge (or can even diverge) on a longer horizon (e.g., 10000 steps). They also often fail to generalize to new tasks. To address the convergence problem, we rethink the architecture design of the meta-optimizer and develop an embarrassingly simple, yet powerful form of meta-optimizers-a coordinate-wise RNN model. We provide insights into the problems with the previous designs of each component and redesign our SimpleOptimizer to resolve those issues. Furthermore, we propose a new mechanism to allow information sharing between coordinates which enables the meta-optimizer to exploit second-order information with negligible overhead. With these designs, our proposed SimpleOptimizer outperforms previous metaoptimizers and can successfully converge to optimal solutions in the long run. Furthermore, our empirical results show that these benefits can be obtained with much smaller models compared to the previous ones.

In this paper, the authors build on the 'learning to learn' work, that aims to leverage deep learning models with optimization algorithms, most commonly with recurrent networks.  The goal is to utilize meta-learners that can adapt the optimization update strategy based on data/experience.  The authors aim to tackle problems that often arise in such meta-optimization schemes, such as covergence and generalization problems.  The paper is overall well-written, and several experiments are presented
Building on previous work, the authors propose some variations to the meta-learning schemes and architecture.  I comment on these below.
- The authors remove the bias term based on the intuition that for the meta-learners, removing the bias can lead to better convergence.  This is supported by a set of experiments in Fig 3a (but with only one learning rate?).  This experiment shows an extraordinary difference in accuracy between including and not including the RNN bias.  I think, because this difference is substantial (along with papers that do use the bias in meta-learners showing good results overall), more evidence/ablations are required.
Can we can be sure that if the bias is non-zero (as in other reported works), we should expect a worse performance?.  Clearly this change seems to speed up the process, and in the example explained in the paper it makes sense, but could there be examples where lack of bias might lead to worse results?
- The authors claim that one of the problems of learning-to-learn algorithms is the late-stage optimization strategy.  While previous works introduce a weighting for the meta-optimizer's loss function (usually binary), the authors extend this to continuous monotonically increasing functions in order to weight the late stage steps of the algorithm more.  The authors propose compensating small loss difference happening during late optimization with a larger weight.   This naturally brings the question of which function should be used, as showing that any function will do seems difficult in practice. Have the authors tried any functions that failed or led to overfitting in comparison to other works?  Could one argue that if the loss difference is not large then perhaps the network can be more prone to overfitting at least in some cases?
Finally, the authors introduce embedding sharing between coordinates (aggregating hidden states), and also propose fine-tuning (sampling 5% of training data 5 times for warm-up).  Sharing hidden state information is expected to be useful (and has been often employed in literature), similarly to fine-tuning.  Also, 5% of dataset and 5 times seems to be an arbitrary choice - probably more related to the experimental section rather than the method itself.
a question on fine-tuning:  if fine-tuning was used for the proposed method, have the compared methods also been pre-trained to provide fair comparisons?  This is particularly relevant to fig 4B where only the fine-tuned method slightly overperforms SGD.


We present an unsupervised method for learning speech representations based on a bidirectional contrastive predictive coding that implicitly discovers phonetic structure from large-scale corpora of unlabelled raw audio signals. The representations, which we learn from up to 8000 hours of publicly accessible speech data, are evaluated by looking at their impact on the behaviour of supervised speech recognition systems. First, across a variety of datasets, we find that the features learned from the largest and most diverse pretraining dataset result in significant improvements over standard audio features as well as over features learned from smaller amounts of pretraining data. Second, they significantly improve sample efficiency in low-data scenarios. Finally, the features confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets, and the features likewise provide improvements in four different low-resource African language datasets.

Overview:
This work uses contrastive predictive coding (CPC) to learn unsupervised speech representations on large amounts of unlabelled speech data and then uses the resulting features in downstream speech recognition systems. Unlabelled data is obtained from several sources (spanning different languages). Supervised systems are then built on top of these features and sample-efficiency and cross-domain robustness is investigated using English data sets. Finally, the approach is applied to four African languages.
Strengths:
Firstly, the paper is very clearly written and motivated. Secondly, a very relevant problem is tackled in a systematic way; compared to transcribed resources, unlabelled resources are much easier to collect and more widely available. This paper shows that these unlabelled resources can be of great benefit in downstream tasks and on languages where few resources are available. Thirdly, the experiments are carried out very systematically to support the claims of the paper: that bidirectional CPC-based feature learning improves same efficiency (they show that much less labelled data is required to achieve the same performance as when using more substantial labelled data with conventional features), and that it improves robustness to out-of-domain data. They perform these experiments on both English and truly low-resource languages.
Weaknesses:
There are two main weaknesses to the paper. Firstly, as the authors note themselves, unsupervised CPC-based speech feature learning was developed and considered in previous work, and has also been subsequently investigated by others. The main technical contribution is therefore only in changing the unidirectional architecture to bidirectional. Secondly, the paper does a very poor job of linking this work with previous work. The work in [1] is very related. In Section 5, the ZeroSpeech challenges are mentioned briefly (with a single citation), but over the last decade there has been substantial work in this community specifically looking at exactly the main problem addressed in this paper (unsupervised speech representation learning). It would be of great benefit to situate this work within that context, and I would recommend that the paper at least mention [2] to [9].
Overall assessment:
Although technical novelty is limited (first weakness), I think there is novelty in the paper's systematic experimental investigation, including ASR experiments on truly low-resource languages. The conclusions of this work also has practical implications for the ASR community. The second weakness can be addressed by amending Section 5. I therefore assign a "Weak Accept" to the paper.
Questions, suggestions, typos, grammar and style:
- In Figure 1, it might be useful to indicate the autoregressive nature of the context vectors by adding arrows in-between the $c$ blocks on the top left. (In the text it says an RNN is used.)
- p.7: "... are suitable for driving recognition different languages ...". A typo or grammatically incorrect sentence.
- p. 9: "Tts without t" -> "TTS without T"
- p. 9: "african" -> "African" (check all citations for capitalization)
Missing references:
1. https://arxiv.org/abs/1904.03240
2. A. Jansen et al. A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition. ICASSP, 2013.
3. Badino, L., Canevari, C., Fadiga, L., & Metta, G. (2014). An auto-encoder based approach to unsupervised learning of subword units. in ICASSP.
4. Versteegh, M., Anguera, X., Jansen, A. & Dupoux, E. (2016). The Zero Resource Speech Challenge 2015: Proposed Approaches and Results. In SLTU-2016 Procedia Computer Science, 81, (pp 67-72).
5. Renshaw, D et al. (2015). A Comparison of Neural Network Methods for Unsupervised Representation Learning on the Zero Resource Speech Challenge. Interspeech.
6. R. Thiolliere et al. A  hybrid  dynamic  time  warping-deep  neural  network  architecture  forunsupervised acoustic modeling. Interspeech. 2015
7. https://arxiv.org/abs/1811.08284
8. https://arxiv.org/abs/1702.01360
9. https://arxiv.org/abs/1709.07902


Label Propagation (LPA) and Graph Convolutional Neural Networks (GCN) are both message passing algorithms on graphs. Both solve the task of node classification but LPA propagates node label information across the edges of the graph, while GCN propagates and transforms node feature information. However, while conceptually similar, theoretical relation between LPA and GCN has not yet been investigated. Here we study the relationship between LPA and GCN in terms of two aspects: (1) feature/label smoothing where we analyze how the feature/label of one node is spread over its neighbors; And, (2) feature/label influence of how much the initial feature/label of one node influences the final feature/label of another node. Based on our theoretical analysis, we propose an end-to-end model that unifies GCN and LPA for node classification. In our unified model, edge weights are learnable, and the LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved classification performance. Our model can also be seen as learning attention weights based on node labels, which is more task-oriented than existing feature-based attention models. In a number of experiments on real-world graphs, our model shows superiority over state-of-the-art GCN-based methods in terms of node classification accuracy.

The paper proposed an unified model for Label Propagation (LPA) and Graph Convolutional Neural Networks (GCN). It is shown how it is possible to infer the relationship between LPA and GCN in terms of label or feature smoothing (how label/feature does propagate over the neighbors) and label or feature influence over the other nodes. The results are given in terms of two theorems (whose proofs are in an appendix) which essentially state that the total label influence of nodes with a particular label “l” on a specific node is proportional to the probability that node is labelled as “l” by LPA. In practice, LPA acts as a regularizer to learn transformation matrices and edge weights simultaneously in GCN. By means of a simple joint loss (eq.8), the regularized training show that transductive learning with the joint model surpasses GCN/GNN baselines.
While the proof of the first theorem is reasonable and seems correct (Taylor+ Schwarz inequality + L-Lip), the second left me a little puzzled, in particular wrt eq 16 and 17. Is it possible to add a graphical explanation or idea of the proof? Why “y” has to be reset at each iteration?
Unfortunately, the improvement brought by the framework is marginal, even if it seems general.
The other limitation is that we have transductive training, but the authors are well aware about this.