The Variational AutoEncoder (VAE) can learn simultaneously a representation and a generative model, but often, in order to generate sharper samples, a useless representation is learnt. Starting from the observation that a meaningful representation is an informative one, in this work we analyse the VAE from an information theoretic perspective. We define the concept of capacity for the variational network, and we associate the two tasks of VAE: generation and representation quality, to two information properties of the network: information of the generative model and capacity of the network. Then we suggest that an optimal generative model is the one optimising the Capacity-Constrained InfoMax (CCIM), a theoretical objective learning the maximal informative generative model while maintaining bounded the network capacity. The theoretical assumptions are confirmed by the computational experiments, where between the different families of VAE, the one optimising a variational lower bound of the CCIM generates sharper samples and learns a clustered and robust representation. We call this the Variational InfoMax AutoEncoder (VIMAE).

The paper develops an information-theoretic training scheme for Variational Auto-Encoders (VAEs). This scheme is tailored for addressing the well-known disentanglement problem of VAEs where an over-capacity encoder sometimes manages to both maximize data fit and shrink the KL divergence between the approximate posterior and prior to zero. Consequently, the latent representations of the observations become independent, making them unusable for any downstream task.
The method developed in Section 3 and proposed explicitly in Eqs 7 to 11 is novel per se, though not groundbreaking.
Figs 2 and 3 are only few examples manually chosen from rather simple tasks. In the absence of a quantitative evaluation metric, they are not informative. In the outcomes of the same runs, there might exist counterexamples where the vanilla VAE generate perceptively more appealing reconstructions than VIMAE.
As a minor comment, I think the paper unnecessarily complicates the presentation of the idea. The development of notions such as f-Divergence, links to MMD, the constrained optimization setting in Eq 5 etc., do not serve to the main story line, but only distracts a mind. I would prefer a brief intro on VAEs and directly jumping into the proposed method. Last but not least, the statement of technical novelty in Section 3 is extremely encrypted. What exactly is "absolutely" novel there and what is prior art? Can the authors give a to-the-point answer to this question during the rebuttal?
The paper has two fundamental weaknesses:
i) The paper misses a key reference, which addresses the same representation disentanglement problem using the same information-theoretic approach, only with some minor technical divergences:
Alemi et al., Fixing a Broken ELBO, ICML, 2018.
This paper is a must-cite, plus a key baseline. This submission cannot be treated as a contribution without showing an improvement on top of this extremely closely related work. The problem is the same, the solution is almost the same, and the theoretical implications of the solution are also the same.
ii) The paper builds the entire story line around the statement that the representation disentanglement problem is caused by the properties of the ELBO formula and attempts to fix it by developing a new inference technique. This is attitude largely overlooks simpler explanations. For instance, the vanilla VAE assumes a mean field q(z|x) across the z dimensions. This makes q(z|x) fall largely apart from p(x) which definitely does not factorize that way. Earlier work has shown substantial improvements on the quality of q(z|x) when structured variational inference techniques are used, such as normalizing flows. Furthermore, there has also emerged techniques that can very tightly approximate p(x) in closed-form using the change of variables formula without performing any variational inference at all. No need to say, a tight approximation on p(x) means the same for p(z|x) simply using the Bayes rule. This leaves no room for further improvements by information-theoretic inference alternatives. For instance see:
Dinh et al., Density Estimation Using Real NVP, ICLR, 2017
The presence of such two strong alternative remedies to the problem addressed by this work makes its fundamentals shaky. The only way to get over this situation is to provide a thorough comparison against these methods, which is obviously missing.
---
Post-rebuttal: Thanks to authors for all the effort they put on the updated version. I appreciate that the paper now provides a quantitative comparison. However, my point (ii) remains fully unaddressed and I fully share the theoretical concerns raised by Reviewer 1. All in all, I keep my position on the below the threshold side with a weak reject.


We present FoveaBox, an accurate, flexible, and completely anchor-free framework for object detection. While almost all state-of-the-art object detectors utilize predefined anchors to enumerate possible locations, scales and aspect ratios for the search of the objects, their performance and generalization ability are also limited to the design of anchors. Instead, FoveaBox directly learns the object existing possibility and the bounding box coordinates without anchor reference. This is achieved by: (a) predicting category-sensitive semantic maps for the object existing possibility, and (b) producing category-agnostic bounding box for each position that potentially contains an object. The scales of target boxes are naturally associated with feature pyramid representations. We demonstrate its effectiveness on standard benchmarks and report extensive experimental analysis. Without bells and whistles, FoveaBox achieves state-of-the-art single model performance on the standard COCO detection benchmark. More importantly, FoveaBox avoids all computation and hyper-parameters related to anchor boxes, which are often sensitive to the final detection performance. We believe the simple and effective approach will serve as a solid baseline and help ease future research for object detection.

## Overview
The paper tackles object detection without predefined anchors (sliding windows).
The paper is well motivated and existing detection methods often rely on anchors, which may limits their potential. So this paper is solving an interesting problem and seems novel.
The paper provides experiments to support the proposed architecture empirically.
## Summary of the contribution:
1. The paper proposed an object detection approach called FoveaBox that does not rely on anchors (sliding widows).
2. The paper shows that FoveaBox outperforms some existing object detection methods.
3. The paper shows that FoveaBox can also be used for object proposals by changing the classification target to class-agnostic head.
## My feedback:
I think the paper is well motivated and has value to the object detection community. So I am leaning positive to accept this paper. The major reasons are:
1. The paper studies an interesting architecture which does not rely on sliding windows and shows its effectiveness. The problem seems not very well studied in the existing object detection literature.
2. The paper provides sufficient ablation study to analyze and understand the proposed methods.
However, I believe the paper could be significantly improved especially in the writing. So my position is not strong.
## Improvements
1. The writing can be significantly improved. The paper reads a bit confusing and unclear especially at the technical part. The paper could benefit from a clear overview figure about the proposed approach. Figure 4 seems to be doing that illustration but giving only the tensor shape seems quite confusing. I also listed some questions below about the actual algorithm.
2. The paper could be improved with experiments and ablation on another dataset. Current ablation shows that \eta should be 2.0 for best performance but without a second dataset it is hard to say this value is general. So one may have to tune this parameters in different datasets.
3. The scale is still discretized. So that is essentially anchors in the scale space. I wonder how could the approach applies to scale to?
## Questions
1. How would the proposed approach address the issue when there are multiple bounding boxes around the same pixels? It seems the current approach is predicting a box per pixel?
2. What is the inference computation cost? And do you have a comparison with existing methods on the compute cost?


Machine learning algorithms for generating molecular structures offer a promising new approach to drug discovery. We cast molecular optimization as a translation problem, where the goal is to map an input compound to a target compound with improved biochemical properties. Remarkably, we observe that when generated molecules are iteratively fed back into the translator, molecular compound attributes improve with each step. We show that this finding is invariant to the choice of translation model, making this a "black box" algorithm. We call this method Black Box Recursive Translation (BBRT), a new inference method for molecular property optimization. This simple, powerful technique operates strictly on the inputs and outputs of any translation model. We obtain new state-of-the-art results for molecular property optimization tasks using our simple drop-in replacement with well-known sequence and graph-based models. Our method provides a significant boost in performance relative to its non-recursive peers with just a simple "for" loop. Further, BBRT is highly interpretable, allowing users to map the evolution of newly discovered compounds from known starting points.

This paper presents a translation-based method for molecular property optimization. It uses a sequence- or graph-based encoder/decoder framework to produce molecules with (hopefully) improved properties, then feeds a subset of these molecules back into the encoder/decoder to generate a new set of molecules; this process is repeated for a fixed number of iterations to arrive at a final set of "optimized" molecules. The method is agnostic to the form of the encoder/decoder; the emphasis is on the iterative approach. Additionally, this approach enables visualization of "molecular" traces that can reveal pathways between molecules that follow relationships similar to matched molecular pairs. The work extends related work in translation-based property optimization [6]. The paper is well-written and generally easy to read.
The method is evaluated on two tasks, logP and QED. Both of these are computed properties that have known issues (see the discussion in [3]), but I understand that these properties are used in many publications and are thus easy to compare. The method presented here performs similar to others on a QED task. They claim superior results on the logP task, but I have concerns about the fairness of the comparison since logP can be exploited by very simple models if there are no limits on the size of the generated molecules (or, similarly, the number of tokens/generative steps allowed for each molecule). Additionally, the authors claim to perform multi-objective optimization but do not actually do this.
The iterative nature of this method is very interesting. However, my concerns about the types of experiments and comparisons that were done (see below for more details) are big enough that I cannot approve this paper in its current form. Weak reject.
Specific notes (starting with page number):
- 1: "potential druggable candidates" does not make sense; compounds are not "druggable" (their targets are), although they may be "drug-like".
- 2: Consider citing Kramer et al.'s seminal work on matched molecular pairs [1].
- 3: Please explain what it means for y to "paraphrase" x?
- 5: For your logP experiments, you need to be more clear about how you are comparing to other models. You are guaranteed to get to higher logP values if you can generate larger molecules (more tokens) than the baselines, since logP is essentially linear in the number of carbons. Are you doing something to limit the number of tokens you can generate in each iteration? Or why should I believe these comparisons are fair?
- 5: In Table 1, note that some literature uses a "normalized" penalized logP, while others use the formula directly without a dataset-specific normalization (which can appear to give better results). Can you confirm which you are using here and whether the baseline models are the same?
- 5: The results in Table 1 would be more compelling if they were not divorced from their starting points. Please include information about the similarity of these molecules to the starting molecule as well as the property delta. Consider an approach like Jin et. al [2], where results were specifically categorized by similarity constraints.
- 5: "All models were trained on the open-source ZINC dataset." What subset of ZINC are you using?
- 5: The supplementary figure showing that logP is broken is missing?
- 5: "Consistent with the literature we report diversity as...". Please cite some literature that you are consistent with.
- 6: "we sample 100 times from a top-2..."; does this mean you are doing 100 iterations? Sampling 100 times from the same top-2 sampler doesn't really make sense, but I'm not entirely sure what you are describing here.
- 7: Figure 4 says these are "ablation" experiments. What exactly are you ablating?
- 8: You state that better performance on logP and similar performance on QED is not known in the literature. In fact, the MolDQN paper [3] calls this out explicitly (and also contains a discussion of bounded vs. unbounded logP).
- 8: "Recent RL methods focus on molecular construction and are therefore not well-suited for the generation of molecular traces"; I disagree with this. RL methods that can start from a predefined graph have the ability to move between compounds, possibly in a way that is orthogonal to traditional similarity-based exploration (see the discussion of "MDP edit distance" in [4]). Also note that one of the key features of graph-based generators like [2] and [5] is that all of the intermediate states are valid, so you could do similar molecular traces for interpretability (although your differences are more like MMPs with functional group-level deltas).
- 9: "synthetic chemists can carry out the individual steps of a molecular trace..."; in general this is not true. The known medicinal chemistry transformations are a relatively small set of operations, and your molecular traces are unlikely to capture them in any systematic way. Please avoid making claims like this unless you can back them up with experimental evidence or comparisons to models explicitly trained for synthetic route planning.
- 9: These experiments are not multi-property optimization. Measuring the value of a secondary property while optimizing a primary property is not the same as optimizing them both simultaneously. The latter requires some strategy for incorporating both property values into the decision function, such as scalarizing (see "multi-objective optimization" on Wikipedia).
References:
[1] Kramer, C. et al. Learning Medicinal Chemistry Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET) Rules from Cross-Company Matched Molecular Pairs Analysis (MMPA). J. Med. Chem. 61, 3277–3292 (2018).
[2] Jin, W., Barzilay, R. & Jaakkola, T. Junction Tree Variational Autoencoder for Molecular Graph Generation. arXiv [cs.LG] (2018).
[3] Zhou, Z., Kearnes, S., Li, L., Zare, R. N. & Riley, P. Optimization of Molecules via Deep Reinforcement Learning. Sci. Rep. 9, 10752 (2019).
[4] Kearnes, S., Li, L. & Riley, P. Decoding Molecular Graph Embeddings with Reinforcement Learning. arXiv [cs.LG] (2019).
[5] You, J., Liu, B., Ying, R., Pande, V. & Leskovec, J. Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation. arXiv [cs.LG] (2018).
[6] Jin, W., Yang, K., Barzilay, R. & Jaakkola, T. Learning Multimodal Graph-to-Graph Translation for Molecular Optimization. arXiv [cs.LG] (2018).


Nowadays, machine learning models are becoming a utility in many sectors. AI companies deliver pre-trained encapsulated models as application programming interfaces (APIs) that developers can combine with third party components, their models, and proprietary data, to create complex data products. This complexity and the lack of control and knowledge of the internals of these external components might cause unavoidable effects, such as lack of transparency, difficulty in auditability, and the emergence of uncontrolled potential risks. These issues are especially critical when practitioners use these components as black-boxes in new datasets. In order to provide actionable insights in this type of scenarios, in this work we propose the use of a wrapping deep learning model to enrich the output of a classification black-box with a measure of uncertainty. Given a black-box classifier, we propose a probabilistic neural network that works in parallel to the black-box and uses a Dirichlet layer as the fusion layer with the black-box. This Dirichlet layer yields a distribution on top of the multinomial output parameters of the classifier and enables the estimation of aleatoric uncertainty for any data sample. Based on the resulting uncertainty measure, we advocate for a rejection system that selects the more confident predictions, discarding those more uncertain, leading to an improvement in the trustability of the resulting system. We showcase the proposed technique and methodology in two practical scenarios, one for NLP and another for computer vision, where a simulated API based is applied to different domains. Results demonstrate the effectiveness of the uncertainty computed by the wrapper and its high correlation to wrong predictions and misclassifications.

Motivated by real-world challenges in applying pre-trained models, the authors propose a model for selective prediction (prediction with an option for abstention) that wraps an existing black-box classification model. The resulting model output is a Dirichlet distribution with mean equal to the categorical distribution produced by the black-box and concentration parameter specified by a separate auxiliary model. This additional model is trained to minimize negative log-likelihood of observations under categorical distributions sampled from the aforementioned Dirichlet along with an L1 regularization term on the concentration parameter. To infer the model’s level of uncertainty, the authors propose computing the entropy of the average of sampled categorical distributions.
The authors evaluated this model on several pairs of sentiment-analysis NLP tasks and one pair of image datasets where a base model is trained on the source dataset, and the auxiliary model is trained on the second, target dataset. Using metrics proposed in (Condessa et al. 2015), the results show positive results at nearly all thresholds compared to a simple entropy baseline.
The paper addresses an important practical challenge in machine learning, but a confusing problem-framing and lack of robust baselines make me skeptical that it is suitable for publication at ICLR 2020.
A primary concern with this work is its framing of the problem as one of measuring aleatoric (irreducible) uncertainty, but the motivation in transfer learning and interdependence in production ML systems requires models that can characterize epistemic (reducible) uncertainty. A black box model that yields distributions over classes expresses aleatoric uncertainty via that distribution, and uncertainty due to a shifted data distribution is epistemic as additional data from the new domain would reduce it.
More problematic is the study’s lack of robust baselines. The authors only present the predictive entropy baseline, but numerous methods exist for out-of-distribution detection and selective classification. Though the specific case of selective classification from a blackbox base model is perhaps more niche, other methods from related problems can either be adapted accordingly or used as upper / lower bounds on what we can expect for this problem.
Some simple baselines I would expect to see include:
* Training a new classification model entirely on the new domain.
* Training an auxiliary classifier to predict if the base model will be correct (similar to SelectiveNet but without a shared network body). Ideally this model should also have access to the base-model’s prediction as input.
* “Confidence score” (i.e. the probability assigned to the base-model’s predicted class) -- this is a common baseline for OOD detection.
Additional questions / concerns:
* Do I understand correctly that when drawing samples for the entropy calculation, $E[\hat{y}]$ will equal the black-box model’s prediction in the limit of sample size? If so, this looks like an inefficient and complicated proxy for measuring the concentration of categorical entropies produced by the Dirichlet.
* Since the paper is focused on scenarios where one is taking advantage of a pre-trained model, one might wonder if the wrapping scheme is indeed less expensive than a new model trained from scratch on the new domain (e.g. with comparable capacity to the proposed wrapping model). Authors should include experiments / baselines to assess this.
* In section 2, the paper asserts that assuming access to logits breaks the blackbox assumption, but these are computable from softmax values (up to constant factors).
* Is the beta regularization term is theoretically required to prevent unbounded growth or is it simply an empirically practical necessity?
* In the image transfer experiments, STL-10 has relatively very few labeled examples (hundreds per class), so why was this only used as the source domain? Realistic transfer learning generally entails an expensive model trained on a source domain with plentiful data transferred to a target domain with scarce data.
* Authors should mention that STL-10 images represent a distribution shift from CIFAR-10 as the two were not generated identically. The paper currently only highlights differences in dataset size.


Long training times of deep neural networks are a bottleneck in machine learning research. The major impediment to fast training is the quadratic growth of both memory and compute requirements of dense and convolutional layers with respect to their information bandwidth. Recently, training 'a priori' sparse networks has been proposed as a method for allowing layers to retain high information bandwidth, while keeping memory and compute low. However, the choice of which sparse topology should be used in these networks is unclear. In this work, we provide a theoretical foundation for the choice of intra-layer topology. First, we derive a new sparse neural network initialization scheme that allows us to explore the space of very deep sparse networks. Next, we evaluate several topologies and show that seemingly similar topologies can often have a large difference in attainable accuracy. To explain these differences, we develop a data-free heuristic that can evaluate a topology independently from the dataset the network will be trained on. We then derive a set of requirements that make a good topology, and arrive at a single topology that satisfies all of them.

The authors propose a new weight initialization method for sparse neural networks and develop a weight topology that satisfies desirable properties. Their derivation is data-free, and thus the analysis should generalize to arbitrary datasets. They demonstrate that their new topology outperforms existing approaches on a matrix reconstruction task.
Overall I think this work is an interesting direction for designing static sparse neural network weight topologies, but it’s lacking in empirical evidence of their claims and could do better to tie themselves to existing literature in training sparse neural networks.
If the authors could strengthen their results by a) experimenting with their newfound topology and initialization on standard sparsification benchmarks like CIFAR, ImageNet, and WMT EnDe b) comparing their approach to other static-sparse [1, 2, 3] and dynamic-sparse [4, 5] training algorithms this could be a good paper, but without more experimentation it’s unclear what can be taken away from this work. If the authors added results in this direction I would be willing to increase my score.
Comments on Claims of the Paper:
1. “Cascades” are a known trick in both dense & sparse neural networks [2, 6].
2. The authors describe their motivation for developing a new sparse initialization method in the first paragraph of section 4. It would be nice to see some of this experimental data, such that we could understand the magnitude of the vanishing gradient problem in these tests and see that the newly derived initialization alleviates it. The reason for my skepticism is that Liu et al [7] used a similar scheme where they re-scale the standard deviation of the gaussian based on the fraction of nonzero weights, but later found that it made no difference in their results for unstructured pruning (which I learned through discussion with the authors).
4. The data-free derivation approach makes sense and I understand that this makes the approach theoretically applicable to arbitrary datasets, but the authors do not apply it to other datasets to show that it generalizes in practice.
5. The authors show that their topology outperforms on the matrix reconstruction task, but they don’t compare with other sparsification approaches used in deep learning like sparse evolutionary training [4], or SNIP [1] (note that these techniques also maintain a sparse network during training, as opposed to pruning approaches like magnitude pruning [8] that are dense during training but sparse for inference).
Comments on the Results of the Paper:
The authors appear to contextualize their work in deep neural networks, but all of their experimental results are on matrix reconstruction or linear models on MNIST. This is sufficient for analysis and motivation, but taking the developed approaches and applying them to a deep neural network and showing improvements would go a long way towards improving this paper.
Assorted Notes:
In the first paragraph of the introduction:
“In other words, doubling the size of layer inputs and outputs quadruples the size of the layer. This causes majority of the networks to be memory-bound, making DNN training impractical without batching, a method where training is performed on multiple inputs at a time and updates are aggregated per batch.”
It is correct that a matrix-vector product (i.e., neural network training with batch size 1) is typically memory bound on accelerators like GPUs, but it’s not clear why the quadratic growth in computational cost with input/output size has anything to do with this. The cause of memory bound-ness is lack of reuse of operands, which can be alleviated by increasing the batch size s.t. the computation becomes a matrix-matrix product. Batch size 1 training is also not desirable. Recent work has shown that large batch sizes do not degrade model quality with proper hyperparameter tuning [9], and larger batch sizes are desirable from a hardware perspective to achieve higher throughput.
References:
1. https://arxiv.org/abs/1810.02340
2. https://d4mucfpksywv.cloudfront.net/blocksparse/blocksparsepaper.pdf
3. https://arxiv.org/abs/1903.05895
4. https://www.nature.com/articles/s41467-018-04316-3
5. https://openreview.net/forum?id=ryg7vA4tPB
6. https://arxiv.org/abs/1811.10495v3
7. https://arxiv.org/pdf/1810.05270v2.pdf
8. https://arxiv.org/abs/1710.01878
9. https://arxiv.org/abs/1811.03600


Multi-task learning has been successful in leveraging the dependency among different tasks to improve general performance. However, when applying to 'discrete' data (graph/text), most of the existing frameworks only leverage the general task dependency with the assumption that the task dependency remains the same for (1) different data samples; and (2) different sub-structures (node/word) in one data sample (graph/text). Yet, this assumption may not be true for real-world 'discrete' datasets. Thus, we propose a novel multi-task learning frameworkLearning To Transfer Via Modelling Multi-level Task Dependency, which leverages both general task dependency and data-specific task dependency to guide the knowledge transfer among tasks. We also propose a decomposition method to reduce the space complexity needed by transfer functions from quadratic to linear. To show the effectiveness of our framework and the importance of modelling multi-level task dependency, we conduct experiments on several public graph/text datasets, on which we obtain significant improvements over current frameworks.

The submission argues for the modeling the relationships between different tasks and incorporating such relationships when training multi-task frameworks. Though the basic concept (usefulness of modeling and incorporating the relationships among tasks) is valid, the submission has a number of critical issues, namely missing prior work that did that that already, missing critical specifics of the method, and unnecessary mix of different concepts.
Elaborated comments:
A) Authors seem to be unaware of critically related prior work that specially modeled task relationships and did much of what's proposed in this submission, especially "Taskonomy: Disentangling task transfer learning". The "relationship among tasks" that this submission frequently talks about is the main concept in taskonomy 2018 paper (see their abstract). Besides the apparent similarities (eg the fig 1 of this submission vs fig 1&2 of taskonomy or fig 4 of this submission vs fig 13&7 of taskonomy), the formulation has strong similarities too (transferring from "task-specific" encoders of source tasks to target tasks using transfer readout functions, or ensembling multiple task-specific representations which seem to be the same as taskonomy's higher order transfer). This submission should be majorly revised in light of prior work and the critically relevant ones should be discussed and experimentally compared to.
B) The presentation suffers from missing critical specifics. For instance, the "general task dependency" matrix  shown in Fig 3 and mentioned in page 4, which seem to be the same concept as taskonomy's task affinity matrix, is only mentioned in passing. While that seem to be one of the most important components of the method and its definition and extraction method should be discussed.
C) Inline with the point B above, the presentation of the "Transfer Block" and what the authors refer to as "Point-wise Mutual Attention Mechanism" has issues and missing details. This block could potentially have new points in it, but it's not feasible to judge that and its technical correctness given the current disposition. For instance eq 2 seem to suggest the authors develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from (to reduce T^2 complexity to 2T). The rest of the section does not provide a clear implementation of this and add mathematical/notation confusions. Eg H_i_j is defined to be the task-specific representation of the source task i but is indexed over both tasks i and j where j is the target, or there is a E_j(X_j) where both indexes are j while E's index is over tasks and X's index is over datapoints.
Similarly the submission seem to jump over certain concepts/terms e.g. "multi-view task dependency" in page 4 vs"multi-level task dependency" in the title, etc. What exactly "view" or "level" mean here? Are those phrases really needed? Dropping any loosely grounded phrase would be a useful practice toward a clearer presentation.
Overall, unfortunately the submission suffers from serious issues in its current shape.
----
Comments after rebuttal stage:
Thanks to the authors for the rebuttal. It provided some help, but unfortunately it doesn't resolve the majority of the issues, as most of them are too major. A clear discussion on how the proposal is different and why it is better than the recent works that were not cited would be needed, and likely authors needed strong experimental comparison with some of them, eg to prove both general and data-specific task dependency is needed.
I also didn't find the hierarchical justification clear or convincing "The reason is that different from the image which [1] focuses on, text and graph data are hierarchical: word -> sentence and node -> graph. The task dependency at the basic level (word and node) may be different from the general task dependency".


While the accuracy of image classification achieves significant improvement with deep Convolutional Neural Networks (CNN), training a deep CNN is a timeconsuming task because it requires a large amount of labeled data and takes a long time to converge even with high performance computing resources. Finetuning, one of the transfer learning methods, is effective in decreasing time and the amount of data necessary for CNN training. It is known that fine-tuning can be performed efficiently if the source and the target tasks have high relativity. However, the technique to evaluate the relativity or transferability of trained models quantitatively from their parameters has not been established. In this paper, we propose and evaluate several metrics to estimate the transferability of pre-trained CNN models for a given target task by featuremaps of the last convolutional layer. We found that some of the proposed metrics are good predictors of fine-tuned accuracy, but their effectiveness depends on the structure of the network. Therefore, we also propose to combine two metrics to get a generally applicable indicator. The experimental results reveal that one of the combined metrics is well correlated with fine-tuned accuracy in a variety of network structures and our method has a good potential to reduce the burden of CNN training.

This paper aims to speed up finetuning of pretrained deep image classification networks by predicting the success rate of the process without running it. The authors suggest running samples from the target task on the (trained) source model, and computing a few sensible measures from the final output layer which indicate how well the trained features are separating the target images. Many experiments are presented, and some of them seem promising.
Overall the idea is, while simple, interesting and potentially promising: DL Training costs have been rising significantly in the past few years, and being able to make sensible predictions about which source dataset/task combination best fits a target task would be a great contribution. Nonetheless, there are some methodological concerns that cast doubt about the presented results, which would need to be addressed before making this paper ready for publication.
Comments:
1. The authors present multiple experimental results, many of which indicate a somewhat noisy signal. The only method that works on most tasks, combining S5 and S6, is somewhat underreported. How many different \alpha values did the authors consider? how were they selected? Was the same alpha values used for each all experiments? Given this large set of measures and combinations of measures experimented with in this paper, I am left wondering whether this approach would generalize to new target datasets.
2. A major assumption the authors are making is that there is a single number that determines whether a given trained architecture would transfer well to a new target task. But the finetuning process is affected by many factors (e.g, hyperparameters, random seeds), and thus it might be that with a different hyperparameter selection, the observed correlations would look completely different. I would have liked to see at the very least an analysis of the correlation between multiple runs of finetuning that show that they are well correlated before computing correlations with external measures.
3. The authors point out that transfer learning works poorly on medical imaging (Sato et al., 2018). It would be interesting to experiment with such datasets and observe whether the proposed method is able to make accurate predictions in this domain.
4. The experiments regarding truncating CNNs seem interesting, and I was disappointed to find out they are not part of the main paper.
Other comments:
1. Moons et al. (2016) and Molchanov et al. (2017) speed up *inference* and not training.
2. The authors argue that Mahajan et al. (2018) "did not apply it (their method) for detection or segmentation tasks.". However, neither did this paper.
3. Writing:
-- Several typos and grammatical errors across the paper. For instance:
- "*In* many previous research efforts suggested" ("in" should be dropped)
- Hypothesis H1 is ungrammatical
-- Many of the citations were in the wrong format (intro: Canziani et al. 2016), repetitive (3.5: Yaguchi et al.), etc.


Predicting not only the target but also an accurate measure of uncertainty is important for many applications and in particular safety-critical ones. In this work we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing definition for calibration of a regression uncertainty (Kuleshov et al., 2018) has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new definition that escapes this caveat and an evaluation method using a simple histogram-based approach inspired by reliability diagrams used in classification tasks. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple scaling-based calibration that preforms well in our experimental tests. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO (Lin et al., 2014) and KITTI (Geiger et al., 2012)  datasets.

This paper focuses on the calibration for the regression problem. First, it investigates the shortcomings of a recently proposed calibration metric [1] for regression, and show theoretically where this metric can be fooled. Later, it introduces new metrics for measuring the calibration in regression problem which are instantiated from similar idea as the ECE calibration metric [2] used for classification problem. The paper defines the uncertainty as the mean square error and like the ECE idea, it divides the samples into different uncertainty bins and for each bin, it calculates RMSE of network output uncertainty. The RMSE versus variance of estimated uncertainty is depicted as the reliability diagram.
To summarize the calibration error as the numbers, the paper proposes two other metrics as ENCE and CV that explain the overall calibration error for the network and calibration error for each sample, respectively. It also shows the effectiveness of the metrics, on the synthetic data where the calibration metric proposed in [1] can be fooled VS. more informative metrics like proposed reliability diagram with RMSE and mVAR.  The other experiment is conducted on object detection problem where the uncertainty of bounding box positional outputs is considered as the calibration for the regression problem.
Pros:
1- Investigating theoretically the shortcoming of the calibration metric for regression is interesting
2- Proposing the new metrics specifically  designed for regression in novel, however it needs more accurate investigation.
Cons:
The paper is not well-written and the experiments and discussions are not support the ideas, more specifically I can mention several concerns:
1- Motivation: The flaw of the previously proposed calibration metric is not explained clearly. The paper only discusses in which scenario, the metric cannot work properly considering. However, the considered assumption (for instance the output of uncertainty estimation from the network has uniform distribution) can never happen in real scenario. Then the importance of redefining of calibration method is not clear.
2-  Accuracy of Proposed Method: The measure proposed for calibration (Eq. 8) should be shown it will converge to the definition of calibration proposed in Eq.5, which is missing in the paper.
3- Lack of Clarity in some Parts: The paper introduces the  new architecture for adding the uncertainty output to the regression network. But this new architecture is not explained clearly. I suggest the authors add more details about this part.
4- Justification: The paper trains the uncertainty part of the network with NLL loss function and later fine-tunes the output with temperature scaling method. But this calibration  method is not related to the proposed metric which is claimed it has better calibration clarity. Then the importance of using the new metric to define the calibration is not clear.
5- Experiments: The experiment setup needs more accuracy. The paper should investigate the shortcomings of the previous metric in the same setting as the new proposed metric. In experiment Sec. 4.1, the settings for obtaining the output of the network uncertainty is different then comparing the results are not fare.  The experiments for real scenario is not wide enough. It just shows that the parameters get calibrated. However we expect to see more results about the CV metric and its importance to define.

Overall, I think this paper should be rejected as 1) the novelty of proposed metrics are not enough. 2) the motivation and justification of why the proposed metrics is better than previous metrics is not clear. 3) the experiments are not supporting the idea.
References:
[1]  Kuleshov, Volodymyr, Nathan Fenner, and Stefano Ermon. "Accurate Uncertainties for Deep Learning Using Calibrated Regression." International Conference on Machine Learning. 2018.
[2] Naeini, Mahdi Pakdaman, Gregory Cooper, and Milos Hauskrecht. "Obtaining well calibrated probabilities using bayesian binning." Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015.


We introduce HUBERT 1 which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.

This paper proposes an alternative way of reusing pretrained BERT for downstream tasks rather than the traditional method of fine-tuning the embeddings equivalent to the CLS token.
For each bert embedded token, the proposed method aims at disentangling semantic information of the word from its structural role. Authors provide two ways to provide this disentagling using LSTM or transformer blocks. with several design choices such as: *  a regularization term to encourages the roles matrix to be orthogonal and hence each role carry independent information *  design the roles and symbols matrices so that the number of symbols is greater than the number of roles
In evaluation authors design several experiments to show that:
* Does transferring disentangled role & symbol embeddings improve transfer learning
* the effectiveness of the TPR layer on performance?
* Transfer beyond Glue tasks?
While those experiments provide empirical gains of the design choices, authors don't show enough study to attribute those  empirical gains to the presented design choices:
One large claim in the paper is that empirical gains in the ability of transfer between similar tasks MNLI and GLUE is because of disentangling the semantics from the role representations. We don't know if the TPR layer really manages to do that, this could have been easily verified using for example clustering word senses of the same word.
The empirical gains in transfer learning can be simply attributed to:
- More params it seems adding an LSTM over bert embeddings already does some improvement, I would have loved to see this more exploited but it wasn't. This aligns with some recent findings that BERT is undertrained (Liu et al. 2019) https://arxiv.org/abs/1907.11692
- Variance in the results (authors report only results of one single run not mean and std of several runs).
- More budget given to hyper-parameter search for the models proposed in the paper.  Hyper param budget isn't also reported in the paper.
- other factors, not the ones associated with the claims in the paper: for example what authors claim is an ablation study was comparing several different models together. It would have been more interesting to see for example the effect of making the # symbols = # roles or removing the orthogonality loss from the roles matrix.
Conclusion: The paper introduces large claims and empirical results that correlate with, however the provided experiments are not done with enough control to attribute gains to the design choices provided in the paper.


While generative adversarial networks (GANs) have revolutionized machine learning, a number of open questions remain to fully understand them and exploit their power. One of these questions is how to efficiently achieve proper diversity and sampling of the multi-mode data space. To address this, we introduce BasisGAN, a stochastic conditional multi-mode image generator. By exploiting the observation that a convolutional filter can be well approximated as a linear combination of a small set of basis elements, we learn a plug-and-played basis generator to stochastically generate basis elements, with just a few hundred of parameters, to fully embed stochasticity into convolutional filters. By sampling basis elements instead of filters, we dramatically reduce the cost of modeling the parameter space with no sacrifice on either image diversity or fidelity. To illustrate this proposed plug-and-play framework, we construct variants of BasisGAN based on state-ofthe-art conditional image generation networks, and train the networks by simply plugging in a basis generator, without additional auxiliary components, hyperparameters, or training objectives. The experimental success is complemented with theoretical results indicating how the perturbations introduced by the proposed sampling of basis elements can propagate to the appearance of generated images.

The paper proposes a model for stochasticity for conditional image generation, building upon the previously available (DCFNet) results on composition of  convolutional filters out of the elements of the filter basis.
The idea of introducing stochasticity by convolutional filters into the conditional generative models seems to be novel and the reviewer thinks it could be of interest for the community.
The following remarks could be given to improve the presentation:
1) Theorem 1 is an existence theorem, so it does not give the procedure for construction of the basis. Does the construction procedure for the basis, described under the theorem formulation, meet the conditions of Theorem 1?
2) The Theorem 1 formulation states that “ If there exists a set of deterministic linear transforms”. Should the linear independence be stated as well as one of the theorem conditions ( so that the space dimensionality would indeed be K)?
3) The reviewer finds the structure of Section 4 confusing: it starts from the problem statement (first paragraph 'Using the method above, filters of each stochastic layer…’), then provides the description of the approach and only then outlines Theorem 1. It might be that stating Theorem 1 and then defining the method for generation of the basis (how exactly could we get to the basis? ) could improve readability of the paper. Essentially, the question is: is there any way to emphasise the procedure for filter generation and inform the reader in which circumstances these filters would be the basis (e.g. why it wouldn't be prone to the analogue of mode collapse when the filters do not effectively have enough diversity for linear independence)?
***
In addition to this list, it might be useful to provide some evidence on whether there is any inherent mechanism to regulate the diversity of filters and therefore of samples (so that to change the variability of the conditional samples from the model with the impact analogous to the one of temperature in Glow (Kingma et al, 2018)). If there is one, further experimental evidence, which shows the impact on diversity of filters, would contribute to improvement of the paper.


Stacked_WAE-2.pdf
Probabilistic models with hierarchical-latent-variable structures provide state-ofthe-art results amongst non-autoregressive, unsupervised density-based models. However, the most common approach to training such models based on Variational Autoencoders often fails to leverage deep-latent hierarchies; successful approaches require complex inference and optimisation schemes. Optimal Transport is an alternative, non-likelihood-based framework for training generative models with appealing theoretical properties, in principle allowing easier training convergence between distributions. In this work we propose a novel approach to training models with deep-latent hierarchies based on Optimal Transport, without the need for highly bespoke models and inference networks. We show that our method enables the generative model to fully leverage its deep-latent hierarchy, and that in-so-doing, it is more effective than the original Wasserstein Autoencoder with Maximum Mean Discrepancy divergence.

The paper aims to develop a deep generative model, which -unlike VAEs or GANs- comprises a hierarchy of latent variables rather than a direct map from the stochastic latent manifold to the observation space. To this end, the paper builds a training objective based on nesting the Wasserstein distance between the data distribution and its estimation arbitrarily many times. The generated objective corresponds naturally to a deep hierarchical generative model.
The principled approach followed to achieve the objective is solid and elegant. It is also intuitive and matches nicely with some valid observations highlighted in the paper such as insufficiency of by-passing intermediate latent variables (sentence above the Sec 2.3 title).
One major weakness of the paper is that it lacks a sufficient argumentation about how it differentiates from earlier attempts to nest Wasserstein distances. For instance,
Y. Dukler et al., "Wasserstein of Wasserstein Loss for Learning Generative Models", ICML, 2019
Apart from the theoretical argumentation, the paper should also compare their solution to this prior work on a number of benchmarks.
Another major weakness is that the paper lacks a quantitative evaluation scheme for its success. The experiments section starts with the claim that the proposed method "significantly" improves on the WAE, which I fail to see on the plots.
Lastly, Having said that the proposed method is novel and elegant, it is still a straightforward extension of the existing and well-known Wasserstein Auto-Encoder (WAE) approach. It extends WAEs by repetitively applying the tricks proposed by this earlier work, putting aside some minor additional adjustments.
Minor on style: The abstract does not give any single hint about the methodological novelty of the work.
---
Post-rebuttal: Thanks to authors for their effort for clarifications. Yet, I'm afraid the author response does not touch at all to any of the concerns I have raised. There are well-known ways to compare the success of generative models, FID being one of them as the authors point out. Another could be the test log-likelihood of a synthetic data set the true distribution of which can be predesigned. I understand the issues the authors raise about the difficulties in comparing generative models, but I kindly disagree with the attitude that there are no ways to compare, so we are obliged to live with qualitative comparisons. If a one-score comparison is not enough, the right way to go is to provide multiple scores. If direct metrics are not feasible, one should go for indirect ones, but should still provide outcomes a reader can reproduce.


Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. We propose new measures for calibration, the Static Calibration Error (SCE) and Adaptive Calibration Error (ACE). These measures taks into account every prediction made by a model, in contrast to the popular Expected Calibration Error.

Paper summary: This paper proposes two simple extensions to Expected Calibration Error (ECE):  1) SCE which accounts for multiclass classification settings by averaging over all the errors due to all classes (as opposed to error in the top one class only in ECE) and 2) ACE which attempts as distributing the predictions equally across the bins (as opposed to have too few highly populated bins in the interval). Authors evaluated their approach using ResNet-110 on CIFAR100 and ResNet-50 on ImageNet against most common post-training calibration methods.
Pros:
(+): The paper is well-motivated.
(+): The problem is important and has direct real world applications.
(+): The idea is simple and viable to improve ECE.
Cons that significantly affected my score and resulted in rejecting the paper are as follows:
1 - Experimental setting and evaluations:
The biggest drawback in this paper is the experimental setting which is not rigorous enough to show the effectiveness of the proposed metrics due to the following reasons:
(a) The first proposed extension (SCE) is too incremental. However, while I am not against simple and effective methods (I have even listed this as a pro above), I think it should be backed up with more thorough experiments and discussions. SCE is supposed to be more effective in case of having more number of classes however authors do not shed light into this properly by taking advantage of comparing their CIFAR100 and ImageNet 1K experiments. They keep emphasizing on the fact that the numbers provided in Table 1 and 2 are not comparable so how is the reader supposed to understand their differences? They also show their metrics give far less error to models than ECE in Table 1 and 2 and yet it is not clear why that is. Not to mention that authors need to state these results are obtained on how many runs and to report stds because the numbers appear too close and are hence not conclusive.
(b) Datasets and architectures: Authors have used MNIST and FASHION-MNIST when discussing the shortcomings of ECE. I was wondering what architecture they used for these experiments? The reason I am asking is that it is a known fact (see Guo et al., ICML17) that the simpler the architecture is, the more calibrated its predictions tend to be and vice versa. For instance LeNet 5 appears to be much more well-calibrated than more modern neural networks such as ResNet variants despite being less accurate (Guo et al, 2017). Therefore, it will probably be in their favor to show these issues on a less calibrated model.
(c) Can authors please explain why they have used ResNet-110 on CIFAR100 and ResNet-50 on ImageNet? I assume it might be because they intended to compare to (Guo et al, 2017) but there are more experiments there for comparison. Have they also tried any other architecture? It has been shown before that there can be a noticeable difference across different architectures used on a fixed dataset. Authors may want to add the arch effect to their evaluation.
2) The second metric (ACE) while it is well-motivated, it is defined such that it leaves the impression that applying it will involve lots of heuristics as there is no systematic procedure is given. In section 6.3 (where I think could be a section to address the implementation details for this) no effective information is given either expect that authors recommend using 50+ bins for CIFAR and ImageNet without any quantitative support. It is also not clear why authors used 15 bins in their experiments for these datasets in Table 1 and 2 (maybe comparing to the baseline? but they could show both specially if 50+ bins is better for ACE). Moreover, in section 5.2 where it says “the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions).“, I was wondering if authors have any suggestions on how to identify the predictions with low confidence as coming from in or out-distribution?
2- Structure of the paper, writing, and visualizations:
(a) Writing: The paper, in its current form, needs to be thoroughly proofread and reorganized. The text does not read well and is vague in most parts. Authors have spent too much time explaining the drawbacks of the prior work in 4 pages (entire sections 2,3, and 4) only to propose their ideas in section 5.1 and 5.2 in almost half a page. Unfortunately the analysis section (6) acts very poorly in providing a thorough exploration into their method.
(b) Dividing each section to too many subsections has hurt the flow as each section is too short and does not provide deep evidence to support its title/subtitle.
<<< Note that the followings are less major and are given only to help, and not necessarily part of my decision assessment >>>
(c) Results shown in Table 3 and 4 are reporting “TACE” which is technically explained in the Appendix. Apart from the fact that authors need to make sure the Table comes before the references (not in the middle) but more importantly, as a paper submitted to ICLR I would expect it to be self-contained and be able to provide all the details needed. Authors should either move this Table to the appendix or move section (A.1) to the main text.
(d) In section 3.4. Authors provide support for their claim on issues with binning scheme with a screenshot of their code in the Appendix. I think it is important to show this effect but in perhaps a table with quantitative results within the main text.
(e) Title for Section 5 as the main contribution of this paper should be changed to reflect that.
(f) Abstract in its current wording, does not provide absolute no detail into the proposed metrics. Whereas it can, because 1) it encourages the reader to keep reading 2) the method is simple enough to be summarized here.
(g) The figures do not meet the conventional scientific standards and have to be significantly improved. Standard deviations are missing.
(h) There are also grammar errors and typos, (for example on page 1 paragraph 5, the word "may" should be "many"), for which I have found passing my writing through the free version of Grammarly very helpful in getting rid of most such errors.
As a final note, I think this paper can be contributive to the field as it provides a novel and simple extension to a widely used calibration metric. However it needs to be written in a more effective way and be supported by a more rigorous experimental setting. Therefore, I will be willing to change my score if the presented issues will be addressed.


We propose NovoGrad, an adaptive stochastic gradient descent method with layerwise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well-tuned SGD with momentum, Adam, and AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has half the memory footprint of Adam.

I am not sure about my assessment of this paper.
The authors propose their approach as a mixture of two other approaches. Then, they proceed with an illustrative 2-D example where they apply the same hyperparameters for all tested methods. This does not seem appropriate especially given that later in the paper they show that SGD with momentum and Adam use very different hyperparameter settings such as learning rate and weight decay. Basically, SGD and NovoGrad perform very similar in Figure 6 and the difference might be due to a too large learning rate for SGD (and other approaches).
Increasing it further does not help as Figure 7 suggests. It is not the case for NovoGrad because of its gradient normalization. However, what the authors don't show us is a different experimental setup with some trivial objective function where it would take NovoGrad an enormous amount of steps to converge to the optimum IF the algorithm is initialized far enough from the optimum. Since NovoGrad is based on normalized gradients, its steps are pretty much just signs of the original gradient multiplied by learning rate, especially in 2D. Thus, with some initial learning rate of 0.1 and when initialized in say (1, 10^10), it would take NovoGrad about 10^9 steps to converge independently on the scale of the objective function. This is not the case for SGD whose gradients are not normalized. This is actually a strong counterargument to the main claim of the paper that "NovoGrad is robust to the choice of learning rate and weight initialization". This argument in the paper is based solely on the results of the toy problem where other methods used large learning rates.
The paper includes a set of experiments containing different methods and their hyperparameters.
When the authors use some cosine function to schedule learning rate in Table 1, they do it for AdamW and NovoGrad but they use polynomial schedule for SGD. Why? Weight decay and learning rate values for SGD and NovoGrad are very different in the same table, why? Do the authors explain why this difference may happen, they don't. I believe that there is a better answer than just "this is the result of hyperparameter tuning".
The authors note that when beta=0, NovoGrad becomes layer-wise NGD with decoupled weight decay. They suggest beta to be in [0.2, 0.5] and I am wondering which beta were used in different experiment (the default suggested value is not given).
The experimental results suggest that NovoGrad performs slightly better than SGD with momentum but it has more hyperparameters and we don't know whether the results are due to different computation efforts in hyperparameter tuning. Figure 5 compares Adam and Novograd on WikiTex-103 and shows that Adam converges faster in terms training perplexity. However, Adam's test perplexity is worse than the one of NovoGrad. Interestingly, NovoGrad's test perplexity is better than its training perplexity especially in the beginning. It seems that NovoGrad is good here because it does not converge well, i.e., the results are likely due to regularization which is problematic in the original Adam.


Model-based reinforcement learning (MBRL) has shown its advantages in sampleefficiency over model-free reinforcement learning (MFRL). Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. In this paper, we propose a simple and elegant model-based reinforcement learning algorithm called soft stochastic value gradient method (S2VG). S2VG combines the merits of the maximum-entropy reinforcement learning and MBRL, and exploits both real and imaginary data. In particular, we embed the model in the policy training and learn Q and V functions from the real (or imaginary) data set. Such embedding enables us to compute an analytic policy gradient through the back-propagation rather than the likelihood-ratio estimation, which can reduce the variance of the gradient estimation. We name our algorithm Soft Stochastic Value Gradient method to indicate its connection with the well-known stochastic value gradient method in .

This paper presents a slightly new model-based reinforcement learning algorithm, claiming that the new method is elegant and combines the merits of earlier methods. In particular, it claims that the new method is more sample efficient and computationally efficient than previous methods.The novelty of the new method has to do with how the model participates in “the policy improvement step”. The paper does not achieve a high level of quality in its presentation or empirical results, as i discuss further below, and as a result I do not consider it to make a scientific contribution, and I recommend rejection.
The presentation of the method is not clear; it uses an informal notation and leaves out many steps; in many cases the statements are not formally correct. I believe that many of these weaknesses are not introduced in this work but are present also in the works that it builds on. It is possible that someone intimately familiar with the previous works would be able to fully understand this method, but perhaps not, and certainly it does not stand alone.
The new method is presented in Section 3. Equation (3) defines an objective for the policy parameter. It uses things called r-hat, \rho-hat, p, and f. It is not clear whether these things are functions, random variables, or distributions. What is the domain of f? Is r(s,a) a random variable? Is r(s1,a1) different from r(s2,a2) if s1=s2 and a1=a2? Do they have the same distribution? This is just scratching the surface. Many, many things are not well defined, and some critical things, like V, appear to be defined multiple times.
The new method is ultimately presented in Algorithm 1, but it is just a sketch, with essential things left out. What is an “iteration”? Where are the “epochs” discussed in the empirical results? Many steps ask us to calculate gradients with respect to objective functions given by equations in the text. Almost all of these are unclear. To literally calculate the gradient of the expected values would involve a full sweep over the replay buffer, which appears never to be emptied, and thus would grow without bound. It this really what is meant.
There are also many problems with the empirical results. To begin with, there is no discussion of how the many hyper-parameters were set. And the statistics are not present.  The results present means and standard deviations. The paper does not appear to say how many runs were done, but from the wide variation in the standard deviations, it would appear that there were not many. The very basics of a valid experiment are thus missing. We really should not conclude anything from such results, but if they are published, then many readers would conclude things. Our field should not go any further this way, in my opinion.


Efficient and robust policy transfer remains a key challenge in reinforcement learning. Policy transfer through warm initialization, imitation, or interacting over a large set of agents with randomized instances, have been commonly applied to solve a variety of Reinforcement Learning (RL) tasks. However, this is far from how behavior transfer happens in the biological world: Humans and animals are able to quickly adapt the learned behaviors between similar tasks and learn new skills when presented with new situations. Here we seek to answer the question: Will learning to combine adaptation reward with environmental reward lead to a more efficient transfer of policies between domains? We introduce a principled mechanism that can "Adapt-to-Learn", that is adapt the source policy to learn to solve a target task with significant transition differences and uncertainties. We show through theory and experiments that our method leads to a significantly reduced sample complexity of transferring the policies between the tasks.

Summary
-------
The authors propose an algorithm for transferring policies across domains differing in their transition model. The idea is to "regularize" the target policy being learned to generate a trajectory distribution similar to the one induced by the source optimal policy in the respective MDP. The method is proved effective in standard simulated robotic tasks (Mujoco).
Major comments
--------------
The problem of transfer in RL is fundamental for scaling RL algorithms to real domains. The presented ideas are interesting and could potentially inspire other works in this field. The paper is well-written and easy to read. However, I believe there might be some technical issues with the derivations that overall make me doubt about the significance of this work. My detailed comments follow.
1. Under Assumption 1, the step from (6) to (7) seems flawed. The definition of the KL divergence requires that p is absolutely continuous w.r.t. q. However, under Ass. 1, whenever the trajectory \tau contains an action that is not taken by \pi^*, q(\tau) = 0, which implies that p(\tau) must be zero as well. The only way to enforce this is to set \pi_\theta = \pi^*, and the solution is trivial. The main issue is that (6) writes the log term of the KL using different action random variables for the numerator and the denominator (a and a'), but these must be the same. Therefore, one cannot simply set to 1 the optimal source actions. Notice also that the action space is defined continuous, hence the value of the pdf in the chosen deterministic action is infinity, not 1.
2. (12) seems to be wrong too. Note that the reward function defined here (r') contains the policy, hence it is a function of \theta and must be differentiated. So the standard REINFORCE estimator written here does not work theoretically.
3. Assumption 2 basically requires that either we can get new samples from the source domain "on-demand", or we have a good simulated model. Both assumptions are often violated in practice. Could the authors elaborate more on why they believe these assumptions to be reasonable?
4. Related to the previous point, assuming the one-step transition models to be Gaussian is convenient, and often reasonable, in practice. But how do we choose the std \sigma? What if the true models are deterministic?
5. I would disagree with the last sentence of section 3.2.1, which states that minimizing the KL divergence is more sample efficient than standard RL/policy-search methods. One can always define an MDP where the immediate reward is the KL term and that is a standard RL problem. Furthermore, if we compute the expected return when the reward is (8), decomposing the log into the sum of two logs, one of the two expectations we obtain is of the form: E_\pi[-(s_{t+1} -s'_{t+1})^2], which is a standard "tracking" problem.
6. I was quite confused by the theoretical section. The introduction states that the proposed method enjoys a sample complexity of O(nH), where I guess n is the number of state-action pairs. Where does that dependence appear in Section 4? Theorem 4.1 is a standard supervised learning bound. How do we read that in RL? What is C? Is |\Pi| the cardinality of the set of policies (assumed finite)? Or is it the pseudo-dimension of the policy space? Lemma 4.3 is also a know result in the literature, but what does it tell us about the proposed approach? The idea is that if I have an accurate model \hat{p} I am also close in value function to the true MDP. But what is \hat{p} here? The proposed algorithm is not computing any explicit transition model.
Minor comments
--------------
- In the last sentence of the introduction, it is not clear what n is in O(nH)
- In the preliminaries, is the reward R: S x A -> R_+ assumed to be positive or is that a typo?
- MDPs are defined with a discount factor \gamma which is however missing in (1) and the next equation (V)
- In the preliminaries, it is not clear whether the rewards differ or not between domains. It is important to specify that they do not.
- In (2), T is defined to map R^n -> R^n. What is n? Is it the number of discrete states? Wasn't the state space continuos?
- I noticed that many sentences in the paper begin without a space after the period of the previous sentence.
- In section 3, the KL divergence (e.g., eq. 3,7) is written as an expectation that contains the pdf p(\tau). Either only the log term is written in the expectation, or E is replaced by an integral.
- In (12), the sum should be up to H and not infinity


The natural world consists of objects that we perceive as persistent in space and time, even though these objects appear, disappear and reappear in our field of view as we move. This can be attributed to our notion of object persistence -our knowledge that objects typically continue to exist, even if we can no longer see them -and our ability to track objects. Drawing inspiration from the psychology literature on 'sticky indices', we propose the AlignNet, a model that learns to assign unique indices to new objects when they first appear and reassign the index to subsequent instances of that object. By introducing a persistent object-based memory, the AlignNet may be used to keep track of objects across time, even if they disappear and reappear later. We implement the AlignNet as a graph network applied to a bipartite graph, in which the input nodes are objects from two sets that we wish to align. The network is trained to predict the edges which connect two instances of the same object across sets. The model is also capable of identifying when there are no matches and dealing with these cases. We perform experiments to show the model's ability to deal with the appearance, disappearance and reappearance of objects. Additionally, we demonstrate how a persistent object-based memory can help solve question-answering problems in a partially observable environment.

This paper presents a model that is able to associate objects seen in a new time step with the objects seen in previous frames and therefore consider the cases of adding new objects, dropping object no longer visible, but keeping them in memory in case they reappear in the future. The model assumes that the location of the objects is given and the only task to perform is the correct association of instances. Results shows that the proposed approach is better than hand-crafted features on different simulated tasks. Additionally the propose model is shown to help to reduce the computational cost for question answering task.
I lean to reject this paper because in my opinion the proposed method is just a set of predefined rules to train a network to be able to perform object association between a new time step and a memory. Additionally the experimental evaluation is very weak in several points (see below).
- Contribution: this approach proposes a set of rules to train a network to be able to learn the correct association  between two set of features. Additionally, the network is used together with a memory to keep track of previously seen objects. In my understanding the proposed work presents a set of hard-coded rules to train a network for tracking. However, the connection with tracking is presented only in related works (at the end of the paper) and no comparison with other tracking approach is presented. Additionally, the network model that is actually used for the experiments (graph network) is not presented in the main paper, which makes more difficult to understand how the network works for the given task.
- Experimental evaluation: Experiments are performed on a very simple, simulated environment. Objects are very simple to recognize and their location (which is often  a difficult component of the problem) is given. As the task is quite straight forward, the model obtains almost 100% on all tests. Comparisons are made only with hand-crafted features. It is quite evident that a learned similarity measure will be better than hand-crafted distances.
Authors should explain more clearly that the proposed contribution is a set of rules used to learn a distance between features in order to associate object instances.
A cleared connection with tracking should be provided from the introduction. In the evaluation there should be a comparison on multiple and more challenging tracking datasets. In this way we can compare the proposed training technique with other tracking approaches that also learn to a distance in order to associate object at different time steps.


Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-k sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top-k operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-k) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during training process through extensive experiments. We then theoretically derive a tighter bound for the Top-k operator. Finally, we exploit the property of gradient distribution to propose an approximate top-k selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead.

This paper makes two contributions to gradient sparsification to reduce the communication bottleneck in distributed SGD.
1) Based on an assumption on the distribution of gradient coordinate values that are backed by an empirical study, the paper derives a tighter bound on the approximation quality of top-k gradient sparsification. This result induces better convergence bounds.
2) The authors note that the top-k cannot benefit from the highly parallel architectures popular in ML, and propose an approximate top-k sparsification operator. This operator tries up to three thresholds and checks how many entries are larger than this value. The initial guess is based on approximating the distribution of gradient coordinates with a normal distribution again.
My score is weak reject. I believe that both observations are valid, and that their solutions might be practically meaningful. However, I would like to see the comparison to another baselines [1]. I would also urge the authors to make it more clear that their theoretical results are based on a strong assumption on the distribution of gradients. The top-k approximation algorithm is practical but I find 3-step threshold search inelegant.
Comments:
1) [1] is another baseline -- compare your method with it too.
2) 4.3 Convergence performance "operator can select close elements with Top_k" --- It seems obvious that it can select similar elements. The question is whether the number of elements chosen is accurate. I would like to see this evaluated. It is unclear if this scheme is biased. As far as I can see, it might be over- or under-sparsifying.
3) 3.1 Gradient Distribution "One can easily prove" --- please do so (in the appendix)
4) Theorem 1 - Looks like this can't be true in general. I think it assumes d -> infinity.
5) 3.1 Gradient Distribution "then pi is a decreasing function" --- should this be pi^2. Also in figure 3, the result of Eqn 7 is only correct if the curve if pi^2.
6) Figure 2: I am not convinced that these distributions are 'gaussian'. In fact, they seem peakier. It seems to me that this should improve the results (i.e. make the descending pi^2 curve more convex). If this is true, I would encourage the authors to discuss this. BTW, the distribution is in terms of the whole model, or just one randomly picked layer?
7) Conclusion "theoretically tighter bound" --- because the assumption on the distribution empirical, I find it slightly misleading to call this a 'theoretical bound'. I would urge the authors to make this very clear. (This does not mean I find the bound meaningless)
8) Introduction: "O(d), which generally limits the system scalability" --- The O(d) does not explain scalability in terms of number of workers as is suggested. Note that even though bandwidth scales with O(d) in all reduce, the latency does scale with n. This should not be ignored.
9) Related work/Gradient Sparsification --- Please add a reference for empirical success of top-k. I am not aware of much use outside of academia.
10) Quite a few language errors (some paragraphs/sentences don't make sense at all and there are many cases with missing 'a's etc.)
11) Some of the experimental details are missing
    - what are the learning rates/batch sizes used in experiments.
    - How topK is performed? Layer-wise or for the full gradient.
    - Table 1 "experimental settings": how these values were chosen.
    - Table 2 --- please define how scaling efficiency is computed.
    - Table 2 --- Do these algorithms achieve the same validation accuracy during the training?
    - Figure 1: which k was used in these plots?
12) Figure 6: in VGG-16, the gap of 1 percentage point is quite large. This seems expected as the compression ratio is very high (1000x).
13) Figure 6: Imagenet training scheme is not standard. SOTA validation accuracy for the Imagenet benchmark with Resnet50 is around 76%. Would it also have similar quality loss on later stages as in training VGG or ResNet20 on cifar?
14) Eqn. 8 - I couldn't follow the first inequality.
15) Introdcution: The first few times "distribution of gradients" is mentioned, it was unclear to me if this was over 'coordinates' (as it seems to be), or over 'data points', or 'training time'. Please clarify.
[1] Jiarui Fang, Cho-Jui Hsieh. Accelerating Distributed Deep Learning Training with Gradient Compression.


Guarantees in deep learning are hard to achieve due to the interplay of flexible modeling schemes and complex tasks. Invertible neural networks (INNs), however, provide several mathematical guarantees by design, such as the ability to approximate non-linear diffeomorphisms. One less studied advantage of INNs is that they enable the design of bi-Lipschitz functions. This property has been used implicitly by various works to design generative models, memory-saving gradient computation, regularize classifiers, and solve inverse problems. In this work, we study Lipschitz constants of invertible architectures in order to investigate guarantees on stability of their inverse and forward mapping. Our analysis reveals that commonly-used INN building blocks can easily become noninvertible, leading to questionable "exact" log likelihood computations and training difficulties. We make use of numerical analysis tools to diagnose non-invertibility in practice. Finally, based on our theoretical analysis, we show how to guarantee numerical invertibility for one of the most common INN architectures.

The paper claims that for invertible neural networks, mathematical guarantees on invertibility is not enough, and we also require numerical invertibility. To this end, the lipschitz constants/condition numbers of Jacobians of both the forward and inverse maps of invertible NNs based on coupling layers are examined mathematically and experimentally. The paper also displays cases that expose non-invertibility in these architectures via gradient-based construction of adversarial inputs, as well as a decorrelation benchmark task, and show that spectral normalization can be a remedy for stabilizing these flows.
I think it’s a good point that we need to monitor the Lipschitz constant/bounds of both directions of these invertible functions. It’s true that the focus for stabilising NNs by bounding Lipschitz constants was always on the forward function, and for invertible functions we should also ensure that the inverse is numerically stable to compute.
The mathematical contribution of the paper is twofold - 1. deriving bounds on the lipschitz constants of the forward and inverse mapping of additive/affine coupling blocks 2. summarising known lipschitz bounds of forward and inverse mappings of other invertible layers (iResNet, neuralODE, invertible 1x1 convolutions etc). The main contribution lies in 1, and the derivation for the additive coupling block (volume preserving) is neat (although fairly straightforward), but the derivation for the affine coupling layer (NVP) is not useful nor insightful; they are local Lipschitz bounds (so require bounds on all intermediate activations, which is difficult as pointed out by the authors), and the numerical value of this bound was not used at all in relation to the numerical experiments - I imagine the bound is loose. Given that it seems difficult to find a tight global lipschitz bound, I think it would be more insightful to compute a lower bound to the lipschitz constant of the model (with fixed parameter values) by maximising the spectral norm of the Jacobian with respect to the inputs (or outputs if looking at the inverse map) - this will yield a lower bound by Lemma 3. This will be numerical, but more informative since it will give you an indication of where in the input space (or output space if looking at the inverse) there could be numerical instabilities. Also I think the bound on the local lipschitz constant of the inverse for the affine coupling block might be incorrect, because in A.1.1, the inverse map is F^{-1}(y)_I1 = y_I1, F^{-1}(y)_I2 = (y_I2 - t(y_I1))/g(s(y_I1)), so the scale and shift is s’(y_I1) := 1/g(s(y_I1)) and t’(y_I1):=- t(y_I1)/g(s(y_I1)), and hence I think this needs to be taken into account for computing the lipschitz bound of the inverse
I have mixed feelings about the experimental section. In section 4.1, it is interesting to see that we can find inputs where trained flow models can show numerical non-invertibility, evident in the poor reconstructions. It would be a nice addition to investigate whether this is coming from the forward function or its inverse, by examining the norm of the Jacobian of F and F^{-1} at the input x_delta and output F(x_delta) respectively.
However, the decorrelation task introduced in section 4.2 is puzzling. I don’t understand why for these invertible models, you are investigating invertibility for parameter values trained to decorrelate, as opposed to parameter values used in the usual task of density estimation with flows (or any other standard application of invertible NNs). The two reasons given in the paper are that 1) decorrelation is a simpler task and 2) it allows both stable and unstable transforms as solutions, but these are not convincing. Point 2) holds for flow-based density estimation as well, and regarding point 1), density estimation is the task we usually care about when using invertible NNs, and this is also computationally plausible/tractable, whereas even if decorrelation is a simpler task, it’s not a task that users of invertible NNs are interested in. It is good to know that these invertible NN architectures CAN admit values that are numerically non-invertible, but I would be much more interested to know whether this actually holds when they have been trained for flow-based density estimation. I’m not sure whether the experimental results on models trained for the decorrelation task are useful, because a model that is stable when trained for the decorrelation task may be unstable when trained for flow-based density estimation and vice versa. The observation that spectral normalization can help address numerical instability is useful, but from the perspective of someone who wants to use these invertible NNs for density estimation, I would like to know what is the sacrifice in expressivity/validation performance (if any) when using spectral normalization in these invertible architectures. Also, the results would be more relevant if the architectures resembled the architectures used for invertible models used in the literature (e.g. GLOW) where we not only have coupling layers but they are interleaved with PLU linear flows.
In section 5, the result that Flow-GANs can be numerically non-invertible is more relevant, and it is useful to know that spectral normalisation can help resolve this issue, but again it would be useful to quantify whether this comes at the cost of the quality of generated samples (Figure 3 shows several samples, but a more thorough quantitative & qualitative comparison would be welcome). Also regarding the point about likelihood in Section 5, where the authors state “it cannot be trusted as true likelihood due to lack of invertibility”, I think it should be emphasised that this point holds specifically for flow-GANs where for F: z -> x, you need a numerically accurate F^{-1} to compute the density, but for standard flow-based density estimation where F:x -> z, you never need to compute the inverse for computing the likelihood, hence if F has a small lipschitz constant then the likelihood will be accurate, regardless of whether the inverse is numerically stable or not.
Overall I believe the experimental section can be largely improved, and given that the motivation of the paper is nice and the paper is clearly written and nicely presented, it would be a shame to leave the experiment section as it is.
Minor typos/Qs:
p2: this problems <- this problem
p8: and with maximum likelihood (ML) - should this be removed?
p13: t(x_I2) <- t(x_I1)


We propose a method to impose homogeneous linear inequality constraints of the form Ax ≤ 0 on neural network activations. The proposed method allows a datadriven training approach to be combined with modeling prior knowledge about the task. One way to achieve this task is by means of a projection step at test time after unconstrained training. However, this is an expensive operation. By directly incorporating the constraints into the architecture, we can significantly speed-up inference at test time; for instance, our experiments show a speed-up of up to two orders of magnitude over a projection method. Our algorithm computes a suitable parameterization of the feasible set at initialization and uses standard variants of stochastic gradient descent to find solutions to the constrained network. Thus, the modeling constraints are always satisfied during training. Crucially, our approach avoids to solve an optimization problem at each training step or to manually trade-off data and constraint fidelity with additional hyperparameters. We consider constrained generative modeling as an important application domain and experimentally demonstrate the proposed method by constraining a variational autoencoder.

This paper proposes a method to impose linear inequality constraints on neural network activations. The method is implemented at initialization (by converting the H-representation to the V-representation) and during training (by modifying the network architecture). Experiments on two setups (projection and VAE+projection on a checkerboard pattern on MNIST) demonstrate a 2-orders of magnitude speed-up with respect to test-time projection (computed with OSQP).
The contributions claimed are:
* Novel technique to impose inequality constraints on neural network activations.
* Significant speed-up w.r.t. other techniques (at test time).
Overall, the approach is well motivated, and well-placed in the literature. However, the experimental analysis does not support all the claims made by the authors as it focuses on a single dataset (i.e., MNIST) and single constraint (i.e., checkerboard pattern). Additionally, while the authors argue that there is no manual trade-off between constraint satisfaction and data representation, the experiment in Fig. 3 appears to show that there is some manual trade-off (using box delay). As such, I am currently inclined to give a "weak reject" score.
The method is clear and the idea of using softmax to get a convex combination of the vertices of the V-representation to guarantee constraint satisfaction is reasonable.
1) The softmax used to satisfy the constraints is preceded by a batch normalization layer. I would expect batch normalization to interfere with the ability to saturate the softmax activation and thus prevent the network from reaching optimality. Could the authors provide experiments that justify the use of the batch normalization layer?
2) An important factor that is unexplored in this manuscript is the softmax temperature. A good scheduling of that temperature could help the optimization. Have authors tried different temperature values?
3) The use of softmax and the integration of the constraints as a network layer seem to create some difficulty during training (even with the rather simple checkerboard pattern used in the experiment). The loss appears to reach some plateau (9% from optimal) and, thus, there appears to be some trade-off between reconstruction and projection. The authors should provide more experiments to explain that trade-off. That trade-off is also visible on Fig. 5 (the zero has a significantly different shape).
4) The method is solely compared to test time projection. Could the authors implement other techniques (if feasible)? e.g., OptNet. Overall, it would helpful to add more setups and different types of constraints (other than a last-layer projection; e.g., monotonicity).


Federated learning involves training statistical models in massive, heterogeneous networks. Naively minimizing an aggregate loss function in such a network may disproportionately advantage or disadvantage some of the devices. In this work, we propose q-Fair Federated Learning (q-FFL), a novel optimization objective inspired by fair resource allocation in wireless networks that encourages a more fair (i.e., more uniform) accuracy distribution across devices in federated networks. To solve q-FFL, we devise a communication-efficient method, q-FedAvg, that is suited to federated networks. We validate both the effectiveness of q-FFL and the efficiency of q-FedAvg on a suite of federated datasets with both convex and non-convex models, and show that q-FFL (along with q-FedAvg) outperforms existing baselines in terms of the resulting fairness, flexibility, and efficiency.

The problem of fairness in federated learning (FL) is important given the popularity of the topic and its immediate impact on the society. Vanilla FL approaches may be subject to poor performance for clients whose data is under-represented across all participants. This paper proposes a new algorithm for federated learning to reduce variance in performance across clients. The inspiration for the algorithm comes from the problem of uniform  resource allocation in wireless networks.
While the problem and the motivation for the algorithm are interesting on the high level, I think this paper does not deliver the key ideas in sufficient detail and clarity.
On the algorithms side, I am still unclear on how the Lipschitz constant L is estimated on the first run with q=0. Are the results for q=0 in the experiments reported for this run or is it repeated with the learned L? Further, this procedure suggests that the number of communication rounds is at least doubled for the end-to-end training. Tuning q, which seems to be necessary, may require even more communication rounds.
While there are a lot of experiments in the paper (across main text and supplementary), none seem to be carried out sufficiently well. Understanding the complete experimental setup for at least one of them is also quite hard due to numerous supplementary references throughout the experiments section. I would recommend to focus on fewer experiments, but present more thorough results. Below are some suggestions.
The importance of resource allocation in FL appears to me to be directly related to the key FL aspects such as degree of data heterogeneity and number of clients. This submission is lacking experiments comparing FedAvg to the proposed method under these settings (which can be simulated using available datasets). To argue in favor of the proposed approach it is important to demonstrate failure modes of the existing algorithms under some realistic scenarios and present a solution using new algorithm.
Accuracies in Fashion MNIST and Shakespeare experiments seem quite poor suggesting some problems with the setup. FedAvg paper reports 54% on Shakespeare, whereas this paper reports 52%. It also appears that the number of considered "devices" on Shakespeare is significantly smaller than in the FedAvg paper (31 vs 1146) - what is the reason for this?
On Fashion MNIST, AFL paper reports 80%+ accuracy while achieving 90%+ on the combined dataset seems relative easy based on the results mentioned on the Github repository of the dataset. This paper reports 78% for the proposed method and AFL. Why is there a discrepancy with AFL paper and what is the performance of FedAvg on this dataset (assuming some suitable CNN architecture)? Is there a reason to believe that this dataset is much harder for federated learning than MNIST, where FedAvg roughly matches full data training?
This statement is ambiguous "uniform sampling is a static method and can easily overfit to devices with very few data points, whereas q-FFL has better generalization properties due to its dynamic nature." If there is a device with very few data points it is easy to overfit to it and q-FFL will essentially ignore that device since the loss on this device is very small. Why does this not lead to more severe overfitting behavior?