We introduce a novel autoencoder model that deviates from traditional autoencoders by using the full latent vector to independently modulate each layer in the decoder. We demonstrate how such an 'automodulator' allows for a principled approach to enforce latent space disentanglement, mixing of latent codes, and a straightforward way to utilize prior information that can be construed as a scale-specific invariance. Unlike the GAN models without encoders, autoencoder models can directly operate on new real input samples. This makes our model directly suitable for applications involving real-world inputs. As the architectural backbone, we extend recent generative autoencoder models that retain input identity and image sharpness at high resolutions better than VAEs. We show that our model achieves state-of-the-art latent space disentanglement and achieves high quality and diversity of output samples, as well as faithfulness of reconstructions.

The submission proposes an autoencoder architecture which combines two recent GAN-based architectural innovations, namely the progressive growing of the decoder architecture (as well as the encoder architecture in this case) and the use of the encoded representation to modulate the decoder via a feature-wise transformation mechanism.
I think the overall idea behind the paper is valuable, but I don’t think the submission meets the acceptance bar from a clarity point of view. I also have concerns with its characterization of the literature.
Clarity-related comments:
- “This allows the layers to work independently of each other.” This is an imprecise use of the term “independently”. How can two layers work independently if one’s input is the output of the other?
- “The reconstructed samples could be re-introduced to the encoder, repeating the process, and requiring consistency between passes.” The rest of the paragraph is built on the premise that this is a desirable property, but I’m not sure I understand why this is a desirable property in the first place.
- How to define “disentanglement” in the context of representation learning is in itself an unsettled question as far as I’m aware, but the submission uses the term without an intuitive or formal definition. What do the authors mean by “disentangled representation”? What is measured by perceptual path length (PPL), and in which ways does PPL relate to the author’s definition of “disentangled representation”?
- “[...] a new autoencoder-like model with powerful properties not found in regular autoencoders, including style transfer.” The term “style transfer” is overloaded; what do the authors mean?
- The submission defines AdaIn as a way to combine “content” and “style”, and defines the style “y” in terms of mean and variance. In the context of the AdaIn paper, this makes sense: the instance normalization shifting and scaling coefficients are heuristically defined as the channel-wise means and standard deviations of a “style” stack of feature maps. However, in the context of this submission I’m not sure this definition makes as much sense: the instance normalization shifting and scaling coefficients are the result of a linear projection of the latent representation and do not involve the channel-wise means and standard deviations of an external stack of feature maps; is this correct?
- “This setup follows the same logic as that of Karras et al. (2019), but we do not require an ad-hoc disentanglement stack.” Can the authors clarify what they mean by an “ad-hoc disentanglement stack”?
- Section 3.2 uses some notation for the encoder without introducing it first. I believe the only way to understand that \phi(x) refers to the encoder network is to look at Figure 2.
- Section 3.2 as a whole is hard to follow, in part due to the use of imprecise language (“mutually independent”, “representation of those levels disentangled in z”). At some point probability distributions are introduced (up until now the reader is operating under the assumption that the model is an autoencoder with no probabilistic interpretation), and mutual information is mentioned to justify an L2 reconstruction loss in z-space (which I would argue is an instance of mathiness that does not serve the reader’s comprehension). Can the authors explain in plain language how layer-specific losses are defined and how the complete loss is obtained?
- The submission presents model samples, but as far as I can tell the procedure for sampling is not provided. Unlike VAEs, autoencoders do not explicitly model the empirical distribution -- although reconstruction in denoising autoencoders is related to the score of the empirical distribution (Alain & Bengio, 2014). How are samples obtained from the trained model?
Literature-related comments:
- The use of normalization layers to implement feature-wise transformation mechanisms is fairly widespread nowadays, but for instance normalization specifically the work of Dumoulin et al. (2017) pre-dates that of Huang & Belongie (2017). Both are cited by Karras et al. (2019) in relation to AdaIn (which is termed “conditional instance normalization” in Dumoulin et al. (2017)).
- I disagree with the characterization of ALI/BiGAN as “hybrid models that combine the properties of VAEs and GANs”: unlike AAE and AVB, which minimize KL-divergence terms in the VAE loss adversarially, the objective for ALI/BiGAN is purely adversarial. I would also include IAE (Makhzani et al., 2018) and BigBiGAN (Donahue et al., 2019) in the list of GAN variants that incorporate an inference mechanism.
- The submission repeatedly asserts that GANs lack an inference mechanism: “Unlike GANs, autoencoder models can directly operate on input samples.”; “To work on new input images, GANs either need to be extended with a separate encoder, or inverted [...].”; “[...] GANs show good image quality, but have no built-in encoding mechanism [...]”; “[...] the problem with GANs is that they lack the encoder [...]”. This is false: see for example ALI, BiGAN, and BigBiGAN. The problem in my opinion is elsewhere: the kinds of reconstructions these models yield are not suited to the downstream applications investigated in this submission, because they oftentimes fail to preserve low-level details.
References:
- Alain, G., & Bengio, Y. (2014). What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1), 3563-3593.
- Dumoulin, V., Shlens, J., & Kudlur, M. (2017). A learned representation for artistic style. In Proceedings of the International Conference on Learning Representations.
- Makhzani, A. (2018). Implicit autoencoders. arXiv:1805.09804.
- Donahue, J., & Simonyan, K. (2019). Large scale adversarial representation learning. arXiv:1907.02544.


Recovering sparse conditional independence graphs from data is a fundamental problem in machine learning with wide applications. A popular formulation of the problem is an 1 regularized maximum likelihood estimation. Many convex optimization algorithms have been designed to solve this formulation to recover the graph structure. Recently, there is a surge of interest to learn algorithms directly based on data, and in this case, learn to map empirical covariance to the sparse precision matrix. However, it is a challenging task in this case, since the symmetric positive definiteness (SPD) and sparsity of the matrix are not easy to enforce in learned algorithms, and a direct mapping from data to precision matrix may contain many parameters. We propose a deep learning architecture, GLAD, which uses an Alternating Minimization (AM) algorithm as our model inductive bias, and learns the model parameters via supervised learning. We show that GLAD learns a very compact and effective model for recovering sparse graphs from data.

This paper proposes an approach to data driven edge recovery for sparse gaussian mrfs. The authors propose an AM procedure for solving the l1 regularized maximum likelihood which can be unrolled and parameterized. This method is shown to converge faster at inference time than other methods and it is also far more effective in terms of training time compared to an existing data driven method. The authors provide a theoretical analysis which explains how the AM procedure should succeed and some insights on how it can potentially converge better using an adaptive model (motivating the learning part).
Experiments:
Overall the experiments demonstrate the method is superior. I have a few comments/concerns however:
-Scalability to larger graphs. The graphs used here a relatively small. I would like to see how well this scales to larger graphs at least in principal if not experimentally. Is there any issues that make this difficult? For example more iterations might be needed for convergence and this becomes problematic for learning. Can this already compete  with large scale methods like BigQUIC.
-Are the training graphs always the same distribution as the test graphs (e.g. in terms of the sparsity level)? It would be good to evaluate how well the model works when the training conditions differ to testing, since applying it to real data would require this gap.
-Closely related to the above if I have understood correctly all the experiments including the gene networks are on synthetic data, it would be good however to see if synthetic data can help generalize to real data.
- How many iterations are used to train the model? Is the number of iterations ever more at inference than training? It seems the NMSE is increasing after hitting a bottom I am wondering if that is related to mismatch in the number of iterations in test/train
- (minor) the authors compare wall clock time per iteration in Table 2 however their method converges much faster, it would be good to also see the overall clock time for each method after some reasonable stopping crieria, to show how big the overall gain is.
Related Work:
The overview of sparse graph recovery for Gaussian random variables is good and concise. However I found the high level motivations given in Introduction/Sec 3 are similar (at times even the wording) to those of Belilovsky et al 2017 which introduced/motivate the data driven approach to this problem. Although this reference is used in the experimental section, it would be appropriate to clarify the difference/contribution compared to this work in the Intro, Sec3, and/or related work as a naive reading of the paper incorrectly suggests it is the first to consider a data driven approach to this problem.
Other comments:
- In equation (9) \Theta^{*(i)} should there be an (i) index there, I suspect this is a typo but  if it is not can the authors explain how the target differs for different (i)
- The data generating process described in Sec 5, how does it assure SPD, the described procedure (sampling off diagonal entries U(-1,1) then randomly setting zeros) does not appear to me to assure SPD without further constraints.
- I appreciated Appendix C10/C11 the overview of other attempts to parametrize the inference procedure
Overall I found this work relevant, the formulation well motivated, and potentially of high impact for the community working on inference of sparse conditional independence structure. I would give the score at the moment between weak accept and accept. There is a few points which I would like the authors to clarify or correct in their rebuttal and I would be happy to increase my score.
-----Post Rebuttal
The authors have addressed my primary concerns and revised the text, I am thus increasing my score. I recommend the authors also itemize the main changes in the text from the initial manuscript as they are not easy to find right now using the openreview revision comparison system (which seems to be broken). For example its not clear if the results shown within text are in the new manuscript.


Previous work showed empirically that large neural networks can be significantly reduced in size while preserving their accuracy. Model compression became a central research topic, as it is crucial for deployment of neural networks on devices with limited computational and memory resources. The majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient, data-independent neural pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework, which finds a small weighted subset of points that provably approximates the original inputs. Specifically, we approximate the output of a layer of neurons by a coreset of neurons in the previous layer and discard the rest. We apply this framework in a layer-by-layer fashion from the top to the bottom. Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input x ∈ R d , including an adversarial one. We demonstrate the effectiveness of our method on popular network architectures. In particular, our coresets yield 90% compression of the LeNet-300-100 architecture on MNIST while improving classification accuracy.

------ after reading authors' response ------
Thanks for the complete response and revision. The new Fig 4 is very nice, and this helps address my concerns. I'm more favorable of the paper now, changing from "Weak Accept" to "Accept".
Note a small typo in the displayed equation just above Fig 4: the sum is over $P_{test}$ but it's written $P_{t}est$ (and also using \text makes it look nicer, $P_{\text{test}}$)
------- original review -----
The paper proposes a neuron pruning technique that can compress an existing pre-trained neural net (though the experiments actually do additional unspecified "fine-tuning" training). It is motivated by the need to compress neural nets so they work on embedded devices (smart phones, etc.), and it is in contrast to most other techniques that prune at training, or prune after-training but prune weights not nodes. They argue convincingly that pruning weights is awkward, as one has to work with sparse matrices which are only actually effective for extreme sparsity levels. They also claim another big benefit is that theirs is the first with with (1) provable guarantees, and (2) is data-independent.
I have a some criticism of the paper, but before I get lost in the details, let me say that I like the overall paper. I think it's a clever idea, it's a useful topic, the authors show very good understanding of the coreset literature, and it has some nice theory.  The paper is also well-written and easy to understand, and the appendix is short enough that I actually read it.
However, I have at least two major comments:
(1) The theorems are nice, but with the exception of Thm 6 (which I like), they are simple applications of existing results. My main issue is that you have not provided an end-to-end bound. There are two things lacking:
(1a) Lack of dealing with several layers, e.g., composing your approximation error. With an additive error instead of a relative error, does composition cause a major problem? Seems like this could be an easy theorem.
(1b) Lack of a clear final statement bounding the overall error. This is somewhat trivial (if you have a single pruned layer), but it makes the assumptions more clear. In particular, you assume the input x has norm bounded by beta. In this sense, you have not provided a "data-independent" guarantee.  Since you do not have a relative error bound, the norm of x is important.  Yet this also exposes something that really confuses me: for the ReLU activation, with non-negative inputs, this is positive homogeneous, i.e., phi( beta ||p|| ) = beta phi( ||p|| ). If you look at step 2 in Algo 2, you see that the choice of beta does not affect the probabilities (if phi is ReLU or anything else with this property). Thus we can choose beta arbitrarily... and thus you have a fixed additive bound, for an arbitrarily large input, which seems impossible!
(2) Experiments were very promising, but I'm not convinced about the baselines.
(2a) The fine-tuning after pruning wasn't described so I don't know how much effect it had. It makes sense to do this, but it means that it is less clear if your results were due to your theorem.  Please show results with and without the fine-tuning, and describe the fine-tuning (how many epochs of training?)
(2b) You do some abstract experiments with random weights, to test your theorem, which is a nice somewhat direct test of your results (I assume here you are not fine-tuning, as it doesn't make sense, right?). Also in the abstract setup, you could test this as a function of depth, since I'm worried that your error guarantees get worse as a function of depth. The experimental setup was vague: what are the inputs x (from a ball, or sphere? uniform?), was this averaged over many runs? What was this network (you change size when you go to the LeNet-300-100), especially, what was the 100% number of samples (1000?)?.
(2c) For table 3, taking the LeNet for example, you have 90% compression and improved error. This is nice, but to really convince me, in addition to adding the results without the fine-tuning, I'd like to see what you get with uniform pruning (say, with 85% compression) with and without fine-tuning. I don't have a good "baseline" expectation here, so while your improved error with 90% compression seems like a fantastic result, I suspect that one might get similar results (with say 85% compression) with trivial subsampling.
Some minor comments:
-- abstract, "guarantees the accuracy of the function" and "... on MNIST while improving the accuracy."  These are 2 very different meaning of "accuracy", so please be more precise, e.g., a per-layer approximation error vs classification error on testing data.
-- First paragraph of intro, saying networks are limited to HPC environments is hyperbole. These networks might need to be trained in an HPC environment, but most can be deployed on laptops (not an HPC environment). A bigger issue is deploying them on a smartphone.  Adding some quantitative numbers would strengthen your case (e.g., size of typical neural nets, and size of RAM in a smartphone).  Note that training requires much more memory due to memory explosion in backpropagation, but this does not effect runtime/deployment.
-- middle of page 2, "is very fast, ..." the comma should be a semicolon to make it grammatically correct.  Page 3, near bottom, "corests" is a typo.
-- personally I dislike things like Table 1, as they feel too much like boasting. You've chosen the columns carefully so it doesn't feel that meaningful, and you've already stated these things in the text. But it's not my paper.
-- def 4 is very hard to parse. Are these subsets or strict subsets?
-- Notation B_alpha could be explained; I'm used to seeing B_alpha(0), and you can always just write \forall x with ||x||<= alpha to make it super clear.
-- Your theorems require phi to be non-decreasing, but intuitively you can clearly handle non-increasing, since the set of inputs x \in B_beta is invariant to sign changes. More generally, you could assume the existence of some 1D function psi(t) such that phi( t ) <= psi( |t| ). I don't know if there are many more common activation functions, but this could give a wider class.  The change to the proofs is trivial, since you just replace psi( |t| ) for phi( alpha beta).
-- If Corollary 9 "follows directly from Theorem 5", why didn't Thm 7 and Corollary 8 also follow directly? You mean, it follows, but using the same simple bounding tricks from the appendix as used for Thm 7 and Corollary 8, right?
-- Fig 3 shows very nice results
-- A.1 Proof of Thm 6, you could use \subsetneq (with amssymb package) to be more clear that it is a strict subset, since I find \subset vague since different authors have different conventions.


Modeling dynamically-evolving, multi-relational graph data has received a surge of interests with the rapid growth of heterogeneous event data. However, predicting future events on such data requires global structure inference over time and the ability to integrate temporal and structural information, which are not yet well understood. We present Recurrent Event Network (RE-NET), a novel autoregressive architecture for modeling temporal sequences of multi-relational graphs (e.g., temporal knowledge graph), which can perform sequential, global structure inference over future time stamps to predict new events. RE-NET employs a recurrent event encoder to model the temporally conditioned joint probability distribution for the event sequences, and equips the event encoder with a neighborhood aggregator for modeling the concurrent events within a time window associated with each entity. We apply teacher forcing for model training over historical data, and infer graph sequences over future time stamps by sampling from the learned joint distribution in a sequential manner. We evaluate the proposed method via temporal link prediction on five public datasets. Extensive experiments 1 demonstrate the strength of RE-NET, especially on multi-step inference over future time stamps.

The paper proposes a recurrent and autorgressive architecture to model temporal knowledge graphs and perform multi-time-step inference in the form of future link prediction. Specifically, given a historical sequence of graphs at discrete time points, the authors build sequential probabilistic approach to infer the next graph using joint over all previous graphs factorized into conditional distributions of subject, relation and the objects. The model is parameterized by a recurrent architecture that employs a multi-step aggregation to capture information within the graph at particular time step. The authors also propose a sequential approach to perform multi-step inference. The proposed method is evaluated on the task of future link prediction across several baselines, both static and dynamic, and ablation analysis is provided to measure the effect of each component in the architecture.
The authors propose to model temporal knowledge graphs with the key contribution being the sequential inference and augmentation of RNN with multi-step aggregation. The paper is well written in most parts and provides adequate details with some exceptions. I appreciate the extended ablation analysis as it helps to segregate the effect of each component very clearly. However, there are several major concerns which makes this paper weaker:
- The paper approaches temporal knowledge graphs in discrete-time fashion where multiple events/edges are available at each time step. While this is intuitive, the authors fail to position the paper in light of various existing discrete-time approaches that focus on representation learning over evolving graphs [1,2,3,4,5]. Related work mentions [1] learns evolving representations but all these methods can do future link prediction and hence this is a big miss for the paper. A discussion and comparison with these approaches is certainly required as most of static and dynamic baselines currently compared also focus on learning representations, hence that is not a valid argument to miss comparison.
- The baselines tested by the authors are either support static graphs, supports interpolation or supports continuous time data. However, as the authors explicitly propose a discrete time model starting from Section 3, it is important to perform experiments on atleast few of the discrete time baselines to demonstrate the efficacy of the proposed method. For instance, authors can augment relation as extra feature or use their encoders and optimization function to perform experiments e.g. Evolve-GCN  only require to replace GCN with R-GCN.
- From the ablation it is clear that aggregation is the most important component as without it, the performance drops much closer to ConvE which is a static baseline and significantly worse than other RE-Net variants. However, the aggregation techniques are not novel contributions but augmentation to the RNN architecture. Hence it is important to show how augmenting aggregation module with other baselines (for instance, ConvE and TA-DistMult)) and the above mentioned discrete baselines would affect the performance of these baselines.
- While the authors describe attentive Pooling Aggregator, the experiments only show mean aggregator and multi-step one. Is there a reason Attentive pooling is not used for any experiments?
-It appears that global vector H_t is not playing significant role based on ablation study. Can the authors explain why that si the case? Also, what aggregation is used to compute H_t? Is it sum over all previous h_t's?
- Algorithm 1 is not very clearly explained. When the authors mention that they only use one sample, does that mean a single subject is sampled at each time point t'? If so, how do you ensure the coverage is good across subjects in the newly generated graph? I admit I am not clear on this and would recommend the authors to elaborate in response and also in the paper. Also, the inference computation complexity is concerning. While it seems fine for the provided dataset, most real-world graphs have billion of nodes and I all of E, L and D would be larger for such graphs. This seems to put a strict limitation on scalability of inference module.
- It is not clear what is the difference between RE-NET and RE-NET w. GT. Could the authors elaborate this more? It seems the authors do not update history when they perform RE-NET w/o multi-step. However, in the RE-NET w. GT, where is the ground truth history used in Algorithm 1?
- The time span expansion for WIKI and YAGO is very unnatural and it is not clear if these experiments provide any value. For instance, can the authors show that in multi-step inference scheme, they can actually predict events at multiple time points corresponding to time span events in actual dataset? As multiple triplets can appear at consecutive time points, the current modification just makes them equivalent which doesn't seem correct.
I am willing to revisit my score if the above concerns are appropriately addressed and requested experiments are provided.
[1] Evolve-GCN: Evolving Graph Convolutional Networks for Dynamic Graphs, Pareja et. al.
[2] DynGEM: Deep embedding method for dynamic graphs, Goyal et. al.
[3] dyngraph2vec: Capturing network dynamics using dynamic graph representation learning, Goyal et. al.
[4] Dynamic Network Embedding by Modeling Triadic Closure Process, Zhou et. al.
[5] Node Embedding over Temporal Graphs, Singer et. al.


Deep symbolic superoptimization refers to the task of applying deep learning methods to simplify symbolic expressions. Existing approaches either perform supervised training on human-constructed datasets that defines equivalent expression pairs or apply reinforcement learning with human-defined equivalent transformation actions. In short, almost all existing methods rely on human knowledge to define equivalence, which suffers from large labeling cost and learning bias, because it is almost impossible to define and comprehensive equivalent set. We thus propose HISS, a reinforcement learning framework for symbolic superoptimization that keeps humans outside the loop. HISS introduces a tree-LSTM encoder-decoder network with attention to ensure tractable learning. Our experiments show that HISS can discover more simplification rules than existing human-dependent methods, and can learn meaningful embeddings for symbolic expressions, which are indicative of equivalence.

This paper provides a novel approach to the problem of simplifying symbolic expressions without relying on human input and information. To achieve this, they apply a REINFORCE framework with a reward function involving the number of symbols in the final output together with a probabilistic testing scheme to determine equivalence. The model itself consists of a tree-LSTM-based encoder-decoder module with attention, together with a sub tree selector. Their main contribution is that this framework works entirely independent of human-labelled data. In their experiments, they show that this deep learning approach outperforms their provided human-independent baselines, while sharing similar performance with human-dependent ones.
Overall, the work in this paper has the potential to be a contribution to ICLR but lacks completeness and clarity. A number of experimental details are missing, making it difficult to understand the setup under which results were obtained. Moreover, the paper does not seem to have been revised with many grammatical issues that make it hard to read.
The following are major issues within the paper and should be addressed:
•	The paper does not mention the amount of compute given to their model, nor the amount of time taken to train. As the REINFORCE framework is generally quite computation-heavy, these are significant details. Without assessing the amount of compute and time allotted for training HISS, the comparisons to previous baselines lose a fair amount of meaning. The paper alludes to processes being ‘extremely time consuming’, but then does not provide any numbers.
•	They do not mention the data used to train the model weights. In the comparisons sections, some details on datasets are given, but these seem to refer to data for inference.
•	There are many grammatical errors that likely could have been detected with 	 revision. A handful of such errors would not affect the score, but they are so numerous as to make the paper much more difficult to understand.
Additionally, these are comments that slightly detract from the quality of the paper:
•	It’s unclear what to glean from Section 5.1, as the dataset and baselines seem to be fairly trivial. If their claim is to have the first nontrivial human-independent approach to simplifying symbolic expressions, there is no need to compare to baselines that can only handle small expressions.
•	Sections 5.3 and 5.4 contribute little to the paper. For 5.3, the model was trained to embed equivalent expressions close together, using L2 regularization. It is therefore unsurprising that equivalent expressions are then closer together than non-equivalent ones. The paper also does not provide a comparison to the method without this regularization, and so it’s unclear if this embedding similarity helps in any way.  For 5.4, the section is extremely short and contains very little content. Moreover, just as many of the variables in their provided examples oppose their conjectures as support them.
•	The most interesting figure provided is the rewrite rules discovered by the model. It would be even better if an additional column containing the rules discovered by Halide (the main baseline) were provided.
Overall, in my understanding, the primary point in favor of the paper is in being the first nontrivial human-independent approach to simplifying symbolic expression. That said, this is not my area of expertise, so I cannot judge novelty or importance as well as other reviewers.


We propose the differentially private lottery ticket mechanism (DPLTM). An end-to-end differentially private training paradigm based on the lottery ticket hypothesis. Using "high-quality winners", selected via our custom score function, DPLTM significantly outperforms state-of-the-art. We show that DPLTM converges faster, allowing for early stopping with reduced privacy budget consumption. We further show that the tickets from DPLTM are transferable across datasets, domains, and architectures. Our extensive evaluation on several public datasets provides evidence to our claims.

This paper proposes a differentially private version of the lottery ticket mechanism using the exponential mechanism, thus improving the utility of DPSGD by reducing the number of parameters. It provides the privacy guarantee of the proposed algorithm and shows experimentally that the proposed algorithm outperforms DPSGD across datasets and privacy parameters.
The proposed algorithm seems quite interesting. Though it is more of a simple combination of the non-private lottery ticket mechanism and the exponential mechanism, improving utility for DPSGD is a very important topic in differentially private machine learning. The experimental results seem pretty strong.
My only concern is on the aspect of privacy, specially Lemma 1. I think if you’re only using the fact that A and C are in [0, 1], then A*(1-nu*C) can change by 1 if you go from (A=0, C=0) to (A=1, C=0). In the last step of equation (5), you replaced A-A’ by 1, which I think needs to be double-checked since there is the absolute value outside (and I guess the equality should  be <=).
If the calculation is correct, I’m still a bit concerned that the sensitivity is a bit too high compared to the signal. A and C are in [0, 1], and it seems like the sensitivity may not be much smaller than 1, which means the exponential mechanism can be pretty random. To that end, I think you may consider experimentally comparing with DPSGD with a randomly selected ticket, or a ticket with a moderate number of parameters kept.
----- Post-rebuttal response -----
I still don't see why the sensitivity is |1-\nu|. (Doesn't that mean sensitivity is 0 when \nu=1?) If we have (A=1, C=0) and (A'=0, C'=0), we have S(A,C)-S(A',C')=1-0=1; if we have (A=1, C=0) and (A'=1, C'=1), we have S(A,C)-S(A',C')=1-(1-\nu)=\nu. Yet for \nu<1, |1-\nu|<1 and for \nu>1, |\nu-1|<\nu. I believe the sensitivity should be something like max(\nu, 1). You may want to further check that. And I still think this is because the last step in (5) is not correct. Since you have absolute sign, you can't do |A-A'-X| \leq ||A-A'|-X|.
I'm also confused by the range of \nu I should think of.
If \nu <= 1, then in the example you give (A = 0.8, C=0.9,0.7,0.4,0.1, eps=0.1), I think the probability distribution used by exponential mechanism is not much higher for 0.9 than 0.1 (I think it's roughly 0.22 vs. 0.29?) with sensitivity |1-\nu|.
If \nu > 1, then I don't quite understand the score function S(A,C) = A*(1-\nu*C), since for C > 1/\nu, (1-\nu*C) is negative and S would thus penalize higher A. The higher \nu is, the wider the range of C that S will penalize. I can sort of understand this as setting a threshold such that any C above that threshold is just considered useless, but I still think it's a bit unreasonable to penalize higher accuracy, even for very complex model. You may want to explain more on that end.
Thanks for the experiments! They're very useful in supporting your proposed algorithm.


Many processes can be concisely represented as a sequence of events leading from a starting state to an end state. Given raw ingredients, and a finished cake, an experienced chef can surmise the recipe. Building upon this intuition, we propose a new class of visual generative models: goal-conditioned predictors (GCP). Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video. GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames. Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned, but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons. Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training and inference very efficient, and allowing the model to train on sequences that are thousands of frames in length. Finally, we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions. Videos are on the supplementary website: https: //sites.google.com/view/video-gcp

This paper reformulates video prediction problem by conditioning the prediction on the start and end (goal) frame. This essentially changes the problem from extrapolation to interpolation which results in higher quality predictions.
The motivation behind the paper is not clear. First of all, the previous work in video predicted is typically formulated as "conditioned frame prediction" where the prediction of the next frame is conditioned on "a set of context frames" and there is no reason why this set cannot contain the goal frame. Their implementation, however, is motivated by their application and therefore these models are usually only conditioned on the start frames. Unfortunately, besides the reverse planning in imitation learning, the authors did not provide a suite of applications where such a model can be useful. Hence, I think the authors should answer these two questions to clear up the motivation:
1. Why conditioning on the goal frame is interesting? It specifically helps to provide more concrete details than getting from Oakland to San Fransico.
2. Where the current conditional models suffer by conditioning on the goal image?
More experiments are required to support the claims of the paper as well.
Given my point regarding context frames, a more fair experiment would be to compare the proposed method with them when they are conditioned on the goal frame as well. This explicitly has been avoided in 5.1.
 The used metrics are not a good evaluation metric for frame prediction as they both do not give us an objective evaluation in the sense of the semantic quality of predicted frames. The authors should present additional quantitative evaluation to show that the predicted frames contain useful semantic information. FVD and Inception score come to my mind as good candidates.
On quality of writing, the paper is well written but it can use a figure that demonstrates proposed architecture. The authors provided the code which is always a plus.
In conclusion, I believe the impact of the paper, in the current form, is marginal at best and for sure does not meet the requirements for a prestigious conference such as ICLR. However, a more clear motivation, a  concrete set of goals and claims, as well as more comprehensive experiments,  can push the quality above the bar.


This paper advocates the use of offline (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler deep RL algorithms. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on all of the replay data of an online DQN agent, can outperform online DQN on Atari 2600 games. We also present Random Ensemble Mixture (REM), a simple Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Using the DQN replay dataset, offline REM surpasses the gains from online C51 and outperforms offline QR-DQN. Furthermore, REM performs comparably to QR-DQN in the online RL setting on Atari 2600 games. The DQN replay dataset can serve as an offline RL benchmark and will be released.

*Summary*
Paper tackles an important issue, which is benchmarking off-policy deep RL algorithms. The motivation is important, because 1) assessing the performance of off-policy RL algorithm is hard due to the mixed effect of exploration and exploitation while collecting data; hence the paper suggest to fix the data collection process by a standard algorithm (DQN) and then compare the off-policy methods. 2) A fixed dataset will help the reproducibility of the results.
Additionally the paper assess the performance of some off-policy RL methods (mainly DQN and QR-DQN) and showed that given offline data, these algorithms can obtain a superhuman performance.
Besides the mentioned contribution, authors strived for a simple algorithm (easy to implement), and suggests a new simple algorithm, Random Ensemble Mixture (REM) by training multiple Q-functions and randomly sampling a subset for ensemble estimate. The mina motivation come from supervised learning literature.
*Decision*
I enjoyed reading the paper and I think authors tackled a very important problem. Although a very important step, I vote for a weak reject for this paper as I believe the contribution of the current paper is limited. I made my decision mainly based on the following points:
[Limited Task Diversity]: I agree with the authors that having a fix offline data for testing different algorithms is important; however the suggestion (DQN logged data on Atari2600) is a very limited dataset, in terms of 1) task diversity 2) Data collection strategy.
For a benchmark, I would like the dataset to have different strategies for data collection, so that the effect of data collection can be stripped off. In the current assessment, we compare the performance of algorithms condition on the DQN e-greedy data collection procedure. (For example, the right claim to make is: QR-DQN showed a better performance condition on the current data generation process). And it is not clear to me the same observation will hold given a different data generation process (for example a random policy, or a human playing data, or …)
Also, I think only focusing on Atari tasks is limited in nature, as many of these games share a similar underlying mechanism.
[Overfitting] I believe having a fixed dataset for testing off-policy RL is great, however the current suggestion is very prone to overfitting. I can foresee that if the community start to use this benchmark as a test case, soon we will overfit to this task, in a sense that I try algorithm X on offline data, test on Atari see the results, tune algorithm X test on Atari again, … This way I finally have an algorithm that only learned from offline data, but is performing great on online Atari. But I am overfitting!
[Simplicity Trade-Off] Authors focused on getting a simple algorithm, but to me it is unclear why should we optimize for simple algorithm, and my impression of simple as authors have in mind is a “easy to implement” algorithm.
Simplicity is great, but usually I believe we seek a simple algorithm to be able to get more insight into the theory and build more intuition. However, in this work there is a lot emphasis on simple algorithm, but no extra significant insight into off-policy learning has been gained. Additionally, I found the suggested algorithm ad-hoc and of limited contribution.
Minor Comments/ Question that did not significantly contribute to my decision:
[Valid Q-function]: authors mentioned a convex combination of Q-values are a valid Q-value, but never defined a “valid” Q-value. Do authors mean, if Q1 is generated by \pi_1 and Q2 by \pi_2 then convex combination of Q1 and Q2, Q^* can be also generated by some policy \pi^*?
If so, this needs clarification.
And if so, I’m not sure why this is important, since in Q-learning the Q-values that we obtain before convergence might be “invalid” (by the definition above, so that no real policy can generate those values exactly), so what is the point of paying attention to a valid Q values in the first place?
*Improvement*
I think some clear improvement avenues for the future are
1. More diverse set of tasks: I would like to see some non-Atari tasks included in the dataset.
2. More diverse data collection strategy: Example can be different exploration strategies than e-greedy. Or even random exploration, or maybe data of human players. Since we don’t want to have an algorithm that is good only for state distribution induced by e-greedy exploration.
3. More diverse task will help with overfitting by itself. But a good benchmark needs a test case that cannot be exploited.
4. More insight into why REM is outperforming QR-DQN or other algorithms. Currently I am not convinced that we understand why REM is outperforming, or if it’s only because we designed it to be good for this specific task and dataset?
Generally, I believe authors made a very important step, and I really enjoyed reading the paper; however, I think the current contribution is not sufficient to merit a publication. This makes a good workshop paper at this point.


We introduce kNN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a k-nearest neighbors (kNN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong WIKITEXT-103 LM, with neighbors drawn from the original training set, our kNN-LM achieves a new stateof-the-art perplexity of 15.79 -a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.

[Overview]
In this paper, the authors proposed a simple but effective way to augmentation the language model through memorization. Specifically, after obtaining a language model on a dataset, the model further uses the dataset to build a lookup table and then a k-nearest neighbor is used to searching the closest tokens for a token during inference. Based on this, the output distribution of a target token during the inference time would be modified accordingly. Through a comprehensive experiments and ablation studies, the authors showed that the proposed strategy can improve the performance of language models significantly for both the in-domain and out-domain testing scenarios. This is very insightful considering recently a lot of language models are focusing on increasing the size of model and training data.
[Pros]:
Overall I think the paper is well-written and presents clearly. Detailed points below:
1. the authors proposed a simple but effective method for increasing the generalization ability of language model through a memorization strategy. Specifically, the authors proposed to build a lookup table which memorizes the representation and output token pairs which are then used for the inference of language model. Different from conventional way, the proposed strategy does not introduce any more parameters in the model and also does not need any more training or fine-tuning on the target dataset.
2. The authors showed that the proposed strategy can improve the performance of language generation model (i.e., transformer) without any extra training or data, as shown in Table 1. Also, using the continuous caches  with KNN-LM further improve the performance.
3. Besides the main results shown in Table 1 and Table 2, the authors also showed using kNN-LM can probably outperforms the model which is directly trained on it. Also, it also supports domain adaptation from one language domain to another domain.
4. Finally, the authors presented a number of ablation studies to investigate how the performance is affected by the method of building datastore, including the size of nearest neighbor, the interpolation parameter, etc. These results are also insightful and meaningful for the readers to understand the method.
[Cons]:
I think this paper is a solid paper. So I would have some suggestions below:
1. The first concern about the method is the efficiency. At page 3, the authors mentioned that the proposed strategy will bring more time cost. It would be good if the authors can perform more systematical analysis on the time cost of building the datastore and inference for the proposed model.
2. Second, the authors should not only evaluate the proposed method based on transformers. It would be good to test on various language models to verify the generalization ability across different models, including the old-fashioned one like RNN and CNN.
3. Also, the authors should try to extend the proposed model to other language tasks, such as translation.
[Summary]
In this paper, the authors introduced a simple but effective method to augment the pertained language model through memorizations. Though this is not absolutely new and relatively simple , the authors successfully demonstrate that it can be applied to improve the generation of language model much. The. thorough ablation studies help to understand the property of the proposed strategy. I think this paper overall is insightful and thoughtful. It would be good to see the authors add more analysis on the computational complexity and also evaluate on more type of language models.


We study the problem of learning permutation invariant representations that can capture "flexible" notions of containment. We formalize this problem via a measure theoretic definition of multisets, and obtain a theoretically-motivated learning model. We propose training this model on a novel task: predicting the size of the symmetric difference (or intersection) between pairs of multisets. We demonstrate that our model not only performs very well on predicting containment relations (and more effectively predict the sizes of symmetric differences than DeepSetsbased approaches with unconstrained object representations), but that it also learns meaningful representations.

This paper presents a framework for learning representations of multisets.  The approach is built on top of the DeepSets (Zaheer et al., 2017) model, which already is capable of learning representations of multisets, with a new objective of predicting the size of symmetric differences between two multisets, plus a small change in the implementation of the DeepSets model by restricting the per-element module to output a point in the probability simplex.
I liked the background on fuzzy sets and the development of fuzzy multisets and different operations on these multisets.  The definitions and arguments are quite clear and helpful.  One small suggestion for page 4 is that I can understand why the formulation is the only sensible choice for multisets with desired properties, but a claim like this deserves a proper proof.
Model-wise the paper made two contributions for learning representations for multisets as mentioned above: (1) proposed the symmetric difference prediction as a task for learning representations, the argument for this task is that predicting symmetric difference implicitly encourages the model to learn about containment; (2) a slight change in the DeepSets model architecture where the outer rho function is identity and the inner phi function has to output a point in the simplex.
I found these technical contributions to be a bit small.  In addition to this, the paper only presents results on MNIST in a toyish setting, this makes me feel the paper may be more suited for publication in a workshop (idea is interesting, small scale experiments to illustrate the insights, but not complete enough to be published at a conference).
Regarding contribution (1), I can see why predicting symmetric difference makes sense as argued in the paper, but I’m not convinced that this is better than other alternatives.  In order to show that this is a reasonable approach for learning representations, some results that compare this with other possible learning objectives would be necessary.  But I don’t see any such results in this paper.
Regarding contribution (2), I feel the restriction of the phi function to output points in simplex is not very well motivated and confusing in the first read.  Again I can understand why we may want to do this but don’t see why we need to do this.  I’m also concerned that such an architecture may only be good for the task of predicting symmetric difference as it is customized for this task.  Figure 3 shows that an unrestricted model seems to learn better representations despite a worse symmetric difference prediction error, which again confirms the concern.
Another thing about the experiment setup: the second baseline, labeled “DeepSets” in Table 1 actually changed two things compared to the proposed approach: (1) changing the psi function and (2) also changed the symmetric difference function.  It would be good to isolate the contribution of the two.
Overall I feel this paper is not yet ready to be published at ICLR.


The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes. Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc. A key architectural feature combating such environmental irregularities is long-range horizontal connections that aid the perception of the global form of objects. In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity. We introduce the Texturized Challenge -a new benchmark to evaluate object recognition performance under perceptual noise -which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing. Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images. We also present the emergence of several biologically plausible horizontal connectivity patterns, namely centeron surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500). Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.

The paper proposes, inspired by the Primates brain, to add horizontal inhibitory and excitatory connections. In practice, the work proposes a variant of convolutional LSTM cells, that incorporates additional convolutions.
Overall, the paper is hard to read, the experimental setting does not fully convince.  It seems hard to reproduce the results using the information given in the paper. A lot of the method section focuses on intuition, using vague vocabulary which makes it hard to understand concretely what is done in practice. In particular, the contributions are not clear enough. The difference between the proposed approach and existing works needs to be made clearer.
The idea is interesting, however, and the paper would benefit greatly from addressing those issues.
Main points
The claims seem somewhat bigger than the actual contributions. In practice, the contribution is a modification to an LSTM cell.
The paper is not easy to read, and mixes various terms without introducing them (e.g presynaptic activity is used to introduce the method but never introduced, not even in related work). It would be good to use standard notation and math font (e.g. small bold letter for vectors, etc) and to define notations. In Figure 1, the kronecker symbold represents convolutions but in (1), the hadamard product (*) represents convolutions. These inconsistensies make the paper harder to follow.
The problem of clarity extends to the experimental section, where the experiments are not clearly explained,
The method section could be more detailed, in particular, a mathematical comparison of LSTM vs the proposed approachh would be useful. Figure 1 is unclear: what do the red, dark blue and light blue lines correspond to mathematically?
The related attempts in ML and Deep learning should be reviewed ( e.g. Lateral Inhibition-Inspired Convolutional Neural Networkfor Visual Attention and Saliency Detection, AAAI 2018).
The experimental setting is not convincing. It seems that a simple state-of-the-art CNN architecture would do better than the proposed approaches. Comparing a ConvLSTM with a single convolution does not seem fair.
Very little implementation details are given. The authors mention a ConvLSTM2D layer is used. However, the inputs are static images: how exactly was the experiment done? Where does the time come from since the dataset considered is static?
About emergent neurally plausible horizontal connections: that section is interesting but would benefit from being more detailed and rigorous. There is no actual measure or study of the emergence of said connections.
3.2 is misleading as it corresponds to a setting that is never used in the experimental setting.
Most of the results are qualitative, except for Table 1. It would be useful to have quantitative comparisons on established benchmarks.


Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets. Yet, the absence of a unified yardstick to evaluate general visual representations hinders progress. Many sub-fields promise representations, but each has different evaluation protocols that are either too constrained (linear classification), limited in scope (ImageNet, CIFAR, Pascal-VOC), or only loosely related to representation quality (generation). We present the Visual Task Adaptation Benchmark (VTAB): a diverse, realistic, and challenging benchmark to evaluate representations. VTAB embodies one principle: good representations adapt to unseen tasks with few examples. We run a large VTAB study of popular algorithms, answering questions like: How effective are ImageNet representation on non-standard datasets? Are generative models competitive? Is self-supervision useful if one already has labels?

###  Summary
1. The paper presents a new benchmark (VTAB) to evaluate different representation learning methods. VTAB mostly consists of a combination of existing classification datasets. (The classification tasks based on Clevr, SmallNorb, and DMLab are introduced by this paper.)
2. It evaluates and compares the existing representation learning method on the proposed benchmark with an extensive hyper-parameter search.
### Decision with reasons
I vote for accepting this paper (weak accept).
1. The experimental evaluation in this paper is outstanding. Different methods are compared fairly with extensive hyper-parameter tuning (Heavyweight tuning).
2. Even though I find the experiment work to be illuminating, I'm not fully convinced that we need more standardized benchmarks. I'm also not convinced by some of the choices made in designing the benchmark (Such as the models can be pre-trained on any data as long as there is no overlap with the evaluation data).
I find the experimental evaluation to be useful enough for considering accepting this paper, but I will not strongly advocate for acceptance (Unless the authors can better explain why we need yet another benchmark.)
### Supporting arguments for the reasons for the decision.
1. The paper compares existing representation learning methods in a very convincing way. Even though it misses some baselines (Meta-learning based representation learning), it compares supervised, self-supervised, semi-supervised, and generative methods comprehensively. Moreover, the hyper-parameter selection process for the benchmark is fully defined in a dataset agnostic way (It does not depend on the evaluation dataset). This is more important than it might seem -- fully specifying algorithms is important and currently rarely done in the research literature.
I'm not aware of any existing work that compares various representation learning methods on a diverse benchmark fairly and comprehensively, and I'm sure I'll be using the results in this paper to support my claims in the future. As a result, I think this work is worth accepting just for its thorough evaluation of existing methods.
2. However, the paper does not add much to discourse apart from the thorough evaluation. Benchmarks are already prevalent in representation learning, and VTAB doesn't focus on a single well-defined problem with-in representation learning.
For example, dSprites is a benchmark to evaluate how well algorithms can disentangle the underlying data generation factors. An algorithm that significantly improves results on dSprites would, as a result, be better at learning disentangled representation. VTAB, on the other hand, is not trying to highlight any specific problem in representation. As a result, it's not clear what it means if an algorithm does better on VTAB (Apart from the vague notion that it is learning 'better' representations.)
This problem is especially exacerbated by the fact that VTAB does not restrict representation learning to a fixed dataset. This means that a trivial way of improving performance is to just get more representative data (Which is not a bad way of improving performance, but not the way that requires scientific study).
### Questions for Authors
1. The paper makes a binary distinction between fine-tuning and linear evaluation when in-fact this is more nuanced. I suspect that the best performance would be achieved by feezing layers up to a point (And not just not freezing any layers or freezing everything and training a linear classifier). Did the authors explore other choices, and if not, why do they think the current binary distinction is enough. (From my perspective, it makes sense to freeze the initial layers which might be extracting more local/general features and adapt the later layers. The boundary point should be a hyper-parameter)
2. Currently, VTAB puts no restrictions on the representation learning dataset as long as the representation learning dataset does not over-lap with the evaluation dataset. This is problematic since an easier way of improving results on VTAB is to just collect more data/ data that is more representative of evaluation tasks. One could argue that the point of VTAB is to not use details of evaluation datasets in making any decisions for representation learning, however, it is hard to enforce this. For example, if I augment Imagenet with a medical dataset different from the ones in VTAB, is that fine or does that count as cheating?
I feel that a much better way of defining the benchmark would be to restrict representation learning to Imagenet.
### Other minor comments
I don't have many minor comments since the paper is very clear and well written. The author could consider augmenting their graphs with patterns in addition to color to make paper friendly for grey-scale printing and use a more colorblind-friendly color combination.
### Post discussion update ###
One of my primary concerns was the benchmark didn't restrict representation learning to a fixed dataset. That made it trivial to improve performance: just train on a more representative dataset. However, as authors clarified, any dataset collected conditioned on the test set is against the spirit of VTAB. Moreover, discovering what kind of data improves representations is also a research problem, and by restricting to a fixed pre-training dataset, VTAB would not be suitable for such research directions.
I agree that solving representation learning might not just be about finding a better algorithm, but a better algorithm coupled with the right data. In that sense, VTAB's decision to allow any data for pretraining makes sense. However, I'm also concerned that the benchmark might push people to (directly or indirectly) start training on data more representative of VTAB evaluation tasks, and the performance gains in the benchmark will not translate to better algorithms. However, I don't see how this can be avoided for an open-ended benchmark. Since the authors have agreed to include guidelines on how to use the benchmark without abusing it, letting the community police itself given the guidelines is not a bad idea.
I can see VTAB as a benchmark for answering interesting questions (mentioned by authors in the response) such as:
1- How to incorporate large amounts of unlabelled data for representation learning. For example, can we learn better representations by self-supervised learning on a large corpus of data (Such as unlabelled youtube videos)
2- Can we generate data using simulators that can help us learn better representations?
Finally, the authors agreed to add meta-data in their leaderboards (which they plan to release after the review period). The meta-data would allow us to compare methods at a more granular level, such as comparing different unsupervised learning methods. Overall, I think the benchmark and the leaderboards would act as a good source for comparing different unsupervised, semi-supervised, and supervised learning methods (and their combination) fairly.
I'm also updating my score from a weak-acceptance to an acceptance.


Generative networks are promising models for specifying visual transformations. Unfortunately, certification of generative models is challenging as one needs to capture sufficient non-convexity so to produce precise bounds on the output. Existing verification methods either fail to scale to generative networks or do not capture enough non-convexity. In this work, we present a new verifier, called APPROXLINE, that can certify non-trivial properties of generative networks. APPROXLINE performs both deterministic and probabilistic abstract interpretation and captures infinite sets of outputs of generative networks. We show that APPROXLINE can verify interesting interpolations in the network's latent space.

summary:
The paper proposes a method to efficiently verify that generative models are consistent with respect to some known (latent) attribute. The authors defines attribute consistency by 1) mapping pairs of input (x1, x2) with matching attribute to a latent space using an encoder n_E(x) and 2) measuring how correctly an auxiliary classifier will classify the known attribute using (decoded) linear interpolations between the two latent encodings. Importantly, the proposed method gives guaranteed bounds on this consistency score, as opposed to simply evaluating the classifier on a fixed set uniformly sampled points between x1 and x2. In experiments the authors use their method to test for attribute independence as well as consistency under left-right flipping of an image using two different autoencoder models (VAE and CycleAE) obtaining tighter bounds on the ‘attribute consistency’ score than competing methods.
Decision & supporting arguments:
Conceptually I found the paper very appealing, and it tackles an important problem in generative modelling. However I have some concerns with respect to the paper in its current state:
1) It is not clear to me why the attribute consistency score, a key component in the paper, is a good measure of consistency in generative models. Notably, I miss motivation for why linear interpolations between encoded inputs should necessarily keep the attribute stable.
2) Although I found the experiments interesting, I did not find the experimental section completely comprehensive. There is no discussion or experiments probing the dependency on the quality of the auxiliary classifier or the encoder/decoder model used.
3) I did not find the description of the proposed method to be reasonably self-contained. Especially section 3 which describes the proposed method is challenging to follow. The background material in section 2 reads very much like a set of definitions. Since ICLR has a quite broad audience, I think the paper should be written in a more pedagogical way, with for instance clarifying examples. An example of a sentence that is incredibly hard to parse is on page 4, describing domain lifting: “Any deterministic abstract domain can be directly interpreted as a probabilistic abstract domain, where the concretization of an element is given as the set of probability measures whose support is a subset of the set produced by the deterministic concretization.” I think making this paper more pedagogical requires major rewriting.
Due to the above reasons I currently score the paper as a ‘weak reject’.
Further detailed questions/comments:
Consistency Score
Q1: What is the motivation behind the definition of the consistency attribute score. Especially, why is an attribute considered consistent if it is stable to linear interpolations in the latent space?
Experiment Results:
Q2.1: Did you perform any experiments on how the L1 score of the auxiliary classifier affects the consistency score? I would also like to see some quantitative numbers on the auxiliary classifier.
Q2.2: Why is the L1 score used for training the classifier instead of bernoulli which seems more natural for binary attributes?
Q2.3: Similarly, I would like to see some numbers on the quality of the encoder/decodes. Simply inspecting the interpolations in figure 3) the reconstructions seem quite blurry, likely due to the relatively small models used. Is it prohibitively expensive to run the proposed method on bigger models (e.g. ResNet based encoder/decoders or Unet-style models)?
Q2.4: I believe it would be more informative to show the actual confidence intervals in figure 2b) instead of only the width of the confidence intervals?
Readability:
Q3: I found it quite challenging to understand how the proposed is implemented in practice - My suggestion is that the authors add a pseudo-code / algorithm to section 3 clarifying exactly how the bounds reported in the experimental section are calculated.


We extend the idea of word pieces in natural language models to machine learning tasks on opaque ids. This is achieved by applying hash functions to map each id to multiple hash tokens in a much smaller space, similarly to a Bloom filter. We show that by applying a multi-layer Transformer to these Bloom filter digests, we are able to obtain models with high accuracy. They outperform models of a similar size without hashing and, to a large degree, models of a much larger size trained using sampled softmax with the same computational budget. Our key observation is that it is important to use a multi-layer Transformer for Bloom filter digests to remove ambiguity in the hashed input. We believe this provides an alternative method to solving problems with large vocabulary size.

The paper proposes to use codes based on multiple hashing functions to reduce the input and output dimensions of transformer networks. The novelty of the idea is mostly to integrate these codes with transformers. The authors present experiments on two tasks showing that keeping overall model size fixed (i..e, number of parameters), transformers with shorter (but denser) binary codes achieve better performances than standard transformers using one-hot encodings. The gain mostly comes from using larger embedding sizes and larger intermediate layers.
While the technical contribution is limited, because most of the principles are already known or straightforward, the main contribution of the paper is to show that random hash functions are sufficient to create significantly shorter codes and maintain good performances. Such codes provide more freedom in terms of where to put model capacity (larger embeddings, more transformer layers, etc.) which may be useful in applications where most of the model parameters are in embedding matrices.
The value of such a paper resides mostly in the experimental study. On the bright side, the experiments present in sufficient details the impact of the various hyper parameters and the new trade-offs model size/performance that can be achieved. On the other hand, the experiments are carried out on non-standard tasks without previously published baselines, and it is unclear why. Since the method is applicable to any problem involving natural language data (and more generally categorical values, such as knowledge base completion), I would have expected experiments on tasks with a well-defined state-of-the-art. This makes the experiments in the paper look more like "proofs of concept", and they are less convincing than they should be.
detailed comments:
- The paper really is *not* about bloom filters (Bloom filters are data structures that represent sets and efficiently answer membership queries). It is about using codes of identifiers of lower dimension than one-hot encoding. This idea has been used in multi class classification setting (i.e., for the output layer) since (at least) Dietterich & Bakiri (1995) "Solving Multiclass Learning Problems via Error-Correcting Output Codes" (with more insights on what makes a good code for prediction problems). The authors borrow from Bloom filter the way to create the codes using random hash functions, but the analogy stops here.
- following the comment above, and assuming I understood correctly: There is an originality on the paper compared to other works that use binary codes/bloom filters: In the current paper, the authors actually predict the result of individual hashing functions. This is different from predicting the binary encoding that results from using the "or" of one-hot encodings generated by several hash functions, as would be done in approaches (truely) based on Bloom filters. For instance, if there are m hash functions taking values in {1, ..., P}, an approach based on Bloom filters would predict a binary output of dimension P, while here there are m multiclass problems with P classes (IIUC). This difference from previous work may be significant in practice.
- I found the description of the link prediction task (section 3.1) rather cryptic:
* "from each page, we take a random contiguous segment of entities". If I understand clearly, the text is filtered out and only links are kept (?). Links are replaced by the entity id they point to. What happens to the entity the page is about? Is it added at the start?
* 'For evaluation, we hold out one random entity from a random segment on a test page. ": what does "holding out" mean? From my understanding, it means replaced by the [MASK] token, but it could also mean removed altogether from the input sequence.
* the task is to predict masked links in sequences of links with the surrounding text filtered out. Does that correspond to any real-life prediction problem (i don't see which one)? Is this "task" intended to serve as unsupervised pre-training of embeddings? If yes, maybe the authors might say so and give example applications.
- For the natural language task:
* ", then apply a Bloom filter to reduce the vocabulary size." -> From my understanding, there is no bloom filter here. If I understand, what is done is to represent unigrams and bigrams by their bloom digest to reduce the input dimension.
- "We hold out 10% random entity pages for testing. " -> is there a validation set? How do you choose the hyper parameters?
minor comments:
- "Since the Wikipedia site usually removes duplicates of links on each page, the distribution of pages is rather long tail." -> I wouldn't be surprised if the distribution was long tailed even without this specific policy
- I found the formalization/notation more confusing than helping because it is not really thorough (there is no distinction between sets and sequences, "1[\eta_j(t_i)] ... where 1[.] is the indicator function" -> what is the "indicator function" of a number?)


In this article, we study a proposal that enables to train extremely thin (4 or 8 neurons per layer) and relatively deep (more than 100 layers) feedforward networks without resorting to any architectural modification such as Residual or Dense connections, data normalization or model scaling. We accomplish that by alleviating two problems. One of them are neurons whose output is zero for all the dataset, which renders them useless. This problem is also known as dead neurons. The other is a less studied problem, dead points. Dead points refers to data points that are mapped to zero during the forward pass of the network. As such, the gradient generated by those points is not propagated back past the layer where they die, thus having no effect in the training process. In this work, we characterize both problems and propose a constraint formulation that added to the standard loss function solves them both. As an additional benefit, the proposed method allows to initialize the network weights with constant values or even zero and still allowing the network to converge to reasonable results. We show very promising results on a toy, MNIST, and CIFAR-10 datasets.

This papers suggests new penalties to avoid the apparition of dead units and minimize the existence of dead points.
Decision
I vote to reject this paper because the formulation of the penalties lacks clarity and because the experiments are incomplete and likely provide misleading results.
Justification
    Formulation of the penalty
It is unclear to me how the slack variables xi depends on the weights, and therefore how they can be minimized by adjusting the weights using a penalty. I believe this should be clarified between equations (12) and (13).
The optimizer used is Adam instead of a plain SGD. How is the penalty integrated in Adam? Is it left out as it is best to do for weight decay or is it integrated in the adaptive estimations? I believe a plain SGD would be better justified when the goal is to understand an improvement to the flow of information in the network.
    Experiments
The experimental setup have many problems.
1) The protocol is creating a bias
2) The comparisons of zero initializations are indirect
3) The tasks chosen cannot corroborate that the penalty is helping information flow deeper without harming generalisation.
1) Protocol
Results may be misleading both because learning rates are not adjusted, which can cause deeper models to diverge while they could still be trained and because number of epochs is limited instead of letting the models train until convergence. I note that plots comparing train and validation accuracy in Figure 1 shows better results on the validation set than on the training set for ReLU and ReLU + BN. This suggests indeed that the optimization diverged, breaking the training accuracy while leaving the validation accuracy close to random. It is difficult however to evaluate the color-maps without an axis for the colors, so my assumption that dark purple is close to random accuracy may be wrong. A related note, description of Figure 4 in section 4.2 discuss about the achieved maximum performance at depth 60 and width 25. I could not read the same, to me it seems to be achieved right at depth 1, width 3.
2) Comparisons of zero initializations
The experiments on the zero initialization scheme is an interesting investigation, but it is incomplete as there is no direct comparison with training without Sep-UP. The Annealed Dropout is important to allow training with zero initialization, and may well be more important than Sep-UP. We cannot measure their respective importance without a direct comparison as in Figure 1.
3) Too simple tasks
The Figure 3 points to an important issue of the paper. Generally, the depth of networks is shown to provide higher capacity which leads to improved accuracy when not overfitting. In the experimental setup of this paper, the tasks and architectures do not improve with depth, or more precisely the improvement in capacity (training accuracy is maximal at depth 10) is associated with overfitting. If empirical results show that the contributed penalty helps improving the flow of information, it should also be shown that it is not at the cost of better generalisation otherwise there is no point in having greater depth. The chosen experimental setup does not allow this however. Even though it does fair better at generalization than the others, they all suffer from worst generalisation with depth. More difficult tasks may be better suited for these experiments. I am not recommending to use very large datasets such as ImageNet, difficult synthetic problems could be sufficient. It is necessary however that depth can be shown to be beneficial for baselines, so that we can confirm that training with the penalty is not degrading the beneficial effect of depth.
Other comments
The problem of dead points should be shown empirically since the contributions of this paper are supported empirically.
Fourth paragraph of section 4.1 mentions results using convolutional layers, but it is not clear which of the provided results are on fully connected or convolutional networks.
Minor comments
In equation (1), \hat{e}_j is not specified. My guess is that represents a basis, but I don't see why this basis necessary in the equation.
In equation (5), the notation is limited to a single layer, with x being the input of the layer. I believe this should be generalized with a composition operator as in equation (7).
There seems to be a mistake in equation (8). The inclusion in X on the right is useless since the set on the left is an intersection with X.
Section 3.1, third paragraph: It is *easily* to see [...]
Section 3.2, first paragraph: [...] slack variables for *each each* point on the batch.
Section 4.1, first paragraph: [...] (see Figures 1b and *1b*). (should be e)
Edit:
I still believe this paper lacks empirical evidence that the proposed loss would help train deep models without harming performance on tasks where depth matters. In light of all the corrections and clarifications, I upgrade my vote to weak reject.


Wisdom of the crowd (Surowiecki, 2005) revealed a striking fact that the majority answer from a crowd is often more accurate than any individual expert. We observed the same story in machine learning -ensemble methods (Dietterich, 2000) leverage this idea to combine multiple learning algorithms to obtain better classification performance. Among many popular examples is the celebrated Random Forest (Ho, 1995), which applies the majority voting rule in aggregating different decision trees to make the final prediction. Nonetheless, these aggregation rules would fail when the majority is more likely to be wrong. In this paper, we extend the idea proposed in Bayesian Truth Serum (Prelec, 2004) that "a surprisingly more popular answer is more likely the true answer" to classification problems. The challenge for us is to define or detect when an answer should be considered as being "surprising". We present two machine learning aided methods which aim to reveal the truth when it is minority instead of majority who has the true answer. Our experiments over real-world datasets show that better classification performance can be obtained compared to always trusting the majority voting. Our proposed methods also outperform popular ensemble algorithms. Our approach can be generically applied as a subroutine in ensemble methods to replace majority voting rule.

Inspired by work in ensembling human decisions, the authors propose an ensembling technique called "Machine Truth Serum" (based off "Bayesian Truth Serum"). Instead of using majority vote to ensemble the decisions of several classifiers, this paper follows the "surprisingly popular" algorithm; the ensembled decision is the decision whose posterior probability (based on several classifiers) most exceeds a prior probability (given by classifier(s) trained to predict the posterior predictions). It's quite a nice idea to bring this finding from human decision-making to machine learning. If it worked in machine learning, it would be quite surprising, as the surprisingly popular algorithm risks that the ensemble makes a decision against the majority vote, which is usually consider the safe/default option for ensembling.
Overall, I did not find the experiments (in the current state) to provide compelling enough support for the claim that MTS is a useful approach to ensembling in machine learning.
* Unless I am mistaken, the authors use a more powerful model (an MLP) as the regressor compared to some of the models they ensemble over. In practice, people ensemble the most powerful models they have available, so it's unclear if using a regressor with the same capacity as the ensembled classifiers will provide any additional benefit. On a related note, it would be nice to know what is the classification performance of each individual classifier? As well as how often the regressor correctly choose to go with several weaker models rather than the strongest model. In particular, I am concerned that the performance of the ensemble might be less than or equal to the performance of a single MLP classifier (or whatever other model does best).
* "In this paper, each of the datasets we used has a small size - we chose to focus on the small data regime where the classifiers are likely to make mistakes." Why not try large data tasks that are challenging for state-of-the-art models? The paper makes a general claim that MTS is a good way to aggregate predictions, so only evaluating on small datasets seems to be a limitation
* As I understand (correct me if I am wrong), the reported results are only on examples with "high disagreement" between classifiers. However, for practical use cases, it is useful to know how the overall accuracy compares. One major risk of using the "surprisingly popular" algorithm is that the algorithm may cause the ensemble to make many incorrect predictions when the majority is right (but the minority prediction is selected). If you have those numbers, I would be interested to see them added to the paper.
I am also unsure about if applying the "surprisingly popular" algorithm in machine learning makes sense. The algorithm is motivated by the fact that difference agents have different information. However, in the ML setting, various classifiers usually have the same information. It's possible to restrict the information given to each classifier, but that would limit the performance of each individual classifier (and hurt the ensemble). I would be curious if the authors have any thoughts on this point.
I also have a few questions/concerns about how the approach is implemented:
* Why not train a single model to predict the average prediction of all models and use that model's prediction as the prior? This approach seems simpler but equivalent to the approach currently taken.
* Why not use model distillation (predicting all output logits/probabilities, or an average thereof) rather than just predicting an average of 0/1 predictions?
* For HMTS, why do the regressors for each of L labels need to be separate? It seems more efficient to use a multi-class model (as many model distillation approaches do)
* If DMTS can learn to predict when most classifiers are wrong, why wouldn't the original classifiers themselves learn to predict the answer correctly? It seems to me that the reason the experiments show that DMTS/HMTS work is that some/many of the underlying classifiers are weaker than the model that is used to ensemble the predictions (an assumption that doesn't hold in practice).
Overall, I really like the high-level idea, and a better ensembling approach promises to bring empirical gains across many ML tasks. However, I have several concerns about the experiments, motivation, and algorithmic decisions which make me hesitant to recommend the paper for acceptance.


Training a classifier over a large number of classes, known as 'extreme classification', has become a topic of major interest with applications in technology, science, and e-commerce. Traditional softmax regression induces a gradient cost proportional to the number of classes C, which often is prohibitively expensive. A popular scalable softmax approximation relies on uniform negative sampling, which suffers from slow convergence due a poor signal-to-noise ratio. In this paper, we propose a simple training method for drastically enhancing the gradient signal by drawing negative samples from an adversarial model that mimicks the data distribution. Our contributions are three-fold: (i) an adversarial sampling mechanism that produces negative samples at a cost only logarithmic in C, thus still resulting in cheap gradient updates; (ii) a mathematical proof that this adversarial sampling minimizes the gradient variance while any bias due to non-uniform sampling can be removed; (iii) experimental results on large scale data sets that show a reduction of the training time by an order of magnitude relative to several competitive baselines.

The paper presents a method for negative sampling for softmax when dealing with classification of data to one from a large number of classes. Its main idea is to negative sample those classes which lead to higher signal to noise ratio than for uniform negative sampling. This is based on building an auxilary model using decision tree from which the adversarial negative classes are sampled, so that the distribution of the negative samples can be close to the positive ones leading to higher SNR while training. The proposed method is compared to other methods for negative samping on two publicly available large-scale datasets from the extreme classification with XML-XNN features.
Positives :
1. The proposed approach with adversarial negative sampling using an auxilary model seems interesting
2. It scales well to datasets with large number of classes.
Negatives :
The experimental evaluation of the proposed approach lacks completeness and does not look convincing for the following reasons :
1. It misses out a recent state-of-the-art method (Slice) for negative sampling on same datasets [1], which also addresses the same problem of sampling most promising negative classes but in a different way. Furthermore, [1] also compares against many other sota methods missed out in this paper on the many other datasets datasets including those in this paper but in a more general multi-label setting.
2. The paper only compares against other negative sampling approaches such as AandR, NCE, and does not show what happens when no negative sampling is done such as done in (DiSMEC) [2]. This is important to understand what (if at all) is lost by doing approximation as proposed. For instance, a quick experiment reveals that DiSMEC can give about 19% accuracy on Wiki500 dataset, which is better than that achieved by the proposed method. Though it is computationally expensive but due to its simplicity, it must be discussed nevertheless to give a complete picture.
Instead the OVE baseline used in the paper seems quite sub-optimal in the first place, and hence stronger baselines [1,2] for which the code and results are readily available and have been duly tested in the community must be used and discussed.
Another aspect that the paper misses out is the role of fat-tailed distribution [3,4] of the instances among labels, which is a property of typical datasets in this regime. It is possible that one can get good accuracy but poor performance on tail-labels due to approaximations. The performance on tail-labels on appropriate metrics other than accuracy, such as MacroF1, should be evaluated.
Also, the proposed approach must be tested on more datasets including the smaller ones such as EURLex (also used in works referenced in the paper) on which it is easier to compare with other methods (such as DiSMEC, Slice and AttentionXML [5]) without encountering computational constraints and also bigger ones such as Amazon3M, also avilable from the repository.
Finally, it must be investigated if the proposed method can be extended to the multi-label setting or are there inherent limitations of the model in this setting. The possibility to extend it to the general multi-label setting would make this approach more promising and directly comparable to wide range of algorithms.
[1] H. Jain,  V. Balasubramanian,  B. Chunduri and M. Varma, Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches, in WSDM 2019.
[2] R. Babbar, and B. Schölkopf, DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classification in WSDM, 2017.
[3] H. Jain, Y. Prabhu, and M. Varma, Extreme Multi-label Loss Functions for Recommendation, Tagging, Ranking & Other Missing Label Applications in KDD, 2016.
[4] R. Babbar, and B. Schölkopf, Data Scarcity, Robustness and Extreme Multi-label Classification in Machine Learning Journal and European Conference on Machine Learning, 2019.
[5] AttentionXML: Extreme Multi-Label Text Classification with Multi-Label Attention Based Recurrent Neural Networks, NIPS 2019