Imitation Learning (IL) is a machine learning approach to learn a policy from a set of demonstrations. IL can be useful to kick-start learning before applying reinforcement learning (RL) but it can also be useful on its own, e.g. to learn to imitate human players in video games. However, a major limitation of current IL approaches is that they learn only a single "average" policy based on a dataset that possibly contains demonstrations of numerous different types of behaviors. In this paper, we present a new approach called Behavioral Repertoire Imitation Learning (BRIL) that instead learns a repertoire of behaviors from a set of demonstrations by augmenting the state-action pairs with behavioral descriptions. The outcome of this approach is a single neural network policy conditioned on a behavior description that can be precisely modulated. We apply this approach to train a policy on 7,777 human demonstrations for the build-order planning task in StarCraft II. Dimensionality reduction techniques are applied to construct a lowdimensional behavioral space from the high-dimensional army unit composition of each demonstration. The results demonstrate that the learned policy can be effectively manipulated to express distinct behaviors. Additionally, by applying the UCB1 algorithm, the policy can adapt its behavior -in-between games -to reach a performance beyond that of the traditional IL baseline approach.

Summary:
The paper proposes behavioral repertoire imitation learning (BRIL) which aims to learn a collection of policy from diverse demonstrations. BRIL learns such a collection by learning a context-dependent policy, where the context variable represents behavior of each demonstration. To obtain a context variable, BRIL rely on user’s knowledge, where the user manually defines a feature space that describes behavior. This feature space is then reduced by using a dimensionality reduction method such as t-SNE. Lastly, the policy is learned by supervised learning (behavior cloning) with a state-context input variable and an action output variable. The method is experimentally evaluated on the StarCraft environment. The results show that BRIL performs better than two baselines: behavior cloning on diverse demonstrations and behavior cloning on clustered demonstrations.
Score:
The weaknesses of the paper are novelty, clarity, and evaluation. Please see the detailed comments below. I vote for rejection.
Comments:
- Novelty of the proposed idea.
The major issue of the paper is the lack of novelty. The idea of learning a context-dependent policy in BRIL closely resembles that of existing multi-modal IL methods (Wang et al., 2017, Li et al., 2017). The main difference is that, BRIL relies on a manually defined context variable (behavioral feature space). In contrast, the existing methods aim to learn the context variable from demonstrations. BRIL is too simple when compared to the existing methods. Moreover, using a manually specified feature space does not go well with the main principle of deep learning, which is to learn informative feature spaces from data end-to-end. I think that ICLR is not a suitable venue for this paper.
- Clarity of the proposed method.
The second issue of the paper is clarity. Specifically, two important steps of BRIL is policy learning by supervised learning (behavior cloning) and dimensionality reduction by t-SNE. However, explanations of these two steps are vague and incomplete. For example, in Section 2.1, the paper describes IL as supervised learning, but does not mention the issue of covariate shift, which is well-known when treating IL as supervised learning (Ross et al., 2011). Also, it is incorrect to state that an IL agent cannot interact with the environment during training, since many IL methods such as GAIL require interactions with the environment during training. Meanwhile, in Section 2.4, it is unclear how probability distributions in t-SNE reflect similarity between data points.
[1] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
- Evaluation of the proposed method is too narrow.
The paper lacks important baseline methods in the experiment. Specifically, the paper does not compare BRIL against multi-modal IL methods (Wang et al., 2017, Li et al., 2017) which also learn a context-dependent policy. Moreover, BRIL is evaluated only on the StarCraft environment with only one kind of manually specified feature. This raises a question of generality and sensitivity against the choice of feature of BRIL. To improve the paper, I suggest the authors to compared the proposed method against multi-modal IL methods on different environment, and evaluate BRIL with different choices of the behavioral features.


Reinforcement Learning (RL) has demonstrated promising results across several sequential decision-making tasks. However, reinforcement learning struggles to learn efficiently, thus limiting its pervasive application to several challenging problems. A typical RL agent learns solely from its own trial-and-error experiences, requiring many experiences to learn a successful policy. To alleviate this problem, we propose collaborative inter-agent knowledge distillation (CIKD). CIKD is a learning framework that uses an ensemble of RL agents to execute different policies in the environment while sharing knowledge amongst agents in the ensemble. Our experiments demonstrate that CIKD improves upon state-of-the-art RL methods in sample efficiency and performance on several challenging MuJoCo benchmark tasks. Additionally, we present an in-depth investigation on how CIKD leads to performance improvements.

This paper introduces a method for using an ensemble of deep reinforcement learning policies, where members of the ensemble are periodically updated to imitate the most promising member of the ensemble. Thus learning proceeds by performing off policy reinforcement learning updates for each individual policy, as well as some supervised learning for inter-policy imitation learning.
I start by what I view as the positive aspects about the paper:
1- The algorithm is quite simple (to understand and to implement).
2- Experimental results are performed on a variety of domains, and more importantly, each experiment is motivated by a question.
That said, I have some concerns about this paper which I list below:
1- Perhaps my biggest concern is that the approach is not motivated from a theory stand point. There has been interesting results in Osband's work [Osband, 2016] (and references therein) for randomized value functions which can serve as a foundation for this work. That said, a) Osband's results, at least immediately, are related to value-function based methods, as opposed to policy gradient b) the KL update which one could argue is the main and only significant contribution of the paper, is not justified by Osband or any other prior work c) there is not anything that this paper adds to the literature to better justify diversity through randomization and/or imitation learning based on the best member of the ensemble.
2- I have found various claims in the paper which are unclear, scientifically not true, or sometimes even contradicting. In Introduction, for example, the authors mention that the agent sometimes gets into a sub-optimal policy and may require a large number of interactions before escaping the sub optimal policy. How does gathering more data help to improve the policy? Either we are in a local maximum, which if we are doing gradient ascent, there is really not much we could do, or that we are in a saddle point, which we can escape by adding some noise to the gradient. [Jin,2017]
3- In section 4.3 the authors talk about on-policy methods requiring importance sampling (IS) ratios. To the best of my knowledge, IS is only used for off-policy learning. Can the authors provide a link to an on-policy method that does IS?
4- Again in section 4.3 authors claim and I quote "Using off-policy methods, all the policies in the ensemble can easily be updated, since off-policy update methods can perform updates from any \tau". But later on in Section 5.3 authors claim that "off-policy actor-critic methods (e.g. SAC) cannot fully utilize the other agent's or past experience." So which statement is true?
5- Again, the KL update is interesting, but is it even surprising that the KL update is necessary for an ensemble of policies updates using policy gradients? In the absence of this KL update, which the authors characterize as the method that Osband proposed, the policies could generally be arbitrarily far from one another. This means that each policy needs to perform policy evaluation using trajectories that are coming from other policies who in principle can be radically different than the policy we want to update. This means that updates will be quite "off-policy" which we know can really degrade the quality of the estimated gradient. This is perhaps why even choosing a random policy to update towards is providing "some" improvement. I think this is the real insight, but it is not really discussed at all in the paper.
6- On the same note, I do not think that one can say Osband's method is the same as CIKD but only without the KL update. Most notably, Osband's work was presented for value-function-based methods like DQN. These methods work fundamentally different than policy gradient methods, which rely on (near) on-policy updates to perform good policy improvements. In that sense, the presented results make sense, but I disagree with the framing of the results and how they are presented here.
7- In section 5.3, when the authors utilize more policy updates to have a fair comparison, are they retuning hyper parameters? Surely they need to do that, at least for hyper-parameters that are known to be super important such as the step size.
8- Overall I liked section 5.5 that is trying to dissect causes for improvement. However, it seems like that the "dominant agent" hypothesis has been rejected hastily, unless I misunderstood the experiment. The authors show that the notion of best is spread across different agents. But of course this will be the case in light of the KL update, since the policies are getting closer to one another. Can you redo the experiment in the absence of the KL update?
9- Have the authors thought about any connection between this and genetic algorithms? In genetic algorithms, the idea is the next set of candidates are chosen based on the most promising candidates in the current iteration. CIKD seems like a soft implementation of this idea.
In light of the comments above, I am voting for weak rejection, though as I said before, I do see some interesting things in this paper. I encourage the authors to think about CIKD from a theoretical lens in the future.


It has been repeatedly observed that convolutional architectures when applied to image understanding tasks learn oriented bandpass filters. A standard explanation of this result is that these filters reflect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass filters are matched to such structure. The present paper offers an alternative explanation based not on the structure of images, but rather on the structure of convolutional architectures. In particular, complex exponentials are the eigenfunctions of convolution. These eigenfunctions are defined globally; however, convolutional architectures operate locally. To enforce locality, one can apply a windowing function to the eigenfunctions, which leads to oriented bandpass filters as the natural operators to be learned with convolutional architectures. From a representational point of view, these filters allow for a local systematic way to characterize and operate on an image or other signal.

This paper claims that convolutional filters in CNNs are not the result of fitting to the input data distribution but they are the optimal solution to a spectral decomposition of the convolutional operator.
Positive things about this work:
1) it is clearly written
2) the fact that Gabor wavelets are the eigenfunctions of convolution is sound.
3) it provides good food for thought about what needs to be learned and what comes from the pre- specified choice of architecture
Negative things about this work:
1) it misses references to relevant works. In particular, there is an article making essentially the same points:
Joan Bruna, Soumith Chintala, Yann LeCun, Serkan Piantino, Arthur Szlam, and Mark Tygert, "A mathematical motivation for complex-valued convolutional networks," Neural Computation, 28 (5): 815-825, 2016
http://tygert.com/ccnet.pdf
where these authors make the same conclusions and observe that learning reduces to figuring out the  windowing, number of scales, etc. but not the type of filters.
This prior work greatly reduces the impact of this contribution, unfortunately.
There are other references that are missing, but these are minor points compared to the above. For instance, I'd recommend to cite Hubel & Wiesel's seminal work on mapping the mammalian receptive fields, and older work by M. Lewicki about analyzing learned receptive fields by sparse coding algorithms, similarly to the cited B. Olshausen et al.
2) The Authors show that Gabor wavelets are eigenfunction of convolutions and they stop here to conclude that filters in CNNs are the way they are because of the architecture, but how about the effect of the non-linearities, depth and the type of cost used for training? The work is unfinished without a thorough analysis and discussion of these crucial aspects.


Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned statevalue function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.

The paper proposes an online variant of MPO, V-MPO. Compared to MPO, the main difference seems to be in the E-step. Instead of optimizing the non-parametric distribution towards a parameterized Q-function, V-MPO learns the V-function and updates the non-parametric distribution towards the advantages, which can be estimated on the samples of the last roll-outs based on the empirical returns and the learned V-function. Of course, updating towards (exponentiated and normalized) Q- or A-function does not make a difference. There are also some minor changes (which might still be crucial) such as an entropy constraint in the M-step and using only the top-k advantages during the E-Step (an option that was discussed in Abdolmaleki, et al. 2018a).
V-MPO is evaluated on DLM, ALE and two humanoid tasks from the DeepMind Control Suite. In most of these tasks V-MPO achieves returns that are to the best of my knowledge higher than any previously reported ones. However, the experiments also use a very large number of system interactions (in the order of billions).
Contribution / Significance:
I think that there will be relatively high interest in the paper due to the reported performances.
The technical contribution seems a bit incremental compared to MPO. Also, by learning a value function V-MPO gets closer to REPS. The submission lists the use of top-k samples and the M-step KL bound as the main differences to REPS. However, the former is not evaluated in the submission and the latter, albeit crucial, seems to be a relatively small modification. I do think that there are more differences to REPS, most important probably in the way of learning the value function and the corresponding differences in the derivations. However, I think that the differences abd similarities to MPO and REPS need to be discussed more thoroughly.
Soundness:
The derivation of V-MPO is relatively sound. The optimization of the KL constraints seems very approximate, although it seems to work well in practice.
Clarity:
I do not like the way the algorithm is presented. The submission specifies the complete loss function already at the beginning of the "Method"-Section and derives/motivates the individual terms in hindsight. The spaghetti-code like structure unnecessarily forces the reader to jump between pages or keeps the reader in the dark. I also do not like the "stop-gradient" notation which in my opinion puts the focus on low-level implementation details at the cost of not properly explaining what the optimization actually does. I think that paper is well-written in general, but the structure needs to be improved.
Experiments:
The evaluation clearly focuses on achieving the best performance, and does a good job in that regard.
However, a good evaluation should also help in understanding the mechanics of V-MPO. How does k (in top-k) affect the performance? How well are the constraints met during optimization? How does V-MPO compare to related on-policy methods (e.g. TRPO/PPO) on slightly more computational constrainted settings (eg rllab/mujoco with < 1e7 steps)?
Questions:
- Did you experiment with controlled entropy reduction akin to MORE, instead of using fixed
epsilon eta?
- Can you give a rough estimate of the computational time required to perform these experiments on a standard desktop pc? It really is difficult for me to even roughly estimate it.
Assessment:
Currently, I am leaning to accept because I think that V-MPO is overall a nice work. However I do think that submission needs to be revised. I mainly think that the structure needs to be improved and that V-MPO needs to better related to closely related work. I listed some additional experiments that would significantly improve the submission in my opinion.


Reasoning about uncertain orientations is one of the core problems in many perception tasks such as object pose estimation or motion estimation. In these scenarios, poor illumination conditions, sensor limitations, or appearance invariance may result in highly uncertain estimates. In this work, we propose a novel learning based representation for orientation uncertainty. Characterizing uncertainty over unit quaternions with the Bingham distribution allows us to formulate a loss that naturally captures the antipodal symmetry of the representation. We discuss the interpretability of the learned distribution parameters and demonstrate the feasibility of our approach on several challenging real-world pose estimation tasks involving uncertain orientations.

The paper proposes a Brigham loss (based on the Brigham distribution) to model the uncertainty of orientations (an important factor for pose estimation and other tasks). This distribution has the necessary characteristics required to represent orientation uncertainty using quaternions (one way to represent object orientation in 3D) such as antipodal symmetry. The authors propose various additions such as using precomputed lookup tables to represent a simplified version of the normalization constant (to make it computationally tractable), and the use of Expected Absolute Angular Deviation (EAAD) to make the uncertainty of the Bingham distribution more interpretable.
+Uncertainty quantification of neural networks is an important problem that I believe should gain more attention so I am happy to see papers such as this one.
+Various experiments on multiple datasets show the efficacy of the method as well as out performing or showing comparable results to state-of-the-art
-In the caption for Table 1 the author’s write: “the high likelihood and lower difference between EAAD and MAAD indicate that the Bingham loss better captures the underlying noise.” How much difference between EAAD and MAAD is considered significant and why?
-In section 4.5 they write “While Von Mises performs better on the MAAD, we observe that there is a larger difference between the MAAD and EAAD values for the Von Mises distribution than the Bingham distribution. This indicates that the uncertainty estimates of the Von Mises distribution may be overconfident.” Same question as above. What amount of difference between MAAD and EAAD is considered significant and why?


Normalizing Flows (NFs) are able to model complicated distributions p Y (y) with strong inter-dimensional correlations and high multimodality by transforming a simple base density p Z (z) through an invertible neural network under the change of variables formula. Such behavior is desirable in multivariate structured prediction tasks, where handcrafted per-pixel loss-based methods inadequately capture strong correlations between output dimensions. We present a study of conditional normalizing flows (CNFs), a class of NFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p Y |X (y|x). CNFs are efficient in sampling and inference, they can be trained with a likelihood-based objective, and CNFs, being generative flows, do not suffer from mode collapse or training instabilities. We provide an effective method to train continuous CNFs for binary problems and in particular, we apply these CNFs to super-resolution and vessel segmentation tasks demonstrating competitive performance on standard benchmark datasets in terms of likelihood and conventional metrics.

Summary of the paper:
The paper proposes an extension of Normalizing flows to conditional distributions. The paper is well written overall and easy to follow. Basically the conditional prior z|x = z=f_{\phi}(y,x), where x is the conditioning random variable, and we apply the change of variable formula to get the density of y|x . For example in super resolution y is the high res image and  x is the low res. image.  To sample from the models authors propose to use f^{-1}_{\phi}(z;x).
The conditional modules are natural extensions of invertible blocks used in the literature (coupling layers, split priors, conditional coupling, 1x1 conv), where the conditioning is done on some hidden representations of the conditioning variable x (i.e one or multiple layers of NN).
Authors propose a dequantization for binary random variables (useful for segmentation applications), where they give an implicit model for the dequantizer (obtain a continuous variable from a discrete binary variable).
Author apply the method in two applications super-resolution and vessel segmentation. the method is compared to supervised learning of the corresponds between x and y and to others competitive methods in the literature and shows some advantage.
Minor comments :
- Formatting the bibliography is messed up and needs some cleaning , Figure 5 is also making formatting issues of the paper.
- Figure 1 for sampling it should be f^-1_{\phi } and not f_{\phi}
Review:
- Figure 2 is hard to get any idea of the sample quality would be good also to put the low resolution input to the algorithm . Also did you use a temperature sampling for the baseline ? otherwise the comparison is not fair.
- The Drive database is too small 20 training samples and 20 testing only? can the model be just overfitting?
- In the vessel implementation why do you drop the scaling modules?
- The conditioning for the vessel implementation on x is on two layers , would be great to put all architectures of the models in details , and to show both sampling and training paths
- It would be great to add the details of the skip connection used from the network processing x, and how ensure that the flow remains invertible.
Overall this is a well written paper and a good addition to normalizing flows methods , some discussion of related works on conditional normalizing flows and more baselines with other competitive methods based on GANs for example would be helpful but not necessary.
It would be great to add details of the architectures and on skip connections and how to ensure invertibility for this part in the model .


Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions. We present a novel and principled solution for modeling both the global absolute positions of words and their order relationships. Our solution generalizes word embeddings, previously defined as independent vectors, to continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can correlate with each other in a continuous function. The general solution of these functions is extended to complex-valued domain due to richer representations. We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding (we make all code available). Experiments on text classification, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings. To our knowledge, this is the first work in NLP to link imaginary numbers in complex-valued representations to concrete meanings (i.e., word order).

### Summary
The authors present a "natural" way of encoding position information into word embeddings and present extensive empirical evidence to support their method. I believe that paper meets the bar for acceptance.
### Details
The paper "Encoding word order in complex embeddings" presents a method for making word embeddings position dependent. The idea in a nutshell is to map each discrete position , n, to a value  `   A exp( freq_{word, dim} × n )` . So a word embedding is a collection of complex valued signals sampled at discrete points.
The frequency is dependent on each word and each dimension in general. The authors motivate/justify this particular formulation via their Claim 1, which argues that their particular formulation uniquely satisfies two intuitive constraints. Although one of those constraints (i.e. linearly witnessed Position-free offset) almost completely specifies the solution.
The experiments in the paper are fairly thorough and cover text classification, machine translation and language modeling. Through the comparative experiments the complex embeddings we can see that the formulation in this paper outperforms existing SOTA methods, sometimes with significantly difference such as a difference of 1.3 BLEU point on the MT task.
I would have liked to say that the ablation are similarly conclusive but there seem to be a problem in the table, which eroded my confidence:
1. The number of parameters in rows 5 and 8  (w/t encoding positions, share / not-share respectively) are reported to be 9.38M and 8.33M which has to be wrong. Similar problem happens with other pairs. And now I am not sure whether the results were also swapped or not. Still the results in general trend in the right direction.
### Possible improvements to the paper
1. The main weakness of the paper is that the authors repeatedly mention that encoding the position as a multiplicative factor which is multiplied to the frequency gives leads to a more decoupled/interpretable embedding but their experiments are solely focused on accuracy measurement. I would have liked to see the authors carry out more experiments to see whether the frequency parameters really are interpretable? For example,
      -  What is the histogram of the frequencies ? Are some of them negative?
      - Which word has the highest frequencies (pooled over all dimensions) in absolute term? Does it make sense that that word's meaning is so position dependent? For example, positions can capture subjects versus objects in english, but they will more reliably reflect the subject versus verb distinction in hindi.
      - Are the word frequencies by themselves predictive of anything? For example, what happens if the word embedding amplitudes are tied across words or dimensions? We expect the performance to be bad but how bad?
These kinds of ablations  / qualitative analysis will really make the paper more informative and interesting. Right now it just seems like yet another paper where the capacity of the model is increased and the accuracy increases. Specially because the delta improvement over the fixed positional embeddings of (Transformer-TPE Vaswani et al. 2017) is so limited.
### Edit after the author response
The authors have made the required corrections and added the necessary analysis. One interesting outcome was that the "word-sharing amplitude schema" seems to drop so little in performance, it's almost like all the information can be coded in just the phase vectors alone. It will be nice if the authors release their trained phase embeddings as well, for the words.


We propose order learning to determine the order graph of classes, representing ranks or priorities, and classify an object instance into one of the classes. To this end, we design a pairwise comparator to categorize the relationship between two instances into one of three cases: one instance is 'greater than,' 'similar to,' or 'smaller than' the other. Then, by comparing an input instance with reference instances and maximizing the consistency among the comparison results, the class of the input can be estimated reliably. We apply order learning to develop a facial age estimator, which provides the state-of-the-art performance. Moreover, the performance is further improved when the order graph is divided into disjoint chains using gender and ethnic group information or even in an unsupervised manner.

This paper presents an order learning method and applies it to age estimation from facial images. It designs a pairwise comparator that categorizes ordering relationship between two instances into ternary classes of greater than, similar, and smaller than. Instead of directly estimating the class of each instance, it learns pairwise ordering relationship between two instances.
For age estimation from facial images, it uses a Siamese network as a feature extractor of the pairwise comparator, concatenates feature maps of dual inputs (test instance and one of the multiple reference instances that are selected from the training data) and applies it to a ternary classifier. Given the softmax probability scores of the classifier results, it estimates the final class as the one that maximizes a consistency rule computed over indicator functions. The comparator is trained to minimize a comparator loss computed over the softmax probability scores.
The paper also provides an extended version for multiple disjoint chains, where each chain may correspond to a higher-level attribute class, for example, gender or ethnic group. When there is no supervision available, it randomly partitions the training set.  and iteratively updates reliability scores and chain memberships.
Novelty-wise, I consider that the proposed solution to be satisfactorily innovative and on a different vein than the existing methods.
One concern about the method is that it imposes the geometric ratio (log distance) between the class distances in age estimation, considering the difference between 5 and 10-year-old instances is easier to detect than 65 to 70-year-old instances as stated in the paper. However, as far as it is understood from the provided discussion compared SOTA methods are not retrained or fine-tuned in the same manner. This raises the question of whether the slight performance improvement is a result of the geometric ratio, in particular when computing the cumulative score CS.
Another concern is that the performance is not stellar as the presented method underperforms in comparison to DRF (Shen et al. 2018) on MORPH II. The presentation of the results for FG-Net is neither complete nor included in the main paper, and results for CLAP2016 are missing. Discussion of the results Table 8 does not seem fair as it is not clear whether MV is retrained keeping the geometric ratio in mind. It might be the case that even for 15-69, MV might attain lover MAEs.
The paper also misses the latest SOTA, for instance, BridgeNet [1] is mentioned in a sentence, yet its results are omitted from the Tables. BridgeNet, according to its results, is on par with the proposed method. Also, the numbers reported in this paper and the numbers in [1] have a discrepancy, which causes confusion.
[1] Wanhua Li et al. Bridgenet: A continuity aware probabilistic network for age estimation, CVPR 2019.


We propose a novel approach for preserving topological structures of the input space in latent representations of autoencoders. Using persistent homology, a technique from topological data analysis, we calculate topological signatures of both the input and latent space to derive a topological loss term. Under weak theoretical assumptions, we can construct this loss in a differentiable manner, such that the encoding learns to retain multi-scale connectivity information. We show that our approach is theoretically well-founded and that it exhibits favourable latent representations on a synthetic manifold as well as on real-world image data sets, while preserving low reconstruction errors.

Summary
-------------
The paper proposes an approach, based on persistent homology (PH), to preserve certain topological structures of the input in the latent representations learned by an autoencoder. This is realized via an additional (i.e., in addition to reconstruction) loss term (optimized over mini-batches) which requires differentiating through the PH computation. While this has been done before (e.g., Chen et al., Hofer et al.), the authors have certainly put an interesting spin on this. The theoretical part of the work deals with the issue of using mini-batches for PH computation and whether this computation is close to the computation on the full point cloud. Experiments and comparisons on multiple datasets are presented to demonstrate that the approach, e.g., preserves nesting relationships in the input. The paper is nicely written and the content is very well presented. There are questions here and there (see below), but I do think they can be answered.
Major comments/remarks:
-------------------------------------
My first question relates to the issue that the loss only incorporates 0-dim. information. The authors do remark that higher-dim. features can be included, but the results were similar. However, after thinking about this issue quite some time, I am curious if it is possible to obtain "zero" of the topological loss (so this term is perfectly optimized), but the encoder introduces, e.g., cavities in the data which were not present in the input (e.g., 1-dim. holes).
Also, can you show formally (maybe this is trivial and I am not seeing it) that L_t = 0 would lead to 0 distance between the corresponding diagrams w.r.t. some common metric? A more formal treatment of the implications of the loss in Eq. (2) would certainly help.
Another question that immediately comes to mind is whether the computation of VR PH in the input space (e.g., CIFAR 10) makes sense, as the authors rely on ||.||_2 if I understood this correctly. I would argue that the topology of the input is basically unknown, especially for images and computing Euclidean distances among images, or vectorized images, does not make sense. For the nice results on the SPHERES data set it does, as the spheres are defined exactly using ||.||_2. If the VR PH in 0-dim. of the input is enforced upon the representations in the AE bottleneck, but the input topology is not captured well, then you might be enforcing something that you possibly do not want.
Apart from that, it is known that the Euclidean distance degenerates quickly in high dimensional spaces, e.g.,
Aggrawal et al.
On the Surprising Behavior of Distance Metrics in High Dimensional Space
Maybe this is also contributing to the fuzzy visualization of CIFAR-10 in Fig. 3 (apart from the low-dim. of the bottleneck)?
Also, maybe the authors could work out (in greater detail) the differences between their results from Thm.1/2 and the results of Chazal et al., in "Subsampling Methods for Persistent Homology". In my point of view, the results in the paper only hold if you would consider just a single batch, right? I mean, if the loss is computed from the batch, and a gradient update is performend, Z^{m} will changes (as the encoder changes as a result of the update), while the input does not.
Finally, how were the KL divergence measures in Table 1 computed, as you need a density estimate of the input as well, not just for the representation space, right? Is this not a very crucial issue in the input space? If so, how reliable are the numbers presented for KL_{0.01},etc., given that the differences are sometimes extremely small.
Minor comments
-----------------------
Sec. 6: We presented a topological autoencoders -> We presented a topological autoencoder
Overall, I think this is a nicely done paper, but with quite some question marks at many places. I do think this is always the case for something new, though, and
actually a good thing.


Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture. In this paper, we present a novel approach, namely Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We solve it by introducing edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability. Experiment results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search. Our code has been made available at https://www.dropbox.com/ sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0.

---
revised score. rebuttal clears my concerns.
---
Summary:
The paper proposes a partially connected differential architecture search (PC-DARTS) technique, that uses a variant of channel dropout for each node's output feature maps, and a weighted summation of concatenating all previous nodes. Searched architecture on CIFAR-10 and ImageNet seems to outperform the one discovered by the original DARTS, however, the results are not directly comparable due to the slight change of search space.
Introducing this edge normalization is a novel contribution, but it is more like a trick to have a better search space rather than the PC-DARTS itself. My main concerns are about the incremental novelty and experiments are heavily done on one search run, especially the search space is not the same as baseline DARTS.
I do not think the current version is ready for ICLR, but I am looking forward to seeing the authors' rebuttal and I am willing to revise my review accordingly.
Main concerns
- Incremental novelty about channel sampling.
Doing edge normalization in the PC-DARTS is indeed novel, however, the channel sampling (abbr. PC for partial channel connection) is not. Dropout is widely adopted in all deep learning training since AlexNet. In NAS with parameter sharing, Pham et al. already exploit the channel dropout as shown in ENAS function "def drop_path"(https://github.com/melodyguan/enas/blob/master/src/cifar10/image_ops.py). It is true that previous works treated like one hyper-parameters and do not provide deeper insight about this term, but it is not correct to say in Section 3.4 "Channel sampling ... has never been studied in prior work". In my perspective, the key difference of channel sampling is the retained channel number is always fixed to K, the non-selected channels are not zeroed, where the dropout usually has only a probability K / total_channel and non-selected feature is multiplied to a zero constant.
Thus, I suggest authors provide additional experiments as in Table 3 to compare the original drop-path with proposed channel sampling. Considering the test error drops from 3% to 2.67% while using PC, it will be more convincing to show the original drop path with probability K / total_channel yields a smaller drop to evidence the effectiveness of proposed sampling.
- Proposed edge normalization is not a new sampling policy but a new search space.
To my understanding, this edge normalization is effectively a change to the search space rather than the sampling policy, and generalize to many other policies as well, and can be a substantial contribution to the NAS community. However, under current experiments setting, it is hard to isolate the improvement is from this new space or the channel sampling, as detailed later.
- About the motivation.
Throughout the paper, in abstract, introduction, section 3.2 and section 4.4, the authors claim that the larger batch size is particularly important for the stability of architecture search which is not well-studied and lack of references. From Table 4, it is hard to tell the stability is from the larger batch size or the proposed partial channel sampling.
- Questions about experiments
1. Experiments comparing to the baseline is not fair.
As in Section 4.2, the CIFAR-10 search is different from the original DARTS and P-DARTS in the following manner. The batch size is changed from 64(in DARTS)/96(in P-DARTS) to 256,  super-net is freezed for the first 15 epochs, and introducing the edge normalization parameter \beta_{i,j} increase the search space.  With all these changes, it is quite hard to isolate the effectiveness of proposed PC-DARTS. Two possible simple experiments to compare is, using the original DARTS space and training set, 1) do not update the \beta but use a fixed initialization that all \beta is the same (to mimic original DARTS concatenation); 2) add \beta to original DARTS as well and re-run 1).
It is completely reasonable to me the contribution of this paper is introducing a novel edge-normalization that is simple and effective to improve the DARTS based approach. If so, the authors could revise the conclusion easily. However, in the least scenario, the experiment comparison should be in a fair way.
2. In original DARTS, error drop from 3% for the first-order gradient to 2.76% while using the second-order one, will this trend occurs with PC-DARTS?
3. Robustness
Recent work about evaluating neural architecture search reveals that NAS algorithms are sensitive to random initialization[1,2] and the search space [3], this in general leads to a notorious reproducibility problem of current NAS and shows it is not reasonable to only compare final performances on proxy tasks over **one** searched architecture. However, in the stability study in Section 4.4.3, multiple runs are still over the same architecture discovered in earlier experiments. In Section 4.4.2, the paper mentioned the search runs multiple times, yet the reported results in Table 3 are against the single run, as indicated by CIFAR-10 no PC- no EN error 3.00 +- 0.14, which is identical to the results in Table 1 DARTS (1st-order). Could the authors report the results with at least 3 different initializations, and possibly release the seeds? It would significantly strengthen the effectiveness of the proposed approach.
Minor comments
- According to Section 4.4.1 and Figure 3, change the K from 1 to 8, the search cost drops significantly. Does this mean the batch size in the ablation study is changing all the time? How could we know if the test-error is reduced due to the sampling ratio or to the batch size?
Typos
1. Table 2, caption below the table, \dag is not aligned with the one in the used table.
--- reference ---
[1] Li and Talwalker, Random search and reproducibility of neural architecture search, UAI’19
[2] Sciuto et al., Evaluating the search phase of neural architecture search, arxiv’19
[3] Radosavovic et al., On Network Design Spaces for Visual Recognition, ICCV'19.


Domain adaptation tackles the problem of transferring knowledge from a labelrich source domain to an unlabeled or label-scarce target domain. Recently domain-adversarial training (DAT) has shown promising capacity to learn a domaininvariant feature space by reversing the gradient propagation of a domain classifier. However, DAT is still vulnerable in several aspects including (1) training instability due to the overwhelming discriminative ability of the domain classifier in adversarial training, (2) restrictive feature-level alignment, and (3) lack of interpretability or systematic explanation of the learned feature space. In this paper, we propose a novel Max-margin Domain-Adversarial Training (MDAT) by designing an Adversarial Reconstruction Network (ARN). The proposed MDAT stabilizes the gradient reversing in ARN by replacing the domain classifier with a reconstruction network, and in this manner ARN conducts both feature-level and pixel-level domain alignment without involving extra network structures. Furthermore, ARN demonstrates strong robustness to a wide range of hyper-parameters settings, greatly alleviating the task of model selection. Extensive empirical results validate that our approach outperforms other state-of-the-art domain alignment methods. Additionally, the reconstructed target samples are visualized to interpret the domain-invariant feature space which conforms with our intuition.

###Summary###
This paper proposes Max-margin domain adversarial training (MDAT) to tackle the problem of transferring knowledge from a rich-labeled source domain to an unlabeled target domain. This is achieved by designing an adversarial reconstruction network. The proposed MDAT stabilizes the gradient by replacing the domain classifier with a reconstruction network.
The motivation of the proposed network is based on the observations that the traditional domain-adversarial training is vulnerable in the following aspects:1) the training procedure of the domain discriminator is unstable, 2) it only considers the feature-level alignment, 3) it lacks the interpretable explanation for the learned feature space.
In the proposed method, the Adversarial Reconstruction Network (ARN) consists of a shared feature extractor, a label predictor, and a reconstruction network. The reconstruction network only focuses on reconstructing samples on the source domain and pushing the target domain away from a margin. The feature extractor tries to confuse the decoder by learning to reconstruct samples on the target domain.
The paper performs experiments on several domain adaptation tasks on digit datasets. The experimental results demonstrate the effectiveness of the proposed results over several baselines such as DANN, ADDA, CyCADA, CADA, etc.
The paper also provides empirical analyses such as t-SNE embedding, plotting the loss, etc. to illustrate the effectiveness of the proposed approach.
### Novelty ###
The model proposed in this paper is extended from the domain adversarial training approach. To stabilize the gradient, the model replaces the domain classifier with a reconstruction network. In this way, the discriminator only discriminates the reconstructed data from the source domain. This idea is interesting and provides some novelty.
###Clarity###
Overall, the paper is well organized and logically clear. The claims are well-supported by the experiments. The images are well-presented and well-explained by the captions and the text.
###Pros###
1) The paper proposes a Max-margin based approach to tackle domain adaptation. Instead of leveraging the domain discriminator to discriminate the source from the target, this paper utilizes a reconstructor to push the target domain far away from the margin. The idea is interesting and heuristic to the domain adaptation research community.
2) The experimental results on digit benchmark demonstrate the effectiveness of the proposed method over other baselines including the most state-of-the-art ones.
3) The paper provides many analyses to demonstrate the effectiveness of the proposed method.
###Cons###
1) The experimental part of this paper is weak. The paper only provides experimental results on the digit recognition experiments, which is not enough to demonstrate the effectiveness and robustness of the proposed approach. Further experimental results on image recognition or NLP task is desired.
It will be also interesting to see how does the proposed method perform on large-scale datasets such as DomainNet and Office-Home dataset:
DomainNet: Moment Matching for Multi-Source Domain Adaptation, ICCV 2019. http://ai.bu.edu/DomainNet/
Office-Home: Deep Hashing Network for Unsupervised Domain Adaptation, CVPR 2017. http://hemanthdv.org/OfficeHome-Dataset/
2) The organization and presentation of this paper should be polished.
Based on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs.


Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images. The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems. However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers. In this study, a new model called 2-Layers Sparse Predictive Coding (2L-SPC) is introduced to assess the impact of this inter-layer feedback connection. In particular, the 2L-SPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers. 2L-SPC and a 2-layers Hi-La networks are trained on 4 different databases and with different sparsity parameters on each layer. First, we show that the overall prediction error generated by 2L-SPC is lower thanks to the feedback mechanism as it transfers prediction error between layers. Second, we demonstrate that the inference stage of the 2L-SPC is faster to converge than for the Hi-La model. Third, we show that the 2L-SPC also accelerates the learning process. Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the 2L-SPC features are more generic and informative.

This work proposed a new model called Sparse Deep Predictive Coding, which introduced top-down connections between consecutive layers, to improve the solutions to hierarchical sparse coding (HSC) problems. Instead of decomposing the HSC problem into independent subproblems, the proposed model added a new term to the loss function, which represents the influence of the latter-layer on the current layer.
#Pros:
-- The proposed model adopted the idea from predictive coding and came up with a relatively novel idea for HSC problems.
-- The experiments are solid. The experiments evaluated the proposed methods with different hyper-parameter settings and three real-world datasets.
-- The figures in the result sections are well designed and concise.
#Cons:
-- The mathematical description of the main problem and the proposed model is not clear. For example, the dimensionality for the variables in Eq.(1) is not clarified.
-- The test procedure is not clear. How the internal state variables are obtained for the test set is not clarified.
-- The proposed model was only compared with a basic Hierarchical Lasso network. There are not any state-of-art methods included as baseline methods.
#Detailed comments:
(1) The proposed model is named as sparse DEEP predictive coding, however, the experiments only considered the SDPC and Hi-La networks with 2 layers. I am wondering if a deeper structure will improve the performance?
(2) For the structure shown in Fig.1, the decoding dictionaries are $D^T_i$, but I am confused why the encoding dictionaries are reciprocal to encoding dictionaries. Does it come from the optimization updates shown in Eq.(3)?
(3) According to Eq.(1), $x$ is a vector and $D$ is a 2d matrix. However, the real inputs in the experiments are images and $D$ is a convolutional filter with 4 dimensions. How are the matrices reshaped?
(4) For section 2.2 and 2.3, the number/index of samples is not shown in the loss function for training. The loss should be over the whole training set. Besides, the test procedure is not clarified.
(4) The number of iterations using FISTA of SDPC and Hi-La networks is shown to compare the rate of convergence. However, considering both models are solving a lasso-type regression problem, I would suggest using coordinate descent for optimization.
(5) For the main result of prediction error, why is the “global prediction error” more important than the reconstruction error? Is the first-layer prediction error the reconstruction error? If yes, Fig.2 shows that Hi-La has a lower prediction error compared to SDPC for the first layer.
(6) Two minor comments on writing:
(a) It would be better to have a separate section for 2.5 since it describes the dataset and is not related to the proposed model.
(b) A typo of “neuronal implementation” exists in the introduction section.


Recurrent neural networks (RNNs) are particularly well-suited for modeling longterm dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate. While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs. iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD). We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation. We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

Update: the authors have addressed my issues below and I have raised my score to 8.
***
This paper on recurrent neural networks goes back to Rosenblatt's continuous time dynamics model and uses a discretised version of that equation (equation (1) in the paper) to build an incremental version of the RNN, called incremental RNN, where the transition from hidden state h_{k-1} at step k-1 to next step's h_k is done using small incremental updates until the system achieves equilibrium. It claims that it manages to solve the vanishing gradient problem by keeping all gradients \frac{\partial h_k}{\partial h_{k-1}} equal to minus identity matrix. The algorithm is then extensively evaluated on a large number of tasks and compared with plain RNNs, LSTMs, and two recently published papers on antisymmetric RNN and FastRNN.
I need to admit that after reading the paper twice, I am not sure I understand how the method works exactly (how does inserting intermediary steps and variables g_0, g_1, ... g_T enable the system to reach equilibrium: is there an iterative evaluation until convergence?) and more worringly, how the single-step SiRNN differs from a normal RNN with an extra residual connection?
According to equation (5), for T=1 and g_0=0, we have:
g_T = g_1 = \eta_k^1 ( \phi (U (h_{k-1}) + W x_k + b) - \alpha h_{k-1} ).
If the gradients are vanishing in the normal RNN, why would they not vanish here for T=1?
Propositions 1 and 2 are for the case K=\infinity, and I could not understand the proof of theorem 1 that shows why \frac{\partial h_k}{\partial h_{k-1}} = - I. This seems to be the major contribution of the paper and should be given prominence.
What is missing is clear explanation, like (Bengio et al, 1994), of the identity gradient and of how the algorithm works. These questions could be solved by including code or pseudo-code explaining how to actually implement incremental RNNs.
There are also several important papers recently published that have approached the problems of continous-time dynamics and relaxation of hidden state to equilibria.
* The paper does not mention at all Neural ODEs [2] [3] where the state flows in a continuously differentiable way thanks to the continuous-time residual network ODE formulation. Moreover, isn't the idea of inserting a relaxation to equilibrium using ODEs already implemented in the ODE-RNNs [3]?
* How do incremental updates related to Adaptive Computation Time [4]?
For this reason, I am currently tending to reject the paper, but am open to change my score upon clarifications and links to other similar work.
Additional remarks:
The first paragraph of the paper explains the Elman RNN, not RNNs in general.
Please cite [1] alongside Bengio et al (1994) for the problem of the vanishing gradient.
Define alpha in equation (1)
Notation k and K is very confusing
Blue vs. green on Figure 2 is hard to read, and where is the new initialisation?
Why do you add h_k^K to h_{k-1} in equation (8)? I thought it was g_k^K?
Keep the same colours for all experiments in figure 4.
[1] Hochreiter (1991) "Untersuchungen zu dynamischen neuronalen Netzen"
[2] Chen, Rubanova, Bettencourt & Duvenaud (2018) "Neural Ordinary Differential Equations"
[3] Rubanova, Chen & Duvenaud (2019) "Latent odes for irregularly-sampled time series"
[4] Graves (2016) "Adaptive computation time for recurrent neural networks"


Graph neural networks (GNN) such as GCN, GAT, MoNet have achieved stateof-the-art results on semi-supervised learning on graphs. However, when the number of labeled nodes is very small, the performances of GNNs downgrade dramatically. Self-training has proved to be effective for resolving this issue, however, the performance of self-trained GCN is still inferior to that of G2G and DGI for many settings. Moreover, additional model complexity make it more difficult to tune the hyper-parameters and do model selection. We argue that the power of self-training is still not fully explored for the node classification task. In this paper, we propose a unified end-to-end self-training framework called Dynamic Self-traning, which generalizes and simplifies prior work. A simple instantiation of the framework based on GCN is provided and empirical results show that our framework outperforms all previous methods including GNNs, embedding based method and self-trained GCNs by a noticeable margin. Moreover, compared with standard self-training, hyper-parameter tuning for our framework is easier.

The paper proposes an approach for learning graph convolutional networks for inferring labels on the nodes of a partially labeled graph  when only limited amount of labeled nodes are available.
The proposal is inspired from Graph convolution Networks with the idea of overcoming the major drawback of these models that lies of their behavior in case of limited coverage of the labeled nodes, which implies using deeper versions of the model leading at the price of what the authors call the over-smoothing problem.
The main idea here consists in relying on self training to get a better coverage of labeled nodes enabling learning with less deep models, this translates to a simple and intuitive algorithm. Using self training is not new in GCN but the way it is used here, computing adaptively a threshold for incorporating pseudo labels and using weights according to the confidence off predictions is new.
Experimental results are reported on citation datasets and compared with many baselines show similar results as baselines when the coverage increases up to 50 labeled nodes /class, but the method brings significant improvements when the coverage is low (e.g. only few, <20, labels /class).
Although the difference with previous approaches do not look like a huge step, the method seems to be quite justified empirically and achieve real good results wrt state of the art.


We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Update 11/21
With the additional experiments (testing a new image, testing fine-tuning of hand-crafted features), additions to related work, and clarifications, I am happy to raise my score to accept. Overall, I think this paper is a nice sanity check on recent self-supervision methods. In the future, I am quite curious about how these mono-image learned features would fare on more complex downstream tasks (e.g., segmentation, keypoint detection) which necessarily rely less on texture.
Summary
This paper seeks to understand the role of the *number of training examples* in self-supervised learning with images. The usefulness of the learned features is evaluated with linear probes at each layer for either ImageNet or CiFAR image classification. Empirically, they find that a single image along with heavy data augmentation suffices for learning the first 2-3 layers of convolutional weights, while later layers improve with more self-supervised training images. The result holds for three state-of-the-art self-supervised methods, tested with two single-image training examples.
In my view, learning without labels is an important problem, and it is interesting what can be learned from a single image and simple data augmentation strategies.
Comments / Questions
It seems to me that for completeness, Table 4 should include the result of training a supervised network on top of random conv1/2 and Scattering network features, because this experiment is actually testing what we want - performance of the features when fine-tuned for a downstream task. So for example, even if a linear classifier on top of Scattering features does poorly, if downstream fine-tuning results in the same performance as another pre-training method, then Scattering is a perfectly fine approach for initial features. Could the authors please either correct this logic or provide the experiments?
Further, it seems that the results in Table 4 might be a bit obscured by the size of the downstream task dataset. I wonder if the learned features require fewer fully supervised images to obtain the same performance on the downstream task?
Can the authors clarify how the neural style transfer experiment is performed? The method from Gatys et al. requires features from different layers of the feature hierarchy, including deeper layers. Are all these features taken directly from the self-supervised network or is it fine-tuned in some way?
While I appreciate the computational burden of testing more images, it does feel that Image A and B are quite cherry-picked in being very visually diverse. Because of this, it seems like a precise answer to what makes a good single training image remains unknown. I wonder how feasible it is to find a proxy metric that corresponds to the performance on downstream tasks which is expensive to compute. It might be interesting to try to generate synthetic images (or modify real ones) that are good for this purpose and observe their properties.
I disagree with the claim of practicality in the introduction (page 2, top). While training on one image does reduce the burden of number of images, the computational burden remains the same. And as mentioned above, it doesn’t seem likely that *any* image would work for this method. Finally, more images are needed to learn the deeper layers for the downstream task anyway.
The paper is well-written and clear.


Determinantal point processes (DPPs) is an effective tool to deliver diversity in multiple machine learning and computer vision tasks. Under the deep learning framework, DPP is typically optimized via approximation, which is not straightforward and has some conflicts with the diversity requirement. We note, however, there have been no deep learning paradigms to optimize DPP directly since it involves matrix inversion that may result in computational instability. This fact greatly hinders the use of DPP on some specific objective functions where DPP would otherwise serve as a term to measure the feature diversity. In this paper, we devise a simple but effective algorithm to optimize the DPP term directly through expressing with L-ensemble in the spectral domain over the gram matrix, which is more flexible than learning on parametric kernels. By further taking into account additional geometric constraints, our algorithm seeks to generate valid sub-gradients of the DPP term in cases where the DPP gram matrix is not invertible (no gradients exist in this case). In this sense, our algorithm can be easily incorporated with multiple deep learning tasks. Experiments show the effectiveness of our algorithm, indicating promising performance for practical learning problems.

Summary: the authors introduce a method to learn a deep-learning model whose loss function is augmented with a DPP-like regularization term to enforce diversity within the feature embeddings.
Decision: I recommend that this paper be rejected. At a high level, this paper is experimentally focused, but I am not convinced that the experiments are sufficient for acceptance.
****************************
My main concerns are as follows:
- Many mathematical claims should be more carefully stated. For example, the authors extend the discrete DPP formulation to continuous space. It is not clear to me, based on the choice of the kernel function embedding, that the resulting P_k(X) is a probability (Eq. 1). If it is not (using a DPP-based formulation as a regularizer does not require a distribution), the authors should clarify that fact; more generally, the authors should be more careful throughout the paper (for example, det=0 if features are proportional, not necessarily equal; the authors inconsistently switch between DPP kernel L and marginal kernel K throughout computations.)
- The authors do not describe their baselines for several experiments. In tables 1, 2, 3, the baseline is never described (I assume it's the same setup without regularization); I did not find a description of DCH (Tab 4) in the paper (Deep Cauchy Hashing?). The mAP-k metric should also be defined. Furthermore, the authors do not report standard deviations for their experiments.
- A key consideration when using DPPs is their compulational cost: most operations involving them require SVD (which seems to be used in this work), matrix inversion, and often both. This, unsurprisingly, limits the applications of DPPs, and has driven a lot of research focused on improving DPP overhead. I would like to see more discussion in this paper focused on to which extent the DPP's computational overhead remains tractable, and which methods were used (if any) to alleviate the computational burden.
- Finally, the paper itself appears to be somewhat incomplete: sentences are missing or incomplete (Section 4), and numbers are missing in some tables (Table 5).
***********************
Questions and comments for the authors:
- When computing the proper sub-gradient, are you computing the subgradient as inv(L + \hat L)?
- You state that by avoiding matrix inversion, your method is more feasible. However, it seems like your method requires SVD, which is also O(N^3); could you please provide more detail for this?
- Could you report number of trials and standard deviations for your experiments?
- Do you have any insight into why DPPs do more poorly than the DCH baseline in Table 4 for mAP-k metrics?
- You might be able to save space by decreasing the space allocated to the exponentiated quadratic kernel.


Event-based neuromorphic systems promise to reduce the energy consumption of deep neural networks by replacing expensive floating point operations on dense matrices by low energy, sparse operations on spike events. While these systems can be trained increasingly well using approximations of the backpropagation algorithm, this usually requires high precision errors and is therefore incompatible with the typical communication infrastructure of neuromorphic circuits. In this work, we analyze how the gradient can be discretized into spike events when training a spiking neural network. To accelerate our simulation, we show that using a special implementation of the integrate-and-fire neuron allows us to describe the accumulated activations and errors of the spiking neural network in terms of an equivalent artificial neural network, allowing us to largely speed up training compared to an explicit simulation of all spike events. This way we are able to demonstrate that even for deep networks, the gradients can be discretized sufficiently well with spikes if the gradient is properly rescaled. This form of spike-based backpropagation enables us to achieve equivalent or better accuracies on the MNIST and CIFAR10 datasets than comparable state-of-the-art spiking neural networks trained with full precision gradients. The algorithm, which we call SpikeGrad, is based on accumulation and comparison operations and can naturally exploit sparsity in the gradient computation, which makes it an interesting choice for a spiking neuromorphic systems with on-chip learning capacities.

EDIT After Rebuttal: My understanding of the contributions of this paper has improved. I now increase my score to a weak accept.
This paper proposes a new backpropagation algorithm learning algorithm "SpikeGrad" for Spike-based neural network paradigm. Simulating this algorithm on a classical hardware would require a lot of time-steps. To circumvent this, they show how to construct a corresponding artificial neural net (that can be trained using the traditional gradient based algorithms) which is equivalent to the spiking neural net. Using this equivalence they simulate a large scale SNN on many real-world dataset (first paper to do so). In particular, they use MNIST and CIFAR-10 for this purpose. They show that training a fixed architecture using their method is comparable to other prior work which uses high-precision gradients to train them. They also show how to exploit sparsity of the gradient in the back propagation for SNN.
This paper is hard-to-follow for someone not familiar with the background material. In particular, without looking at prior literature it was hard to understand that "integrate and fire neuron model" is essentially the feedforward mechanism for the SNN. I would suggest the authors make this a bit more explicit. Moreover, it would serve the structuring of the paper to have a formal "Preliminaries" section, where all known stuff goes. It was hard to discern what is new in this paper, and what is from prior work and these are mixed in section 2. For instance, section 2 states "SpikeGrad" algorithm; but the main contribution (ie., the back propagation algorithm) only appears in the middle of this section. Likewise, I think section 3 can be arranged better. In particular, the equivalence is a "formal" statement and thus, could be stated as a theorem followed by a proof. It will also make it explicit as to what does it mean by an "equivalent" network. In fact, it is still not clear to me at this point what that statement means. Could you please elaborate this in the rebuttal?
Regarding the conceptual contribution of this paper, if I understood things correctly, the main claim is that they give a new way to train SNN whose performance on MNIST and CIFAR-10 is comparable to other works. The second contribution is that they give the equivalence between ANN and SNN (point above). It is also unclear to me what the point regarding the sparse gradient in the backpropagation in the experimental section is trying to make? Could you please clarify this in the rebuttal as well?
At this point, the writing of this paper leaves me with many unanswered questions that needs to be addressed before I can make a more informed decision. Please provide those in the rebuttal and based on those will update my final score. But with my current understanding of this paper, I think this does not meet the bar. The contributions in this paper do not seem too significant.


Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019). Such a dilemma is shown to be rooted in the inherently higher sample complexity ) and/or model capacity (Nakkiran, 2019), for learning a high-accuracy and robust classifier. In view of that, give a classification task, growing the model capacity appears to help draw a win-win between accuracy and robustness, yet at the expense of model size and latency, therefore posing challenges for resource-constrained applications. Is it possible to co-design model accuracy, robustness and efficiency to achieve their triple wins? This paper studies multi-exit networks associated with input-adaptive efficient inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency. Our proposed solution, dubbed Robust Dynamic Inference Networks (RDI-Nets), allows for each input (either clean or adversarial) to adaptively choose one of the multiple output layers (early branches or the final one) to output its prediction. That multi-loss adaptivity adds new variations and flexibility to adversarial attacks and defenses, on which we present a systematical investigation. We show experimentally that by equipping existing backbones with such robust adaptive inference, the resulting RDI-Nets can achieve better accuracy and robustness, yet with over 30% computational savings, compared to the defended original models.

The paper exploited input-adaptive multiple early-exits, an idea drawn from the efficient CNN inference, to the field of adversarial attack and defense. It is well-motivated by the dilemma between the large model capacity required by accurate and robust classification, and the resulting model complexity as well as inference latency.
Overall, this paper presents an interesting perspective, with strong results. The usage of input-adaptive inference reduces the average inference complexity, without conflicting the "larger capacity" assumption for co-winning robustness and accuracy.
Since no literature has discussed the attacks for a multi-exit network, the authors constructed three attack forms, and then utilized adversarial training to defend correspondingly. The design of Max-Average Attack is particularly smart - to balance between "benefiting all" and "maximally boosting one" (its result is also convincingly good).
The authors presented three groups of experiments, from relatively heavy networks (ResNet38), to very compact ones (MobileNet-V2). It is especially meaningful to see their strategy work on MobileNet too (though the computational saving is a bit less, no surprise). The authors also did due diligence in ablation study and comparing with recent alternatives.
Several points that could be addressed to potentially improve the paper:
- The authors want to make it clearer that: their "triple win" is not about constructing a light-weight model that is both accurate and robust. It's instead about given an accurate + robust, yet heavy-weight model, how to reduce its AVERAGE computational load per sample inference, by routing "easier" examples to earlier exits.
- Can the authors think of and construct more diverse and stronger attacks for RDI-Nets? For example, it would be interesting to attacking RDI-Nets (e.g., defended by Max-Average) with randomized weighted combinations of single attacks?
Note that, at inference time, the same "randomized combination" cannot be also adopted as defense, because an input always wants to exit the earliest possible for efficiency gains.
- The advantage over ATMC is not obvious: slightly lower TA, slightly higher ATA, and slightly more parameters. Could the authors try to align their parameters more closely (to the extent possible)?
- A missing related work: "Shallow-Deep Networks: Understanding and Mitigating Network Overthinking", ICML 2019. It also discussed how to append early exits to pre-trained backbones.


Recent works have developed several methods of defending neural networks against adversarial attacks with certified guarantees. We propose that many common certified defenses can be viewed under a unified framework of regularization. This unified framework provides a technique for comparing different certified defenses with respect to robust generalization. In addition, we develop a new regularizer that is both more efficient than existing certified defenses and can be used to train networks with higher certified accuracy. Our regularizer also extends to an 0 threat model and ensemble models. Through experiments on MNIST, CIFAR-10 and GTSRB, we demonstrate improvements in training speed and certified accuracy compared to state-of-the-art certified defenses.

At a first glance, this paper proposed an interesting refinement of interval bound propagation (IBP). However, it has a major flaw in empirical evaluation, and the proposed "theory" and "bounds" are also questionable and have many issues.
In short, the main results of the paper in Figure 1 and Table 2 are problematic and not the right comparison, so they cannot justify the claim that the proposed method outperforms other state-of-the-art baselines like IBP.  Specifically, when comparing to IBP, the certified error should be computed by IBP; however the verification algorithm used in this paper performs extremely poor on IBP based models (giving vacuous bounds like 0%, and the authors are able to outperform this 0%).  Under fair comparison metrics (Table 12), the proposed method is worse than the IBP baseline in almost all settings. I will explain the reason the proposed method does not work in detail below.
Besides the empirical results, the "theory" developed in this paper also has several fundamental weakness (will discuss in detail below) and are lack of solid connections to robustness and the proposed "bounds"; the proposed "bounds" are questionable and are not sound bounds, and they can hardly be justified theoretically; so it is not surprising that they cannot outperform IBP, which is based on rigorous minimax robust optimization and sound over-approximations of neural networks.
On the positive side, the authors considered the ensemble of multiple IBP trained models, as well as extending IBP to L0 norm setting. Both of them are valid (but small) contributions, but they are not sufficient. Also, overall the writing of the paper is great and easy to follow and understand.
I really do not want to make the authors of this paper upset, especially, the main author might be the first time submitting a paper to ICLR or a undergraduate student new to this field. However I have to say this paper has significant flaws and should not be published. Especially, the wrong evaluation methodology used in this paper can be very misleading for new comers to this field, and misguide future research.
I encourage the authors to read my detailed comments below and learn from the failure of the proposed method. If the authors can rephrase this paper significantly (especially, removing the entire section 3), and emphasize on the contributions of ensemble or L0 perturbation, it might become a good paper for a next venue. My suggestions for improvements are below:
1. Be honest with your findings and do not try to hide the weakness of your method, and do not overclaim. Especially, the authors are aware of the problem that IBP based models are better if evaluated under IBP bounds (in Table 12), but still make strong and wrong claims that the proposed method outperforms other state-of-the-art methods in certified accuracy in Introduction.
2. For the ensemble part, consider more "smart" ensembles rather than directly adding them together. For example, we can consider balancing the accuracy and certified error of each model and choose a blend of them. IBP is a strong method, and an ensemble of IBP can yield the best defense.
3. For the L0 robustness with IBP, it is not a significant contribution alone since it only converts the L0 norm to interval bounds at the very first layer.  However the authors can consider more interesting settings, like adversarial patches or masks (https://arxiv.org/pdf/1712.09665.pdf), which can be dealt with similar techniques.
4. When evaluating a certified defense method, it is also good to conduct PGD attacks to the networks, to see how tight the certified bounds are.  If the authors attack the models in Table 1, we can actually see that IBP based models can perform much better than the proposed method. From this, the authors should have realized that the verification method they used is not appropriate to evaluating IBP.
5. Evaluation on only 200 test data points is not sufficient. Certified accuracy is computed on the entire test set (10,000 examples) in almost all previous certified defense papers (Wong et al., 2018; Mirman et al., 2018; Gowal et al., 2018; Wang et al., 2018). The authors should use a proper implementation of verification algorithm, like DiffAI (https://github.com/eth-sri/diffai), convex adversarial polytope (https://github.com/locuslab/convex_adversarial) or symbolic interval (https://github.com/tcwangshiqi-columbia/symbolic_interval). In my experience, on a single GPU they can verify small models over entire dataset (10,000 examples) within a few minutes; large models may take a few hours, but still quite reasonable. The verification method used in this paper is lesser-known and was probably implemented poorly and inefficiently. It is better to use a mature and well accepted library.
6. A Minor issue: the first paper that proposed IBP training is Mirman et al., ICML 2018 (where the "box" domain was used for training), not Gowal et al. 2018. So some sentences in Introduction and Related works are not accurate.
Now let's discuss the issues in this paper in detail, and let's focus on the empirical comparisons to IBP first.
The authors made the main claim based on Table 2 and Figure 1, where the "certified accuracy" (or most commonly referred to as "verified accuracy") for models trained using the proposed method seems to be higher than other methods, especially IBP. "Certified accuracy" is a lower bound of accuracy under any norm bounded perturbations (given a certain epsilon). Conversely, attack based methods like PGD give an upper bound, as there can be stronger attacks that further decrease accuracy.
There are many neural network verification methods to obtain certified accuracy; some of them can be particularly weak on certain models (giving vacuous lower bounds like 0%).  Generally, you choose the best possible (and computationally feasible) verification method to verify the robustness of a model. For example, if verification algorithm A gives a certified accuracy of 10%, but algorithm B gives 90% for the same model, we should use 90%. As an analogy in the adversarial attack setting, you pick the strongest possible attack to evaluate robustness: a model has high accuracy under weak FGSM attacks is not necessarily robust; conversely, a model has low certified accuracy (even 0%) does not necessarily mean it is vulnerable, as the verification method can be particularly weak on this model.
In the original IBP training paper (Gowal et al., 2018), the certified error is computed efficiently using IBP, and the error is about 8% for MNIST (epsilon=0.3), and 68% for CIFAR (epsilon=8/255).  My first hand experience on IBP can confirm that it is very easy (without too much tuning effort) to get 10% certified error for MNIST and 73-75% for CIFAR, even using small models.  These numbers translate to 90% certified accuracy on MNIST (eps=0.3) and 25-27% certified accuracy on CIFAR (eps=8/255). However, in Table 2 and Figure 1 of the paper the authors show 0% (!) certified accuracy for IBP trained models for both MNIST eps=0.3 and CIFAR eps=8/255, and their method outperforms this 0%.
Unfortunately, the verification method ("cnncert") used in this paper performs extremely poor on IBP trained models (giving vacuous bounds like 0%); IBP trained models should be certified using IBP bounds to give non-vacuous results.  What Table 2 and Figure 1 really show is the weakness of their verification method used, rather than the true robustness of the model. What we really want to show here is how robust the models are, not how good a verification method is, so we need to use the best possible verification method; for IBP trained models, using IBP for verification is almost mandatory since it not only gives tight bounds but is also much more efficient.
The authors are aware of this problem - in appendix, Table 12 (a table never discussed anywhere), they listed IBP certified error for IBP trained models.  The MNIST numbers for IBP trained models are close to those on IBP paper (90% at eps=0.3), significantly better than their method in Table 2 (68%) or Table 12 (79%). The CIFAR numbers for IBP (22.5% certified accuracy at 8/255 in Table 12) are apparently de-tuned (in my experience IBP can easily do at least 25%, and Gowal et al. reported 32%), yet it is still better than the proposed method (less than 20% in Table 2 and 12).  So under the right metrics (IBP trained models certified using IBP), even if the IBP models are detuned, they can outperform the proposed method by a large margin. The proposed method only makes IBP worse under the right metrics.
Now let's understand why the proposed method cannot improve IBP. The bounds themselves have a few issues:
1. The "s" bound is not a sound bound for interval analysis anymore, because it uses the wrong center z_nom (the correct center is (l+u)/2 if you propagate the "center" and "difference" along the network, as an alternative implementation of IBP). The author claims that it is fine since we don't need sound bounds thanks to their "theory", however the "theory" itself is implausible, as will explained below. Although this tampered "s" bounds may empirically help to improve robustness, it is not theoretically sound; training a sound bound helps to obtain better certified accuracy.
2. The "v" bound is claimed to capture second derivative of activation function. However, first of all, for ReLU the second order derivative does not exist at all. The author also argue that "v" is a finite difference based bound, however it is also not accurate since when the bounds propagate to later layers both "s" and "v" can become large, and this can be a very bad "finite difference".
3. I do agree "v" somewhat regularizes linearity (assuming the "finite difference" is partially working). However, linearity does not guaranteed to produces good robustness, nor it is necessary. In fact, we should not impose unnecessary regularization to neural networks, since any regularization restricts its learning power. In some papers on empirical defense, linearity sometimes can help to reduce PGD error; however in the certified setting, a direct surrogate to certifiable robustness like IBP usually produces the best results. The addition of unnecessary regularizations mostly makes results worse, unless you have a very good reason and demonstrate strong empirical evidence that it can significantly outperform the baseline. See https://arxiv.org/pdf/1807.09705.pdf for a case study on the failure of over-regularization.
I think the reason the authors still get a somewhat verifiable model is that the "s" bound sort of propagates a bound that is not sound but carries some similarity to IBP. IBP is a strong method so even tampering it a little bit, you can still get something.  The "v" bound implicitly regularizes the norm of weight matrices, which helps to gain better certified accuracy only under convex relaxation based verification methods.  I believe simply IBP+L1 regularization can achieve similar results as the proposed method, under the *wrong* evaluation metric in Table 2 and Figure 1. Under the correct evaluation metric, we shouldn't add this regularization term at all as it harms performance.
The "theory" developed in section 3 is unconvincing and cannot support the "bounds". There a few problems:
1. The "theory" does not help us to find a good regularizer. When the authors argue that the gradient needs to be close to an "optimal" regularizer, we don't have the optimal regularizer at hand and have no idea how to approach it. Also the inverse Hessian used for distance metric in (6) is never known, so it is impossible to say which gradient is good and which is bad.
2. The assumption that lambda is close to zero is almost never true, yet Proposition 1 and 2 strongly depends on it. In the paper the authors use lambda=0.5 (and other similar numbers) and never decay it to zero. So the proposed training method cannot be supported by the "theory".
3. The "theory" makes weak or no connection to robustness guarantees; (4) is a classical results for the connection between test error and global Lipschitz constant, and the connection between this bound and our goal (robust classifier under adversaries) is too general and too weak. A more direct formulation, like minimax robust optimization will be a much better surrogate.
4. The connection between the theory and the proposed bounds is vague; the authors claim "the gradient of a regularizer rather than its bound validity determines certified test loss", and under this sense, I can use any arbitrarily loss function and call it a "regularizer". For example, I can use a "regularizer" that encourages BAD robustness, and it still fits into the authors explanation. This is like someone publishes a proof showing that P=NP, yet you can use the same argument to show P != NP. This is embarrassing.
The bottom line: I am not saying the propositions in this paper are technically wrong (under the strong assumptions the authors proposed); at least their derivations are straightforward and simple enough to check within a few minutes. However, they are too weak to guide us to find a good training method, too far-fetched to our goal of obtaining good robustness guarantee, and too general that you can use them to prove both sides. So I don't think the "theory" is useful, and the proposed "bounds" guided by the theory has also failed to improve the baseline.
Sorry for the long comments and I hope they can be helpful for the authors.
****** Reply to general author response:
The comparison in the "certifier" table is misleading. "CNN-Cert-Zero" seems to be a special case of CROWN, with a special setting of lower bounds for ReLU. CROWN allows any slope between 0 to 1 as the lower bound, and "CNN-Cert-Zero" is just a special case of that.
Most importantly, the main issue with the paper is not the verifier used; the main issue is that the proposed method performs worse than baseline under correct evaluation, and the proposed "theory" is distracting or wrong.
The new empirical results still do not address any of my concerns - IBP still significantly outperforms the proposed method. For MNIST, Gowal et al. reported over 90% verified accuracy (the proposed method is 75%); the IBP results provided in author response has 0% verified accuracy at , which seems to be a problem or bug. For the CIFAR=8/255 case, the authors keep detuning IBP models and obtain an IBP baseline with less than 20% verified accuracy, yet the IBP model reported in literature (Gowal et al., 2018) can perform over 30%. The proposed method only performs around 20% and the performance gap is huge.
****** Conclusions after author response
After reading the author response, I am still keep my score of reject since the paper contains major technical errors. In a word, the theory is distracting or wrong, and the empirical results provided are intentionally misleading (the proposed method cannot outperform baseline under the right evaluation metrics).
The author response does not address any of my concerns raised, yet the authors insisted that their "theory" is useful (which is apparently not true according to all reviewers) and provided more confusing and misleading results. I have written a long review with detailed reasons and hope the authors can understand why the proposed method fails, but it seems they completely ignored it and did not learn anything from it.
This is quite disappointing.