We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized ondemand to different bit-widths as energy and memory requirements of the application change. Unlike quantization-aware training using the straight-through estimator that only targets a specific bit-width and requires access to training data and pipeline, our regularization-based method paves the way for "on the fly" posttraining quantization to various bit-widths. We show that by modeling quantization as a ∞ -bounded perturbation, the first-order term in the loss expansion can be regularized using the 1 -norm of gradients. We experimentally validate our method on different architectures on CIFAR-10 and ImageNet datasets and show that the regularization of a neural network using our method improves robustness against quantization noise.

This paper shows that if we add L1 regularization on gradient in the training phase, the obtained model can achieve better post-training quantization performance since it is more robust to Linf perturbation on the weights. I like the intuition of the paper but there are several weaknesses:
1. The main concern is that the proposed method cannot outperform quantization-aware fine-tuning. This probably limits the application of the method --- it will only be used when there's not enough time budget for quantization-aware fine tuning for each specific choice of #bits. It will be good if the authors can discuss in what practical scenario their algorithm can be applied.
2. The method is only tested under uniform symmetric quantization. I believe to demonstrate that the L1 regularized models are indeed easier to be quantized, we need to test it on several different kinds of quantizations.
3. I have concerns about the hyper-parameter selection for lambda. The authors mentioned that lambda is chosen by grid-search, but what's the grid search criteria? In other words, are the hyper-parameters trying to minimize the validation error of the "unquantized model", or they are minimizing the validation error of the "post-quantized model"?
4. Some minor suggestions:
- The current paper uses boldfaced n as perturbation which is quite confusing (since small n is the dimension). I would suggest to replace it by something else, e.g, \Delta.
- Section 2.3 seems redundant. It's clearly that L1 regularization is better given it's the dual norm of Linf, so clearly it's better than L2 norm. You have proved L2 is not good anyway in experiments.
===========
After seeing the rebuttal, my concerns about the parameters have been well addressed. Also, I agree with the authors that there are use cases for post quantization, and personally I think post quantization is much easier to do in practice than quantization-aware training. However, this is quite subjective so the fact that the proposed method doesn't outperform quantization-aware training is still a weakness of the paper.
I would like to slightly raise the score to borderline/weak-accept. I hope the authors can have some experiments on non-uniform quantization if the paper is being accepted; I really think that will demonstrate the strength of the method. People will likely to use this method if it can consistently improve many different kinds of post quantization.


We propose a generative latent variable model for unsupervised scene decomposition. Our model, SPACE, provides a unified probabilistic modeling framework to combine the best of previous models. SPACE can explicitly provide factorized object representation per foreground object while also decomposing background segments of complex morphology. Previous models are good at either of these, but not both. With the proposed parallel-spatial attention, SPACE resolves the scalability problem of previous methods and thus makes the model applicable to scenes with a much larger number of objects without performance degradation. Besides, the foreground/background distinction of SPACE is more effective and intuitive than other methods because unlike other methods SPACE can detect static objects that have been difficult to detect as foreground. In experiments on Atari and 3D-Rooms, we show that SPACE achieves the above properties consistently in all experiments in comparison to SPAIR, IODINE, and GENESIS.

Positives:
+The system makes sense and is explained well
+The factoring of scenes into objects and multiple background components is good
+I think overall the experiments are reasonable, although I have a number of questions about whether aspects of them are apples-to-apples
Negatives:
-Some of the experiments do not appear apples-to-apples
-There are a large number of changes, and there aren't any ablations. It's a little hard to follow and verify that the gains are credited properly.
Overall, I'm favorably inclined towards accepting this paper so long as the experiments are more clearly made apples to apples. Right now, since I'm forced to give a binary decision and I'm not positive about comparisons, I have to lean towards rejection -- I'd peg my actual rating as 4.5.
Method:
+The method is well-explained and straight-forward (in a good way).
+The factoring of scenes into objects and multiple background components is good
+The parallelization is good, and the fact that it works far faster than SPAIR with similar results is quite nice
Experiments:
+Overall the experiments are pretty good and compare against the baselines I would expect, and have both qualitative and quantitative results.
+The method appears to do a good job of segmenting the objects, and if Figure 1 is representative, this is quite impressive.
-Why does Figure 1 show results from different systems on different images? This makes comparison impossible. Paired samples are always more informative.
-It's not clear to me that fair comparisons were done, especially to GENESIS.
(a) It's never listed how K for genesis was picked -- this should presumably be tuned somewhere to optimize performance. The paper mentions in the 4.2 that it was impossible to run the experiments for GENESIS for more than 256 components -- but the GENESIS paper has numbers more like K=9. If there are an overabundance of components, this might explain some of the object splitting observed in the paper.
(b) Unless I'm missing something, in Figure 5, for 4x4 and 8x8, it doesn't appear that IODOINE or GENESIS have converged at all. Does the validation MSE just then flatten (or go up) there? This is also wall-clock, so I'm not sure why things would stop there. This seems to conflate training speed with performance (although also note that the wall clock times being discussed are pretty small -- the rightmost side of the graph is ~14 hours -- hardly a burdensome experiment).
(c) Similarly, for 16x16 cells, SPAIR seems to be improving consistently. Is it being cut off?-Figure 5 -- The caption for the figure things appears to not make sense: GENESIS is listed as having K = HxW+5 components and SPACE has K=5 listed. Neither make sense to me. Are they out of order?
(d) For SPAIR in Table 1, it's not clear whether it's the slow SPAIR that was mentioned previously or the fast one (e.g., the predicted boxes are described as the same quality as SPAIR -- but is it the slow SPAIR or fast one?). I think the paper would benefit from being a bit clearer about this. I get that the parallel decomposition, in some sense, may be necessary to get any results. But I wish the paper were a bit more explicit.
-There are some reasonable results. I realize that there isn't existing ground truth on atari and other games, but why not label a few hundred frames manually?
-It would have been nice to have an ablation of some of the components, including the boundary loss. Unfortunately, there's a complex multi-part system and it's not clear how to break off components apart for reuse elsewhere.
Small stuff that doesn't affect my review:
1) Figure 5 -- the figure text size is tiny and should be fixed.
2) Eqn 3, subscript of the product "i" -> "i=1"
3) Table 1 -- captions on tables go on top
4) Now that the systems work like this, I'd encourage the authors to go and try stuff on more realistic data.
5) I would be a little wary of making a big deal out of the discovery of the Montezuma's Revenge key. I realize this is indeed important, but I don't see why something like the slic superpixel objective, or felzenswalb-huttenlocher wouldn't find it either. I think it's great that there's a setting in terms of network capacity (for fg/bg networks) that yields this result, but this seems to depend heavily on the particular networks used for each of the parts of the method, and not on the general method. Also, it seems largely a function of the fact that they're a small region with a different color.
-----------------------------------------
Post-rebuttal update: I have read the authors' response and I am happy to increase my rating to 6.


Generative adversarial networks (GANs) have shown great success in applications such as image generation and inpainting. However, they typically require large datasets, which are often not available, especially in the context of prediction tasks such as image segmentation that require labels. Therefore, methods such as the CycleGAN use more easily available unlabelled data, but do not offer a way to leverage additional labelled data for improved performance. To address this shortcoming, we show how to factorise the joint data distribution into a set of lower-dimensional distributions along with their dependencies. This allows splitting the discriminator in a GAN into multiple "sub-discriminators" that can be independently trained from incomplete observations. Their outputs can be combined to estimate the density ratio between the joint real and the generator distribution, which enables training generators as in the original GAN framework. We apply our method to image generation, image segmentation and audio source separation, and obtain improved performance over a standard GAN when additional incomplete training examples are available. For the Cityscapes segmentation task in particular, our method also improves accuracy by an absolute 14.9% over CycleGAN while using only 25 additional paired examples.

I found this paper very easy and clear to follow - the authors present, what I believe to be an elegant, approach to training a GAN in the presence of missing data or where many marginal samples might be available but very few complete (e.g. paired) samples. The approach proceeds by identifying that the joint distributions (true and approximate) can be factored so as to yield a number of different density ratios which can then be estimated by specific discriminators; in particular, these include the appropriate marginal density ratios and then corresponding overall correction factors. As a caveat to the review I should point out that while I am familiar with GANs, they are not my main area of expertise so this should be taken into consideration - apologies if there is literature I have missed.
Experiments: The authors provide a number of illustrative experiments that demonstrate the efficacy of the approach across a number of tasks. There are many differing GAN models but due to the nature of the problem I don't have a big issue with the majority of the comparisons being against a standard GAN since the tasks are suitably designed. For the paired MNIST experiment I found it hard to assess the qualitative results visually and am always concerned about the ad-hoc nature of Inception Distances - I find it difficult to attribute weight to them quantitatively since they are usually being used to assess things where they might suffer from a common error (e.g. they are both based on NNs). Also, I'm not fully on board with the dependency metric in (5) but then the authors also point out the same concerns. The other experiments I found more convincing.
I appreciated having error bars on some of the plots to help assess significance - would it not be possible to put error bars on all plots?
I found the additional extensions presented in the appendix to be interesting ideas as well and would be interested to see how the approach works with other GAN objectives as mentioned for future work.
I am mostly very positive about this work - my main concern is really common to most GANs - all the analysis relies on the premise that the discriminators can be setup as good estimators for the density ratios. We know that this is not always the case since everything comes from samples and if the capacities of each of the discriminators are not set appropriately then I would expect problems to occur - has this been explored by the authors? It would be no detriment to the work to include failure examples where the authors purposefully make use of inappropriate architectures for some of the discriminators to check for this? For example, there will be large imbalances in the number of training samples used for the different discriminators - how does this affect stability?
Other notes:
- Whilst I understand the point about independent marginals in 2.4 I'm not sure I see the motivation as clearly since it seems that the model is much more useful when there is dependent information but maybe there's a use-case I'm not thinking of?


An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise -these models can now synthesize strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual manifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem.

This paper propose to study the generalization properties of GANs through interpolation. They first propose to learn a linear (and non-linear) interpolation in the latent space for a specific type of image transformation for example zoom, translation, rotation, luminance, etc... They show that linear interpolation in GANs can produce really realistic images along the path and enable to control and transform generated images to some extent. They then propose to measure to what extent the generated images can be transformed without "breaking".  Finally they show that the quality of the interpolation can be improved by learning the interpolation and generator jointly.
I'm in favour of accepting this paper. The paper is well written and organized. The experiments and observations are very interesting and really illustrate the generalization capacity of GANs.
Main argument:
- I think those observations are very valuable to the community and are a good way to get insight into the capabilities of GANs. This also give interesting informations about the different bias present and learnt in the dataset. This could also lead to very nice applications.
- The interpolation with StyleGAN and BigGAN seem to give qualitatively very different results. It would have been very interesting to study the quality of interpolations on more models and datasets, and compare their generalization capabilities as well as the bias present in the different datasets.
- Does training the generator and interpolation jointly improve the quality of the generator in general ? It would have been nice to run this method on more complicated dataset like CIFAR10 and see if this method increase the overall FID score.
Minor comments:
- In appendix A.2 the authors explain how the range of $\alpha$ is set for the different experiments. However it's not clear how is this range used in practice ? Do you sample uniformly $\alpha$ in this range to train the linear interpolation ? Also how many steps are required to learn the linear interpolation ? How much the does it influence the quality of the interpolation ?
- There is a typo in equation 6
- In figure 6: What does the right figure represent ? especially what are the different colours ?


Learning meaningful graphs from data plays important roles in many data mining and machine learning tasks, such as data representation and analysis, dimension reduction, data clustering, and visualization, etc. In this work, for the first time we present a highly-scalable spectral approach (GRASPEL) for learning large graphs from data. By limiting the precision matrix to be a graph Laplacian, our approach aims to estimate ultra-sparse (tree-like) weighted undirected graphs and shows a clear connection with the prior graphical Lasso method. By interleaving the latest high-performance nearly-linear time spectral methods for graph sparsification, coarsening and embedding, ultra-sparse yet spectrally-robust graphs can be learned by identifying and including the most spectrally-critical edges into the graph. Compared with prior state-of-the-art graph learning approaches, GRASPEL is more scalable and allows substantially improving computing efficiency and solution quality of a variety of data mining and machine learning applications, such as spectral clustering (SC), and t-Distributed Stochastic Neighbor Embedding (t-SNE). For example, when comparing with graphs constructed using existing methods, GRASPEL achieved the best spectral clustering efficiency and accuracy.

In this paper, the authors present a method that transforms data into graph. They emphasize on the fact that the proposed method is scalable, using a spectral embedding to construct the graph.
We think that the paper is not of enough quality to be accepted in ICLR. Without going in detail in the derivations, we give below some major issues in this submitted paper.
The studied problem has been widely investigated in the literature. Many methods have been proposed within the same objective, including taking care of the scalability issue. The authors fail to provide the state of the art, as well as describe the contributions with respect to previous work. As a consequence, the contributions are not clear. Maybe the proposed framework is original, but the there has been plenty of methods that have considered the same problem.
Experiments are poor and not convincing. The authors compare the proposed method to only two spectral clustering methods, which as the standard kNN and the Consensus kNN from 2013. These two methods are pretty old and many more recent methods have been introduced in the literature. Moreover, the results in Table 1 are somehow misleading, as the standard kNN is faster that the proposed method on 3 out of 4 datasets. Experiments in graph recovery are not clear, starting from the fact that the datasets are not defined (what are the Gaussian graph and ER graph?), neither the experimental setting (what is the problem at hand?). The same goes to the application of t-SNE which is also very weak.
--------------
Reply to Rebuttal
The authors have modified the paper to take into consideration our previous comments and suggestions. However, we think that it is still of not sufficient quality. We give below some elements, without providing a thorough review.
It is pretty pretentious to say that "this is the first work that introduces a spectral method for learning ultra-sparse (tree-like) graphs from data", while not comparing to the state of the art. There have been many spectral methods in graph learning for large-scale datasets.
In experiments, the only added method is the one of Kalofolias and Perraudin (submitted in 2017 to ArXiv). However, results show that this method is the worst of all methods. It is even the worst compared to the simple standard knn. It is not clear how the authors get such results; It looks like something is wrong in experiments, or they are cherrypicking.


When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together. Similarly, a robot with multiple end-effectors can perform a complex task by coordinating sub-skills of each end-effector. To realize temporal and behavioral coordination of skills, we propose a hierarchical framework that first individually trains sub-skills of each end-effector with skill behavior diversification, and learns to coordinate end-effectors using diverse behaviors of the skills. We demonstrate that our proposed framework is able to efficiently learn sub-skills with diverse behaviors and coordinate them to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside a container while pushing the container with two robot arms, and pushing a box with two ant agents.

The paper presents a hierarchical reinforcement learning method for coordination of multiple cooperative agents with pre-learned adaptable skills. These skills are learned via a maximum entropy objective where diversity of behaviour given each skill is maximised and controlled via a latent conditioning vector. This allows controllability of the variation in skill execution by the meta-policy via changing the skill-specific latent vectors. The paper presents empirical results in manipulation (pick-push-place and moving a long bar by coordinating two Jaco arms) and locomotion (two Ants pushing a large block to a goal location). The method proposed outperforms the baselines reported.
Overall, this paper addresses an interesting problem and can be impactful with the caveat for some clarifications and analysis. Given that the authors address my concerns, I would be willing to increase my score.
The main novelty of this work lies in how one can learn sub-skills that can be leveraged and adapted for down-stream tasks. The problem setting used to test the method is a multi-agent setting where it is crucial that skills are adapted to enable cooperation. I found the environments and the problem setting generally interesting and important for testing the proposed method. I have a few concerns that I listed below:
1) I found the notations at times inconsistent and confusing. It would have helped to see some more details on the diagram (Figure 2) to understand how everything fits together.
2) The set of skills for the two agents are selected by the meta-policy in every T_low steps. It looks like in the Jaco environments T_low = 1. Can you comment on this? This seems slightly concerning since it seems like the meta-controller is treating these skills as primitive actions rather than temporally extended behaviour.
3) Looking at the training curves in Figure 4, there seems to be a really high variance in performance of the method. Can you comment on this as this seems concerning. Could you run more seeds to improve this?
4) It is nice to see in section 4.5 how the hyper-parameters balancing diversity in combination with external reward (equation 4) is tuned and how sensitive that is to achieving adaptability for downstream tasks. The only criticism I have is that it is difficult to understand from "Episode reward" on y-axis what the success rate is (similar to Figure 4)? It would’ve been nice to report results in a consistent way throughout the paper for these environments.
5) Given that all the tasks in the experiments are cooperative multi-agent settings, I would have liked to see more in depth discussion regarding alternative multi-agent methods. The multi-agent baseline provided (which is using a decentralized policy with a shared critic, inspired by Lowe et al., 2017) seems fair, but I wonder if there has been more recent work in this direction that could have been highlighted?


A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training. However, recent work suggests that current machine learning methods are too data inefficient to be trained in this way from scratch. In this paper, we investigate the relationship between two categories of learning signals with the ultimate goal of improving sample efficiency: imitating human language data via supervised learning, and maximizing reward in a simulated multi-agent environment via self-play (as done in emergent communication), and introduce the term supervised self-play (S2P) for algorithms using both of these signals. We find that first training agents via supervised learning on human data followed by self-play outperforms the converse, suggesting that it is not beneficial to emerge languages from scratch. We then empirically investigate various S2P schedules that begin with supervised learning in two environments: a Lewis signaling game with symbolic inputs, and an image-based referential game with natural language descriptions. Lastly, we introduce population based approaches to S2P, which further improves the performance over single-agent methods.

Summary
---
(motivation)
To develop language speaking agents we can teach them to mimic human language
or to solve tasks that require communication. The latter is efficient, but
the former enables interpretability. Thus we combine the two in an attempt
to take advantage of both advantages. This paper studies a variety of ways to
combine these approaches to inform future work that needs to make this tradeoff.
(approach)
The trade-off is studied using reference games between a speaker and a
listener. Goal oriented _self-play_ and human _supervision_ are considered two contraints one
can put on a network during learning. This work considers algorithms that vary
when self-play and supervision are used (e.g., training with self-play then supervision,
or supervision then self-play, or alternating back and forth between the two).
Additional variations freeze the speaker or distill an ensemble of agents into one agent.
(experiments)
A synthetic Object Reference game (OR) and a Image-Base Reference game (IBR) with real images are used for evaluation. Performance is accuracy at image/object guessing.
1. (OR) Like previous work, this work finds that emergent languages are imperfect at supporting their goals and cannot be understood by agents that only understand a human language like English.
2. (OR) Pre-training with supervision then fine-tuning with self-play is superior to pre-training with self-play then fine-tuning with supervision. This is presented as surprising from the perspective of language emergence literature, which is though of as pre-training with self-play.
3. (IBR) Distilling an agent from an ensemble of 50 independently trained agents outperforms training single agents from scratch, but is still not as good as the whole ensemble.
Self-play vs supervision schedules:
4. (IBR) Supervision (using image captions) followed by self-play performs much worse than all other approaches.
5. (IBR) Alternating between supervision and self play (e.g., randomly choosing supervision or self-play every iteration) performs best.
Strengths
---
The curricula considered by this paper seem to have a sigificant impact on performance. These are new and could be important for future work on language learning, which may have considered the sup2sp setting from figure 7a without considering the sched setting.
The diversity of experiments provided and the analysis help the reader get a better sense for how emergent communication models work.
It's nice to see experiments on both a toy setting and a setting with realistic images.
Future directions suggested throughout the paper are interesting.
Weaknesses
---
* The 3rd point of section 5 is presented as a major conclusion of this paper, but it is not very surprising and I don't see how it's very useful. The perspective of language emergence literature is presented a bit strangely. The self-play to supervision baseline seems to be presented as an approach from the language emergence literature. I don't think this is what any of that literature promotes exactly, though it is close. Generally, I (and likely others) don't think it's too surprising that trying to fine-tune a self-play model with language supervision data doesn't work very well, for the same reasons cited in this paper (point 3 of section 5). I think the general strategy when trying to gain practical benefits from self-play pre-training is a translation approach where the learned language is translated into a known language like English rather than trying to directly align it to English as does the supervision approach in this paper. This particular baseline would be more useful if the paper considered learning some kind of translation layer on top of the self-play pre-trained model.
* How significant are the performance differences in figure 7a, especially those between the frozen and non-frozen models? Is the frozen model really better or this performance difference just due to noise?
* I'm somewhat skeptical that these trends will generalize to other tasks/models. The main goal of this paper is to inform future work. That makes it even more important than normal that the trends identified here are likely to generalize well. Are these trends likely to generalize well? Does the paper address when these trends are expected to hold anywhere?
Minor Presentation Weaknesses:
* Figure 4: I think the sub-figures are mis-labeled in the caption.
* In the related work I'm not sure the concept of generations is right. I think it should refer to different languages of different agents across time rather than different languages of the same agent across time.
Missing details / clarification questions:
* What exactly does Figure 4c compare? Are both methods distilled from ensembles or is the blue line normal S2P while the other is distilled from an ensemble of compositional languages? It's not clear since point (3) in section 5 refers to the S2P result (not Pop-S2P) in that plot. I'm also assuming that PB-S2P means the same thing as Pop-S2P, but that's not made clear anywhere. Does PB stand for Population Based?
* In the rand setting how is convergence defined? Do both objectives need to converge or just one?
* In the sched_rand_frz setting what is r?
* In the IBR how are the distractor images picked?
Suggestions:
* Can't both self-play and supervision be used at the same time (just use a weighted combination of the two objectives)? I don't think the paper ever did this but it seems like a very useful variation to consider.
Preliminary Evaluation
---
Clarity: The writing is fairly clear, though some details are lacking.
Significance: This work could help inspire some future models in the language emergence literature.
Quality: Experiments are aligned with the paper's goals and support its conclusions.
Originality: The distillation approach and curricula are novel.
Overall the work could prove to be an interesting and useful reference point inside the language emergence literature so I recommend it for acceptance.


While natural language processing systems often focus on a single language, multilingual transfer learning has the potential to improve performance, especially for low-resource languages. We introduce XLDA, cross-lingual data augmentation, a method that replaces a segment of the input text with its translation in another language. XLDA enhances performance of all 14 tested languages of the crosslingual natural language inference (XNLI) benchmark. With improvements of up to 4.8%, training with XLDA achieves state-of-the-art performance for Greek, Turkish, and Urdu. XLDA is in contrast to, and performs markedly better than, a more naive approach that aggregates examples in various languages in a way that each example is solely in one language. On the SQuAD question answering task, we see that XLDA provides a 1.0% performance increase on the English evaluation set. Comprehensive experiments suggest that most languages are effective as crosslingual augmentors, that XLDA is robust to a wide range of translation quality, and that XLDA is even more effective for randomly initialized models than for pretrained models.

Summary: This paper proposes to augment crosslingual data with heuristic swaps using aligned translations, somewhat like what bilingual humans do in code-switching. I think this paper investigates a neat extension of the XNLI dataset, which is in fact the sort of thing it was created to enable! It also looks at SQuaD translations (but, I'd have preferred a bit more depth on one of these datasets over having both, but I understand why you made this rhetorical choice).
Your augmentation extension to XNLI also uncovers a bunch of surprising results, like that code-switched utterances help models do better than monolingual ones! My main issue, if i had to find one, is that the paper doesn't try to offer (even possible) explanations for the unexpected results; maybe try to find space for more of these in a discussion section? Finally, this paper is really fun and well written, thanks for the effort! I'm going to leave a bunch of questions: it would be cool to see some in the final, but if they don't fit, you can consider them for a follow up.
Questions:
-Are all "portions" full sentences? Did performance change based on which "portions" you swapped?  In the human code-switching literature, there are syntactic generalizations about what gets switched. If you analyze the swapping, you could figure out which parts of the sentence (say, verb phrases v. prepositional phrases, beginning v. middle v. end, etc.) mattered more for NLI performance. I'd love to know the answer to that question!
-you say this: "The BLEU score of the translation system has little effect on a language’s performance as a cross-lingual augmentor. " Any ideas on why?
-you also say this: "for every language a XLDA approach exists that improves over the standard approach", what a tantalizing statement! Why did that happen?!
-Are there any generalizations over whether typologically similar languages are better augmentors for each other than they are for really different ones? I feel like if you could redo your XLR method (fig. 4) by adding augmentors in order from most similar to least (or vice versa), and you might find the answer to this.
-for XNLI, I'd love to see if you have differences by label (maybe in an appendix?)
Small Notes:
-the text in fig1 should be bigger.
-too many Ms and Ls, you had me chuckling at all the acronym puns!
-define "augmentor" somewhere


We proposed Additive Powers-of-Two (APoT) quantization, an efficient nonuniform quantization scheme that attends to the bell-shaped and long-tailed distribution of weights in neural networks. By constraining all quantization levels as a sum of several Powers-of-Two terms, APoT quantization enjoys the overwhelming efficiency of computation and a good match with weights' distribution. A simple reparameterization on the clipping function is applied to generate a betterdefined gradient for updating of optimal clipping threshold. Moreover, weight normalization is presented to refine the input distribution of weights to be more stable and consistent. Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models demonstrating the effectiveness of our proposed APoT quantization. For example, our 5-bit quantized ResNet-50 on ImageNet achieves 76.8% top-1 accuracy without bells and whistles, meanwhile, our model is capable to decrease 31% fixed point computation overhead in uniformly quantized counterpart.

Summary:
The authors propose to compress Neural Networks (NNs) by quantizing their weights to sums of powers-of-two. This allows both to take the non-uniform distribution of the weights into account and to perform fast inference on dedicated hardware.
Strengths of the paper:
- The problem is clearly stated (in particular the two questions about the clipping operation and the quantization levels are clearly presented in the Introduction). Similarly, the authors alleviate what they call the "rigid resolution" problem by using sums of powers-of-two and clearly explain their choice.
- The paper addresses the inference time, which is an important metric. Indeed, researchers tend to focus on the size of the compressed weights as this is easily measurable and not questionable. Inference time however depends on the hardware but is crucial as compressed models often that run on embedded devices are often required to run in real-time (say image detection models). The paragraph "Computation" in Section 2.1 is therefore useful for the reader.
Weaknesses of the paper:
- Both the reparameterization of the clipping function and the weight normalization before quantization approaches seem not novel to me, see for instance: "Weight Normalization based Quantization for Deep Neural Network Compression", Cai et al.
- The experiments are lacking the widely used ResNet-50 baseline (and Table 4 should be in the main paper instead of the appendix). Other tasks such as Image Detection (Mask R-CNN) could also strengthen the impact of the paper.
- Moreover, unless I am mistaken, the authors "do not quantize the first and last layers". While not quantizing the first layer has no impact on the compressed size (weights of size 7x7x3x64 = 36 KB for a ResNet-18), not quantizing the last layer (ie the classifier, weights of size 512x1000 = 1.95 MB) seems problematic. It is indeed challenging to quantize the classifier, however this adds an overhead of 1.95 MB. Assuming the ResNet18 is compressed with 2 bits/weight, the compressed part has a size of  44.6 MB/16 = 2.8 MB. Thus, with the classifier, its size is 2.8+1.95 = 4.75 MB, which is roughly a x10 compression factor.
Justification of rating:
The paper presents an interesting idea that accounts both for the inference time and the bell-shaped distribution of weights in NNs. However, the results must be pushed further (see "weaknesses").
=== After the rebuttal ===
I thank the authors for their rebuttal. They answer to all the raised concerns with clarity and concision and performed additional experiments.
In particular, the authors:
- perform additional ResNet50 experiments
- quantize the input layer and the classifier to 8 bits
- clarifying the positioning and in particular with respect to weight normalization
The paper is substantially better in its updated version. Therefore, I am happy to update my rating to "weak accept".


Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which results in a mixture policy that neither conducts exploration nor task fulfillment resolutely. In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning. Moreover, we introduce a new type of intrinsic reward denoted as successor feature control (SFC), which is general and not task-specific. It takes into account statistics over complete trajectories and thus differs from previous methods that only use local information to evaluate intrinsic motivation. We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite. The results show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives. A video of our experimental results can be found at https://gofile.io/?c=HpEwTd.

## Summary
This paper proposes a novel intrinsic reward for exploration called SFC (successor feature control), to deal with sparse-reward and hard-exploration task. The main idea of SFC is to provide an agent with intrinsic reward defined to be the L2 distance between the successor features of two consecutive states (Equation 4). An underlying motivation of SFC exploration is that high SFC would encourage the agent to enter the "bottleneck states" and therefore helps to explore the entire state space.
Another line of contribution is SID (scheduled intrinsic drive), where a scheduler is used to determine which of the two separate policies (one for extrinsic reward and intrinsic reward) is chosen, with a fixed probability in their implementation, and executed for the next rollout of experience. It has an effect of longer-term exploration and prevents the agent from collapsing to a local-optimum behavior.
Empirically, the SFC+SID algorithm is evaluated on custom sparse-reward navigation-type environments such as VizDoom and DeepMind Lab (as well as a simple pixel-based continuous control), and outperforms other intrinsically motivated RL algorithms including RND and ICM.
## Overall Assessment
Overall, I like the idea of this paper and finds it very interesting and promising, but feel it would be on the borderline or slightly below the bar.
This paper studies a very interesting and novel approach of leveraging successor features for exploration. Successor features are a promising way of learning dynamics-related, task-agnostic representation for RL, which can provide a temporally extended exploration signal. The resulting method presents an improvement over existing intrinsic-reward exploration algorithms.
However, I think there are some weaknesses of the paper that would put the paper slightly below the acceptance threshold: empirically the environments are not diverse enough, and they have an implicit structure assumed and favorable for the proposed method --- there remains a question whether the method is general and not task-specific. I also think there are some misleading overclaims. Please see the detailed comment below.
## Detailed Comments
**[Problem Motivation and Significance]**
This paper address an important, long-standing problem in RL of efficient exploration under sparse-reward environments.
A minor comment: in the introduction, it is said that "terminal reward RL settings" are considered (the reward is given when the goal is achieved) --- which is an extreme case of sparse-reward RL problems --- but in the experiments non-terminal reward environments are studied, e.g. "AppleDistractions" where each of apples yields +0.05 reward. I think the overall claim could be a bit toned down to, for instance, dealing with sparse-reward environments.
**[Clarity]**
Overall the paper is clearly written and easy-to-follow. Descriptions of implementation details are well provided. However, there are some parts that can be better clarified and improved more. Please see more detailed comments below.
**[Justification of Method (SFC & SID)]**
The choice of successor feature for driving a novelty-like intrinsic reward signal seems well-motivated. This is because learning of successor features is task-agnostic and only related to multi-step dynamics (though SF is with respect to "a policy"), which gives a good representation that captures topological characteristics of the environment. The way intrinsic reward is derived, the squared distance of SFs of $S_t$ and $S_{t+1}$, basically encourages the agent to visit and go across the "bottleneck" state.
In the description of methods, it should be clearly noted that what is the underlying policy being learned for deriving successor features (i.e. $\pi$ in Eq.3 and 4) --- for example, is it a behavoral policy (which is a mixture of two policies) induced by scheduled drive? Another comment related to this about "SFC captures statistics over the full distribution of policies that have been followed, ..." (section 2), which sounds a bit overclaiming to me. Please note that a SF is with respect to a specific policy (e.g. behavioral policy), from the expectation in the definition; I think the use of past experience for minimizing the TD error is basically for estimating the expectation term through approximation, so I am not sure that this claim is well-justified.
It is not very clear to me why the SFC reward agrees with bottleneck states. I don't think the explanation given in Section 3.2 is logically enough. Also, isn't it only true under a random exploration policy? How would you defined the "bottleneck states" (e.g. Tomar et al. 2019) -- which can be helpful for making the main idea more understandable? Moreover, there was no enough explanation or reasoning about why SD (successor distance) is roughly the shortest path between the states.
In SID, a policy for extrinsic rewards and another policy for intrinsic one are learned. But in case the extrinsic reward was never received (in case of terminal-reward environments), the former policy would behave no different than random policies. Is this interpretation correct?
Also, given the presented form of SID, it sounds like a bit overclaiming to say it is a hierarchical RL agent, since the scheduler just picks one of the policies with equal probability (rather than being learned) --- especially one policy would become a random exploration policy --- and there is no notion of abstraction or goal/options.
**[Environment choice]**
I feel (1) that the environments being evaluated on are not diverse enough, and (2) that the environment in the experiment seems to exhibit specific properties that are favorable to the algorithm.
(Bottleneck State) One implicit assumption is that the structure of navigation is chosen such that following bottleneck states would lead to an optimal trajectory. I agree that even on the maze like FlytrapEscape the navigation/exploration problem is not easy in the absence of rich reward signals, but this is exactly a sort of environments on which SFC can perform better, especially compared to RND/ICM which are not attracted by bottleneck states (Appendix A). It is good though, and could be beneficial in many cases with the presence of bottleneck states, but seems general applicability is a little bit short (not as much as claimed).
(Distinctive appearance) Another assumption is about a choice of appearance. One important thing to note about SF learning is that a feature for state or transition (cumulant) is kept fixed after random initialization, rather than being learned as in (Machado et al. 2018; Kulkarni et al. 2016; Barreto et al. 2017). This is because this method does not need to do regression of reward function. Then, the state-feature $\phi(s)$ should be discriminative enough so that it can capture some topological and global characteristic of the state space. In general, this is not an easy problem (for first-person view POMDPs), but seems on the environments (FlytrapEscape, AppleDistractions) it was possible because each room/sector has uniquely identifiable wall color and texture. I feel this is somewhat strong assumption made to make SF work. Thus, "We believe this is the first time that SF are shown to behave in a first-person view environment as one would expect from its definition" would sound a bit overclaiming. Would this method work on more general environments that do not have this property --- specifically, what will happen if rooms are not distinguishable from color and texture (and the walls were looking similar)?
Control from pixels (DM Cartpole) is an example of environment that does not have these assumptions, but one downside is that action space was simplified and discretized. Indeed, the improvement shown on Cartpole over ICM/RND is not substantial enough. To demonstrate that SFC+SID is "generally useful" as claimed in the paper, presenting benchmark results on standard discrete-action Atari environments, or more diverse RL environments would have greatly strengthened the paper to be more convincing.
**[Analysis of successor distance]**
Figure 11 (visualization of successor distance) is a great analysis, and I liked it. It clearly shows a smooth topology of the environment thanks to the temporally-extended representation that SF captures. I found that the difference heatmap is a little bit difficult parse. Also, under which policy the SF was computed (I guess this is a behavior policy derived by SID; it should be clearly mentioned somewhere in the paper)?
**[More minor comments about experiments]**
- Was the same K-step objective (e.g. K=5) used for all of SFC, ICM and RND? If so, what would the result look like when K=1?
- The ablation study (appendix 1) is interesting and very important. The "Ours" algorithm in the main text is actually a combination of SFC and SID, so the comparison shown in this ablation study could be a main result.
## Feedback for Improvement
More related work:
* Learning decomposed value functions for extrinsic and intrinsic rewards have been discussed in (Burda et al, 2018b), though in their work a single policy is being learned.
* [Comparison with Machado et al. 2018] It is discussed that (Machado et al. 2018: count-based exploration with SR) is very similar because of the use of SR/SF. The ways of how to derive intrinsic reward signal are indeed different, but it would be great to have a detailed discussion about how they are different or similar.
Minor comments:
* Citation needed on section 3.1 --- (Kulkarni et al. 2016 or Barreto et al. 2017)
* Please consider putting the environment name in the title of each learning curve.
* Typo: Therfore (right before section 3.2)
* Typo: temporarily -> temporally (introduction bullet point 2)


Enhancing model robustness under new and even adversarial environments is a crucial milestone toward building trustworthy and reliable machine learning systems. Current robust training methods such as adversarial training explicitly specify an "attack" (e.g., ∞ -norm bounded perturbation) to generate adversarial examples during model training in order to improve adversarial robustness. In this paper, we take a different perspective and propose a new framework SPROUT, self-progressing robust training. During model training, SPROUT progressively adjusts training label distribution via our proposed parametrized label smoothing technique, making training free of attack generation and more scalable. We also motivate SPROUT using a general formulation based on vicinity risk minimization, which includes many robust training methods as special cases. Compared with stateof-the-art adversarial training methods (PGD-∞ and TRADES) under ∞ -norm bounded attacks and various invariance tests, SPROUT consistently attains superior performance and is more scalable to large neural networks. Our results shed new light on scalable, effective and attack-independent robust training methods.

This work proposes training robust models without explicitly training on adversarial examples and by "smoothing" the labels in an adversarial fashion and by using Dirichlet label smoothing. Training robust models without adversarial training is indeed an important problem as mentioned by the authors since it can potentially (as the authors demonstrate) result in faster model training and less drop in clean accuracy. Overall the idea is interesting but I have some concerns mainly about evaluations and baselines which I am including below. If the authors can address my concerns, I am willing to increase my score:
1. Based on equations (9) and (10), if we set \alpha to be large, then the network is not trainable (since the worst-case adversary will increase the loss on the image by flipping the label during training). As a consequence, we can see that the value of hyper-parameters that the authors use is indeed very small (0.01 and 0.1). Even between these small values, the smaller value results in a better model. In the extreme case, where \alpha is zero there is no regularization and \beta becomes irrelevant. This illustrates that the performance of the model is very sensitive to \alpha. On the other hand, we can prevent the model from not learning anything by constraining \beta in equation (10) — similar to adversarial training where we constrain \delta. It seems that without constraining \beta, if the step-size for \beta is large, \beta can grow and completely mess up the labels even when \alpha is tiny. If we constrain \beta on the other hand, we can make sure that in no case, the top label for any augmented image is an incorrect label. Can the authors elaborate on why they did not set any limit on the value of \beta?
2. What is the batch-size used for ImageNet? The reason that I am asking is that you compute the gradient of \beta for the previous mini-batch but use it for the next mini-batch. Is it possible that the previous mini-batch's \beta is not accurate for the current mini-batch? For CIFAR, since the number of classes is 10, I would assume that you can update the statistics for the class (\betas) using the previous mini-batch since you always see examples from all the classes using any reasonable batch-size. What happens if you do the same but for a dataset with more classes but have the mini-batch be smaller than the number of classes. In this case, your \beta is getting updated only using information from a few classes and not all classes at once. In that case, what happens if you just use a random \beta every step?
3. For the white-box attacks, I also have a few questions. Do you use multiple random restarts? It is known that random restarts can be more effective than increasing the number of PGD attacks. See for example the leaderboards for MNIST and CIFAR-10 challenges by Madry. I would like to see a table where you plot how the accuracy changes by doing 100 step PGD attacks and by increasing the number of random restarts from 1 to 10 for example.
4. Do you do L-infinity CW attacks? I see that you have done L-2 CW attacks but I can't find any L-infinity CW attacks. It would be great to show numbers for that and also compare it with TRADES and PGD adversarial training. In previous smoothing methods, the L-infinity CW attack seems to be a stronger attack compared to PGD.
5. For the ImageNet task, the authors state that the evaluation of non-targeted attacks can result in label leaking. Label-leaking happens when one trains on adversarial examples built using a single-step attack and it means that the accuracy of adversarial examples is higher than natural examples at test-time. For this, I do not understand why the authors mention that they only evaluate targeted attacks while they are not doing any adversarial training.
6. Also, for ImageNet, there are recent methods such as Adversarial training for Free! where the authors do adversarial training on ImageNet with no overhead cost compared to natural training. Maybe this could be added as a better base-line than a naturally trained model.
7. In Figure 4. (a), why is the loss for the validation image illustrated so high? What image is this from the validation set?
8. In terms of Scalability, its good to mention new scalable methods such as YOPO and Adversarial Training for Free.
9. In the ablation study, including Dirichlet Smoothing indeed results in a huge boost compared to having no smoothing. However, it would be better to show that Dirichlet smoothing is indeed better than label-smoothing or adversarial smoothing by including results for other smoothing methods in Fig. 5.


Convolutional neural network (CNN) for time series data implicitly assumes that the data are uniformly sampled, whereas many event-based and multi-modal data are nonuniform or have heterogeneous sampling rates. Directly applying regular CNN to nonuniform time series is ungrounded because it is unable to recognize and extract common patterns from the nonuniform input signals. Converting the nonuniform time series to uniform ones by interpolation preserves the pattern extraction capability of CNN, but the interpolation kernels are often preset and may be unsuitable for the data or tasks. In this paper, we propose the Continuous CNN (CCNN), which estimates the inherent continuous inputs by interpolation, and performs continuous convolution on the continuous input. The interpolation and convolution kernels are learned in an end-to-end manner, and are able to learn useful patterns despite the nonuniform sampling rate. Besides, CCNN is a strict generalization to CNN. Results of several experiments verify that CCNN achieves a better performance on nonuniform data, and learns meaningful continuous kernels.

The paper proposes a continuous CNN model to accommodate the nonuniform time series data. The model learns the interpolation and the convolution kernel functions in an end-to-end manner, so that it can capture the signal patterns and be flexible. A layer has three networks, which learn a kernel function to represent the combination of interpolation and convolution, a bias function to represent the error correction with convolution, and then produce the output based on them. The authors introduce two assumptions and a two-hot encoding scheme for the input to control the model complexity. The paper also introduces an application of the proposed CCNN by combing with temporal point process. Experiments on simulated data compares the proposed method with some degenerative baselines show the advantage of learning the interpolation and the two-hot encoding configuration. The authors compare the performance on time interval prediction task based on real world dataset to show the model produces a better history embedding for the task.
Overall, the paper has some incremental improvements on the existing methods that dealing with the nonuniform time series data. Instead of using preset interpolation kernels, the proposed model can learn it with the convolution in a data-driven manner.
The paper includes clear explanation on module structure and detailed experiment settings.
The experiments of signal value prediction support the claims of the advantages of the proposed model.
The notation in the caption of figure 1 is a little confusing: is x(t_i’) the same as hat x(t) in the algorithm?
It is good that the related works section mentioned the adapted RNNs that are used as baselines in the real-world dataset experiment, and the differences between the proposed model and the related SNNs are introduced.
However, this section and the introduction can be better organized to distinguish the novelty and the contribution of the work.
In page 7, the purpose of the reference in the sentence “The time information is either two-hot encoded (Adams et al., 2010) (CCNN-th), or not encoded (CCNN).” is not very clear.
It will be better if there are a little more analysis of the experiment on predicting time intervals to next event.
The upper plots in the figure may be not convincing enough to support the claims.
The advantage of two-hot encoding seems subtle in the figure 5. Is there any reason for the significantly higher deviation of CCNN-th in StackOverflow?
Sometimes the usage of “CCNN” is not clear, for example, the experiment on speech interpolation compares the “CCNN-th” method with baselines, but uses “CCNN” in the analysis. Also, it could be better to show the “CCNN-th” result in the upper plots of figure 5 instead of “CCNN”.
Minor comment:
There are some typos in the paper, for example, missing the right parenthesis in page3 “(refer to Appendix A.1”, in page4 section 4.1 “According to Eq.(4), the input is …”.
“The left plot shows” in the last line of the caption of figure 3 should be “right”.


Seeking effective networks has become one of the most crucial and practical areas in deep learning. The architecture of a neural network can be represented as a directed acyclic graph, whose nodes denote the transformation of layers and edges represent information flow. Despite the selection of micro node operations, macro connections among the whole network, noted as topology, largely affect the optimization process. We first rethink the residual connections via a new topological view and observe the benefits provided by dense connections to the optimization. Motivated by this, we propose an novel method to optimize the topology of a neural network. The optimization space is defined as a complete graph, and through assigning learnable weights which reflect the importance of connections, the optimization of topology is transformed into learning a set of continuous variables of edges. To extend the optimization to larger search spaces, a new series of networks, called TopoNet, are designed. To further focus on critical edges and promote generalization ability in dense topologies, an auxiliary sparsity constraint is adopted to constrain the distribution of edges. Experiments on classical networks prove the effectiveness of the optimization of topology. Experiments with TopoNets further verify both availability and transferability of the proposed method in different tasks e.g. image classification, object detection, and face recognition.

I. Summary of the paper
This paper describes a principled strategy for searching for the most
suitable neural network architecture out of a particular class of
architectures. Specifically, the problem is framed as an optimisation
problem over a set of directed acyclic graphs (DAGs) that correspond to
potential network architectures. By optimising the edge weights of this
representation, a suitable architecture can be generated.
In addition to the aforementioned optimisation scheme, the paper also
presents a regularisation that results in *sparse* networks, i.e.
networks with a smaller number of edges. Multiple experiments on
'tuning' existing architectures on several data sets conclude the paper.
II. Summary of the review
This paper discusses a highly relevant subject, namely how to select
neural network architectures in a principled manner. While the presented
work already goes into a good direction, I cannot give it my endorsement
for acceptance because of the following reasons:
  - The paper is lacking clarity: concepts could be explained somewhat
    better, and the paper is suffering from language/grammar issues that
    make it harder to understand the contents.
  - Lack of experimental or theoretical depth: the proposed method is
    presented as-is; no theoretical analysis of its behaviour is
    performed; while this is not necessarily a problem, as there are
    several empirical experiments, the experimental section is not
    sufficiently detailed: for example, no limitations of the method are
    being discussed and the presented results are not state-of-the-art
    accuracies.
Nevertheless, I want to point out that this paper has the potential to
become an important contribution to the community; it is absolutely
clear that more principled approaches are required to select network
architectures.
In the following, I will comment on the individual aspects in more
detail.
III. Detailed comments (clarity)
- The abstract could be improved in terms of its logical flow. Instead
  of trying to introduce new terminology (macro/micro etc.) here, the
  abstract should rather state directly that this paper frames network
  architecture selection as an optimisation problem over DAGs.
- The use of topology is slightly non-standard here. What is the meaning
  behind the 'macro' and 'micro' operations? This should be explained
  somewhat better.
- Figure 1 should be extended to show an example of how the depicted
  graphs are described through the terminology mentioned in the paper.
  For example, individual edges or nodes could be highlighted and
  referred to in the text to make the 'mapping' clearer.
- I do not understand how operations such as *addition* are represented
  in the DAG. Ideally, this should also be elucidated by a figure.
- When discussing 'intervals' of residual connections, I am assuming
  that the paper refers to how many layers are skipped? If so, this
  should be mentioned and defined explicitly.
- The term 'searching space' should be replaced by 'search space', as
  the latter is more standard usage.
- I do not understand why the optimisation of the topology can decrease
  the computational burden, as claimed on p. 2. The optimisation process
  still has to be performed, just like the training of the network,
  correct? Am I misunderstanding this?
- The caption of Figure 2 could be extended; does a single node type
  mean that the complete network only consists of nodes of that type?
  Moreover, accuracies/errors should be shown in addition to the loss
  curves.
- The term 'dense connection' is vague; I think the paper should use
  'densely-connected graph' here.
- The sentence 'Among these nodes, [...]' refers to the *whole* network,
  and not to the way the output tensor $\mathbf{x}_i$ is processed. Am I
  understanding this correctly?
- I do not understand the initial sentences in Section 2.2; what is the
  meaning of 'cell' in this case?
- The term 'topological structures' should be renamed to '(sub)graphs'
  in order to improve clarity.
- I do not understand the comment on sparsity in Section 2.4. How are
  'moderate sparsity' and Figure 2 connected?
- In the algorithm, I would use 'Sparsity Type' instead of 'Sparse Type'
  to refer to the parameter.
- What does 'Complete' (without $\mathbf{\alpha}$) mean in Table 3?
- The footnote below Table 3 is not referenced anywhere in the text or
  in the table.
- In Figure 3, are the adjacency matrices consistent? What happens if
  the training process is repeated? It would be highly interesting to
  show 'averaged' matrices over multiple runs.
IV. Detailed comments (experiments & theory)
- A theoretical analysis of the proposed method would be interesting.
  Does the optimisation always converge? Are minima unique? What is the
  computational complexity?
  At least some of these aspects should be discussed.
- The limitations of the proposed method are not explained. For example,
  what is the meaning of the sentence on p. 2 about 'excluding the
  influence of the mixture of different layers/nodes'?
  It is my understanding that the proposed method can only change the
  *connections* between blocks of a network, but not the type of layers.
  Is this correct? If so, it would be a major limitation that should be
  mentioned explicitly.
- Another limitation that is not discussed is the scaling to very deep
  networks. How problematic is it to model all potential connections in
  such a network? Are there limits to the current optimisation scheme?
  This needs to be assessed in the experimental section.
- For all experimental tables, standard deviations and means should be
  provided. This is necessary in order to assess the stability of the
  proposed method, because there are multiple sources of stochasticity:
  one arising from the optimisation procedure, the other one arising
  from the training of the network itself.
- The results reported for the experiments are somewhat behind the
  state-of-the-art in terms of accuracy values. This should be stated
  more clearly; I assume that it is caused by limitations of the
  proposed method, which prohibit an application to very recent
  architectures. Is this correct? If so, it should at least be
  mentioned.
- The claim that nodes at the start of a topological ordering contribute
  more to specific stages needs to be (empirically) proven.
V. Style issues
The paper is not easy to read because of several non-standard phrases or
expressions.
- The phrase 'in topology' is often added to a sentence where it does
  not entirely make sense. For example, '[The] architecture can be
  expressed as a directed acyclic graph (DAG) in topology'. I do not
  see the necessity of adding 'in topology' here. There are other places
  at well from which I would remove this phrase.
- 'effective networks' --> 'effective network architectures'
- 'largely affects' --> 'largely affect'
- 'Motivated by which' --> 'Motivated by this'
- 'innovative method' --> 'method' (or 'novel method')
- 'as a complete graph, through' --> 'as a complete graph, and through'
- 'auxiliary sparsity constraint' --> 'an auxiliary sparsity constraint'
- 'named as TopoNet' --> 'called TopoNet'
- 'At initial periods' --> 'Previously' (I am not sure I understand
  this correctly)
- 'red signs' --> 'red arrows'
- 'for its topology' --> 'in terms of its topology' (?)
- 'both combining' --> 'both a combination' (?)
- 'number of interval' --> 'number of intervals'
- 'straight connected' --> 'directly connected'
- 'conduct transformation' --> 'performs a transformation'
- What is the meaning of the phrase 'These may cover the influence
  [...]'? Is this a reference to limitations of existing networks?
- 'opted from' --> 'chosen from'
- 'Following two simple design rules' --> 'We follow two simple design
  rules'
- '1000-dimension' --> '$1000$-dimensional'
- 'We raise two ways' --> 'We describe two ways'
- 'consited' --> 'consisted' / 'consists'
- 'deepen the depth' --> 'increase the depth'
- 'origin' --> 'original'
- 'promotions' --> 'improvements'
- 'can make more profit' --> 'can be useful to improve performance' (I
  am guessing this from the context)
- 'sparseness on representation' --> 'sparsity'
- 'Adaptive one' --> 'The adaptive one'
- 'At the fore' --> 'At the beginning/start'
- I do not understand the sentence about the 'free lunch'. Does it refer
  to the fact that some connections can still be removed from the
  network without decreasing accuracy?
- 'less computation costs' --> 'lower computation costs'
- 'shortcut offers' --> 'shortcuts offer'
- 'benefits optimization' --> 'benefit optimization'
- 'feasible way to the optimization' --> 'feasible way for the optimization'
- Some references in the bibliography are not capitalised consistently


Recently proposed normalizing flow models such as Glow (Kingma & Dhariwal, 2018) have been shown to be able to generate high quality, high dimensional images with relatively fast sampling speed. Due to their inherently restrictive architecture, however, it is necessary that they are excessively deep in order to train effectively. In this paper we propose to combine Glow model with an underlying variational autoencoder in order to counteract this issue. We demonstrate that such our proposed model is competitive with Glow in terms of image quality while requiring far less time for training. Additionally, our model achieves state-of-the-art FID score on CIFAR-10 for a likelihood-based model.

The paper proposes a combination of a conditioned flow-based model with a VAE. The main contribution of the paper is a two-phase training that allows to train the model. Unfortunately, a joint training of the model failed. In general, combining VAEs with flow-based models is an important research direction. Unfortunately, the paper is not clearly written. A lot of details are missing that makes the paper impossible to reproduce and fully understand. Moreover, the most interesting part about training procedure, is discussed superficially. Finally, I find the comparison to SOTA methods disappointing. The authors skipped many recent papers. I do not expect to see SOTA results among all models, but at least comparable results within a group of approaches. However, the authors provide only a small subset of models that makes me wonder whether they are aware of other papers.
Remarks:
- The following statement is not fully correct:
"In our implementation we use a normalizing flow similar in structure to Real NVP Dinh et al. (2016) (which is a special case of autoregressive normalizing flows Papamakarios et al. (2017)), as it allows both efficient training and sampling"
Real NVP is a bipartite flow, while Masked Autoregressive Flow is an autoregressive flow. In special cases, these two flows coincide, but their original implementations are different.
- The paper misses a lot of important details. For instance, a reader needs to figure out what is the objective function. Further, the authors do not mention how they deal with images represented by integers. Do they use dequantization as in other papers (e.g., Theis, L., van den Oord, A., and Bethge, M.  A note on the evaluation of generative models. In Workshop Track,ICLR, 2016)?  These are very important details to fully understand the proposed approach. Providing a very general diagram (Figure 1) and generic equations (2-4) are not sufficient. Currently, there are different implementations of flow components, and describing them would be definitely beneficial. Moreover, it is important to show how conditioning is used in the flow model.
- Section 3.4 is incredibly interesting part of the paper and it should be further analyzed. The proposed two-phase is reasonable, but it leaves an open question why a joint training fails. I agree with the authors that a possible explanation is training instability. Nevertheless, I would be more than interested in seeing a more thorough analysis of this phenomenon.
- I do not understand why the authors skipped reporting bpd for CelebA.
- In general, the comparison in Table 2 is far from being complete. For instance, please see the following paper:
Ho, J., Chen, X., Srinivas, A., Duan, Y., & Abbeel, P. (2019). Flow++: Improving flow-based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.00275.
On Cifar10, current non-autoregressive models are able to achieve 3.08 bpd (Flow++)  and 3.11 bpd (IAF-VAE). This is much better than the reported 3.17 bpd.
- In Section 4.2.2, first line, a number to a figure is missing.
======== AFTER REBUTTAL ========
I would like to thank the authors for their rebuttal. However, I am not fully satisfied with some answers. First of all, the paper should be clearly written so that a reader could be able to find all necessary details and seamlessly implement the presented idea. If some details do not fit the main text, then they should be included in an appendix. Second, the objective function is an important component of any ML problem and, therefore, it should be included in the paper. Sometimes a verbal description is not sufficient. Third, by conditioning a flow I meant whether the based distribution was conditional or/and a single flow layer was conditioned on z. Currently, there are multiple ways to condition and I was curious which of them was used by the authors. Fourth, I do not believe that achieving nicely looking images is the ultimate task for generative models. However, this is an open discussion. Fifth, I agree, the discussion on why the joint training failed is an incredibly interesting question.
Again, I really appreciate all the answers, and I believe that the authors did their best to improve the paper. However, as a reviewer I must look for novelty and evaluate how the paper is readable for others. In my opinion, combining VAE with GLOW is not sufficiently novel idea. If the paper was very clearly written, I would think about rising the paper. However, right now, I decide to keep my original score.


Collaborative personalization, such as through learned user representations (embeddings), can improve the prediction accuracy of neural-network-based models significantly. We propose Federated User Representation Learning (FURL), a simple, scalable, privacy-preserving and resource-efficient way to utilize existing neural personalization techniques in the Federated Learning (FL) setting. FURL divides model parameters into federated and private parameters. Private parameters, such as private user embeddings, are trained locally, but unlike federated parameters, they are not transferred to or averaged on the server. We show theoretically that this parameter split does not affect training for most model personalization approaches. Storing user embeddings locally not only preserves user privacy, but also improves memory locality of personalization compared to onserver training. We evaluate FURL on two datasets, demonstrating a significant improvement in model quality with 8% and 51% performance increases, and approximately the same level of performance as centralized training with only 0% and 4% reductions. Furthermore, we show that user embeddings learned in FL and the centralized setting have a very similar structure, indicating that FURL can learn collaboratively through the shared parameters while preserving user privacy.

In this paper, the authors propose using federated learning (FL) to train personalized models, which improves the scalability and privacy preservation of the existing personalization techniques. The empirical results show good performance.
However, in general, I think the contribution is limited. The reasons are as follows:
1. The proposed algorithm, FURL, is a direct and simple combination of personalized model and FL. Although the authors claim that there is significant improvement in the performance, such improvement comes from the personalization. And, the personalization itself is not a novel thing (I think the personalized model used in this paper is similar to [1] or some other references. Please correct me if the personalized model used in this paper is new, since I'm not an expert in personalization.) Thus, in general, this paper simply use FL to replace fully synchronous SGD in the training of the personalized models. All the benefits claimed in the introduction, including scalability, privacy preservation, and improvement of performance, come from either vanilla personalization or vanilla FL. I fail to find any new contribution in this combination.
2. The authors emphasize a lot on the "independent aggregation constraint". Although it sounds like such constraint is designed especially for FL + personalization, it is actually a feature only for personalization, which has nothing to do with FL. Note that when doing inference/prediction, each user uses his/her own private part of the model. Different users' private part of models will never affect each other. It is equivalent to training a global model, which concatenates the private parts of models into a big model, and each user update the global model in a sparse manner. Thus, we can also train such personalized model with fully synchronous SGD with sparse gradients, which also does not synchronize the private parts. The private part is never shared by different users, no matter trained by fully synchronous SGD or FL.
------------
References
[1] Jaech, Aaron, and Mari Ostendorf. "Personalized language model for query auto-completion." arXiv preprint arXiv:1804.09661 (2018).


We present an image segmentation method that iteratively evolves a polygon. At each iteration, the vertices of the polygon are displaced based on the local value of a 2D shift map that is inferred from the input image via an encoder-decoder architecture. The main training loss that is used is the difference between the polygon shape and the ground truth segmentation mask. The network employs a neural renderer to create the polygon from its vertices, making the process fully differentiable. We demonstrate that our method outperforms the state of the art segmentation networks and deep active contour solutions in a variety of benchmarks, including medical imaging and aerial images.

EDIT: The rating changed from '1: Reject' to '6: Weak accept' after the rebuttal. See below for my reasoning.
The submission considers two-class image segmentation problems, where a closed-contour image region is to be specified as the 'object'/region of interest, vs. 'no-object'/background. The approach taken here is end-to-end learning with an active-contour type approach. The main loss, in contrast to other active contour approaches, contains a direct difference of the estimated polygon area vs. ground truth polygon area.
The applied method seems conceptually quite simple (as admitted by the authors in Section 5), and the neural rendering approach seems quite neat, but both method presentation (Section 3) and evaluation (Section 4) seem incomplete and leave significant open questions.
One of my main concerns is related to the fact that the displacement field is static and, according to Figure 1 and Algorithm 1, is evaluated only once per image.
If the displacement field J is not conditioned on the current polygon shape (and this does not seem to be the case), then I am wondering why T iterations in the sampling/rendering part are necessary at all. When only considering L_seg, the optimal solution should be found within one iteration, since the displacement field will be able to provide the optimal answer. So maybe these iterations are only necessary when L_B and L_K are incorporated?
In any case, it is unclear why even L_seg is accumulated (using unweighted mean) over all T iterations before being backpropagated. Does this mean that these iterations are not meant to yield shape improvements? Why is ||M^t-M|| not evaluated per iteration, for the purpose of minimization?
It is also not sufficiently clear whether M^t in Equation 4 is a filled polygon mask, or if the mask is just related to the boundary (with a certain width). In absence of explanatory image material, I am assuming the former.
Overall the method description remains weak, since obvious questions/concerns such as the above are not addressed.
The experimental results look good from a quantitative point of view, and indeed, the strongest baselines, e.g. DARNet, are outperformed significantly in many cases.
Section 4 mostly focuses on quantitative evaluation and lots of picture examples, but fails to give insight into particular behaviors, failure cases, etc.
The evaluation procedure is cast a bit into doubt by two things: 1) In Figure 4, the initializations (blue circles) between the DARNet method and the proposed method are very different in size. I am wondering if this then still constitutes a fair comparison, and I have some doubts there. 2) In Figure 6, the proposed method consistently looks much worse than the DARNet baseline (and, in contrast to the baseline, completely fails for 4 vertices), unless the colors were swapped in the description.
Overall, I do not think the submission is in a good enough shape for acceptance.
Minor remarks:
- The values for lambda_1 and lambda_2 seem to come out of thin air, and they also seem quite small. It needs to be mentioned how they were determined.
- Data augmentation by rotation seems to be missing several values (between 270 and 260 degrees) and also not evenly spaced. Is this a typo or on purpose? In the latter case, an explanation is needed, since this seems weird.
- Section 4.3: There is no "Figure 4.2", I assume you mean Figure 6, which otherwise remains unreferenced.
- Section 4.3, Ablation Study: Don't use the word "derivatives" when you're talking about variations.
- Section 4.3, Ablation Study: "even without no auxiliary loss" -> remove "no" or change "without" -> "with"
-------------
Post-rebuttal comments:
I have read the revised version, as well as the other reviews and all authors' comments. The inclusion of an evaluation on a larger-size data set is highly appreciated, and seems to indeed validate the robustness of the method. Typos were fixed, including the switched color descriptions in Figure 7 (which should not have passed initial submission in the first place, if the text had been proofread properly).
Several of the open questions (e.g. "Why is L_seg accumulated before backpropagation?", "Why is the algorithm iterative if the displacement map is computed only once, if not for the other loss terms?", "Choice of values for lambda_1, lambda_2", Initial diameter of initialization") have been somewhat addressed by the authors in the rebuttal comment, though not in great detail.
Based on the quality of the results across data sets, and because I believe that the timely publication of this rather simple method can benefit further research in this area, I have adjusted my score to a 'Weak accept'. That said, I still do not think it is a good manuscript, and my score should be seen as a massive benefit of the doubt toward the authors.
Most importantly, above questions have NOT been adequately addressed in the actual revised text. The authors claim they have "improved the manuscript considerably", but yet I see more reasoning for certain choices described in the comment here than in the actual manuscript. Most of the changes are in Section 2 and the new Section 4.3, but not much relevant to my comments changed in Section 3.
For example, balloon and curvature losses aside, it is still not clear why an iterative approach would be helpful past the first iteration. An ideal displacement map that is not conditioned on the polygon should point, for each pixel, straight to the closest contour pixel. It is clear to me that this may not be what is being learned when multiple iterations are forced, yet it is not addressed why multiple iterations should be beneficial. (I could see why they could be beneficial if the approach was conditioned on the polygon vertices, to avoid vertex collapsing, but it's not.)
A good submission preempts these kinds of questions by addressing them carefully. What seems crystal clear to the authors will not be crystal clear to every reader. The authors should be more careful to include their reasoning in the actual text, which I believe this is essential for proper, easy understanding of the paper.


We propose a new method to train neural networks based on a novel combination of adversarial training and provable defenses. The key idea is to model training as a procedure which includes both, the verifier and the adversary. In every iteration, the verifier aims to certify the network using convex relaxation while the adversary tries to find inputs inside that convex relaxation which cause verification to fail. We experimentally show that this training method is promising and achieves the best of both worlds -it produces a state-of-the-art neural network with certified robustness of 58.1% and accuracy of 78.8% on the challenging CIFAR-10 dataset with a 2/255 L ∞ perturbation. This significantly improves over the currently known best results of 53.9% certified robustness and 68.3% accuracy.

Summary:
This paper provides a promising new general training methodology to obtain provably robust neural networks (towards adversarial input perturbations). The paper provides promising experimental results on CIFAR-10 by obtaining state-of-the-art certified accuracy while also simultaneously improving clean accuracy. The paper is overall well-written and the algorithm is clearly described.
Important questions to be answered:
I find the need to clarify my understanding and request for more information in order to make a decision.
--Methodology/motivation for the method: I am trying to understand abstractly what the proposed layerwise training is trying to optimize. To be concrete, let's compare to the relaxation of Wong and Kolter (which this paper uses in the instantiation of layerwise adversarial training). What's the exact difference?
One way to view this is the following: The same training objective, but a different way to optimize. The new proposal to train involves freezing weights until one layer iteratively starting from the input layer. It is possible that this kind of training provides some inductive bias in finding better solutions. Is this an appropriate understanding?
However, the paper’s experimental results unfortunately change the certification procedure. In other words, they haven’t evaluated the same training objective as that of Wong and Kolter. Hence, it’s not clear if the gains are from the better networks, or better certification method, or network being better suited for certification by the method used. The phrase “same relaxation” is not appropriately used. Their certification procedure uses a different (and tighter) relaxation.
--Effect on latent adversarial examples: I am unable to understand why this training procedure would reduce the number of latent adversarial examples. The definition of latent adversarial examples seems to suggest that it’s the gap between the actual set of activations corresponding to the input perturbations and the convex hull. However, the proposed layerwise adversarial training procedure involves replacing the actual set S_i with the convex hull C_i when freezing things till below i-1. I do not follow how the proposed method tries to make C_i = S_i. Implicitly the optimization objective does try to make C_i small because the bounds being optimized are tighter when C_i is small. But this is true even for normal certified training, and not sure what changes in the new training procedure.
Specific experimental results that would help:
--Certified accuracy on using the same LP based certification procedure used in Wong and Kolter with the new layerwise trained networks
--The paper’s own certification procedure (a combination of previous methods) on the network from Wong and Kolter or a note on why that doesn’t apply (if it doesn’t)
--The paper currently provides only one data point to suggest this training method is superior. Would be good to try SVHN or MNIST. MNIST is perhaps “essentially” solved for small \eps. But would be good to see if the training method offers gains at larger \eps. In general, would be good to see more consistent gains.
--The paper reports results on first 1000 examples of CIFAR10 test set. From my personal experience, there is a lot of variability in the robustness of test examples when evaluated on 1000 random test instances. Especially since the paper doesn't take a random subset, it might be good to make sure the gains are consistent on some other subset. The Wong et al. baseline is evaluated on the entire test set for example, and hence might not be a fair comparison? What's the Wong et al. accuracy on just the first 1000 test exampes
Overall, I am leaning towards accept but need some conceptual and empirical clarification from the authors (detailed above).


In many learning situations, resources at inference time are much more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient at inference time. Given a resource-intensive base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alternatives. The high-level technique can be applied to any neural architecture, and we report experiments on state-of-the-art convolutional neural networks for image classification. For a WideResNet with 97.2% accuracy on CIFAR-10, we improve single-sample inference speed by 2.28× and memory footprint by 5.64×, with no accuracy loss. For a ResNet with 79.15% Top1 accuracy on ImageNet, we improve batch inference speed by 1.29× and memory footprint by 3.57× with 1% accuracy loss. We also give theoretical Rademacher complexity bounds in simplified cases, showing how DARC avoids overfitting despite over-parameterization.

This paper addresses the problem of limited resources that may generally be available at inference time (as compared to the lack of constraints during training time). The paper addresses architecture search as well as model compression by simultaneously optimizing all layers/weights/submodules of the network. The approach is motivated well in terms of comparisons to other work: Although I am not an expert in this particular area of model compression, the intuitive comparisons and mentions of prior work felt satisfying and informative. This problem is definitely timely, and the paper showed results of improved inference speed and memory footprint on both a smaller 10-way classification task as well as a larger 1000-way one.
My current decision is a weak accept, for a well-written paper and an interesting problem, but in the presence of some concerns and suggestions as listed below.
(1) My main concern is that without weight regularization, this optimization problem seems ill-posed. Looking at algorithm 1 specifically, consider the case where j=10 and the learned alphas are indeed sparse, like (0.1, 0, 0,…, 0, 0.9). The cost of J=0 here might be very high (i.e., large, slow, expensive model) and cost of J=9 might be very low (i.e., small fully-connected network that basically maps to an identity function). Here, the cost during optimization could be very low, and the results could be very accurate, but the weights of J=0 may be very high such that the entire work of getting the right answer is from this model. More generally, a low alpha can be attained for any model by simply increasing the network weights themselves. This doesn’t seem to be taken care of anywhere in this paper; if not addressed in any way, it would be a large reason for me to suggest rejecting this work.
(2) For the experiments, both of the chosen tasks were classification. Although one was smaller/easier and one was larger/harder, it would serve to be much more convincing if a task requiring regression and/or higher-dimensional output were tested. For these classification tasks, I can imagine that for certain model changes, the decision boundaries may remain the same even if the network itself is changing in detrimental ways. Thus, a task such as an image-to-image depth perception task (or other tasks in this category) would solidify the results and make it more convincing that DARC does indeed help.
Minor:
(a) The motivation of going from L0 to L1 for the cost constraint seems relevant also to the choice of h being the convex combination of different models. Can the motivation and problem setup address these motivations together, instead of separately?
(b) Although this paper defined R(h) and it also defined h=(\sum (alpha_j * h_j), it would still be helpful for the reader if the paper could explicitly define R(\sum (alpha_j * h_j)) somewhere. Although I know what was intended was to compose these h_j functions as sequential stages to pass data through, the sum notation seems a bit misleading like you may be averaging the final predictions/outputs from different models and computing the loss on these averaged results.


Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet ; (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we match the state-of-the-art on CoNLLAida without linking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as Who was the last inmate of Spandau jail in Berlin?.

This paper proposes to learn entity representations by matching entities to the context it occurs in. It also shows that using these representations is very effective for a wide variety of down-stream entity-centric tasks such as entity typing, linking, and answering entity centric trivia questions. They train the model using a corpus of entity linked Wikipedia contexts (sentences unto length 128 tokens). The context is encoded with a BERT model and the CLS representation is used as the representation of context. After obtaining the representation, they train the entity embedding (present in the sentence) to be similar to the context embedding. They test their embeddings on few down-stream entity-centric tasks — linking, typing and trivia question answering.
Strengths:
1. They try the entity representations on a wide variety of entity-centric tasks and get reasonable results.
Weaknesses:
1. The biggest weakness of the paper is wrt novelty. Masking out entities and training to context is not a new idea. As pointed by the paper, Yamada et al., 2017 have a very similar objective and it is not very clear from the paper what is the additional contribution that this paper makes. Is using pretrained LMs the major difference? If not, it would have been nice to see Yamada et al’s results with BERT. Over all, this paper needs to make its own contribution clear compared to Yamada et al., 2017.
2. The paper needs to be written more clearly at several places. Few examples are, even though in entity linking results (Table 1) the model achieves 83.0 with other papers achieving 90.9. I didnt see a discussion on how to close the gap. Even in the coNLL benchmark, the initial results of the paper is significantly behind. Claims like “CoNLL -Aida is known to be restricted and idiotic-synctatic domain” should be backed by detailed analysis or atleast a citation. Even after finetuning on the CoNLL benchmark, the result is 2.2 points behind state of the art and no discussions have been provided. As a result, I think the entity linking section needs major re-writing and explanation of the results.
3. The paper makes an interesting observation that masking of entities is better for typing tasks and it affects linking performance, because spelling features are really important for linking. It would be interesting to see a discussion on what could be done to remedy this. Because if we have to retrain entity embeddings for different tasks, then it goes against the hypothesis of the paper which is to use entity representations for a wide variety of down-stream tasks.
4. I found it confusing to read the setup in sec 5.4. especially where it says we represent each category with three random exemplars. Initially I thought 3 randomly sampled entities formed a category, which didnt make sense, but from figure 3, I think I understood that you first pick a category from Typenet and Wikipedia and then 3 entities are sampled from there. Is that correct? Regarding the results, can the poor results of Yamada et al., can be understood by the fact that it was trained using smaller number of categories? Also, why are the numbers wrt All entities left blank in Table 4. Given that your model is similar, I am assuming its easy to retrain Yamada et al and test it on the all entities benchmark?