For the past few years, Deep Neural Network (DNN) robustness has become a question of paramount importance. As a matter of fact, in sensitive settings misclassification can lead to dramatic consequences. Such misclassifications are likely to occur when facing adversarial attacks, hardware failures or limitations, and imperfect signal acquisition. To address this question, authors have proposed different approaches aiming at increasing the robustness of DNNs, such as adding regularizers or training using noisy examples. In this paper we propose a new regularizer built upon the Laplacian of similarity graphs obtained from the representation of training data at each layer of the DNN architecture. This regularizer penalizes large changes (across consecutive layers in the architecture) in the distance between examples of different classes, and as such enforces smooth variations of the class boundaries. Since it is agnostic to the type of deformations that are expected when predicting with the DNN, the proposed regularizer can be combined with existing ad-hoc methods. We provide theoretical justification for this regularizer and demonstrate its effectiveness to improve robustness of DNNs on classical supervised learning vision datasets.

To improve the robustness of neural networks under various conditions, this paper proposes a new regularizer defined on the graph of the training examples, which penalizes the large similarities between representations belonging to different classes, thus increase the stability of the transformations defined by each layer of the network.
The paper is overall well written, and the idea involving the Laplacian of the similarity graph is interesting. I have reviewed this paper before. Compared to the previous version, this paper made a good improvement in its experimental results, by adding two different robustness settings in section 4.1 and section 4.3, and also include DeepFool as a strong attack method for testing adversarial robustness.
However, my main concern about the paper is still about its significance.
1. It is still not clear why would this regularization help robustness especially when considering adversarial examples. Example 1 seems not obvious to me why maintaining the boundary margin (rather than expanding or shrinking) is preferred. As stated in the second paragraph in section 3.4, “lower value of \sigma^\ell(s) are indicative of better separation between classes”, what is the reason of not directly penalizing this value, rather than requesting a “stability” property on this value? How is this stability related to the robustness? This would request a deeper analysis and more empirical proofs in the paper.
2. Experimental results still seem not convincing to me. On one hand, based on the reported result, I am not very convincing that the proposed method outperforms Parseval, especially when considering the inconsistent behaviour of “Proposed + Parseval”. On the other hand, for adversarial robustness, the authors should have compared to the method of adversarial training as well. Beyond that, the authors should also be careful of the gradient masking effect of the proposed method. I am not sure if there is some other obvious benchmarks should be included for the other two robustness settings.
Other comments:
1. Descriptions in the last 3 paragraphs in section 3.2 are not very clear. It always took me a while to figure it out every time I read the paper. It would be very helpful if the computation process and the discussions can be separated here, maybe with a pseudo-code for computing the regularizer.
2. On the other hand, while the proposed regularizer can be interpreted in a perspective of the Laplacian of the similarity graph, the third part in Equation (4), that expresses the smoothness as the sum of similarities between different classes, seems more intuitive to me. Emphasizing in this interpretation may also help convey the message.


Effective inference for a generative adversarial model remains an important and challenging problem. We propose a novel approach, Adversarial Inference by Matching priors and conditionals (AIM), which explicitly matches prior and conditional distributions in both data and code spaces, and puts a direct constraint on the dependency structure of the generative model. We derive an equivalent form of the prior and conditional matching objective that can be optimized efficiently without any parametric assumption on the data. We validate the effectiveness of AIM on the MNIST, CIFAR-10, and CelebA datasets by conducting quantitative and qualitative evaluations. Results demonstrate that AIM significantly improves both reconstruction and generation as compared to other adversarial inference models.

This paper presents a variant of the adversarial generative modeling
framework, allowing it to incorporate an inference mechanism. As such it is
very much in the same spirit as existing methods such as ALI/BiGAN. The
authors go through an information theoretic motivation but end up with the
standard GAN objective function plus a latent space (z) reconstruction
term. The z-space reconstruction is accomplished by first sampling z from
its standard normal prior and pushing that sample through the generator to
get sample in the data space (x), then x is propagated through an encoder
to get a new latent-space sample z'. Reconstruction is done to reduce the
error between z' and z.
Novelty: The space of adversarially trained latent variable models has
grown quite crowded in recent years. In light of the existing literature,
this paper's contribution can be seen as incremental, with relatively low novelty.
In the end, the training paradigm is basically the same as InfoGAN, with
the difference being that, in the proposed model,  all the latent
variables are inferred (in InfoGAN, only a subset of the latent
variables are inferred) . This difference was a design decision on the part of the InfoGAN
authors and, in my opinion, does not represent a significantly novel
contribution on the part of this paper.
Experiments: The experiments show that the proposed method is
better able to reconstruct examples than does ALI -- a result is not
necessarily surprising, but is interesting and worth further
investigation. I would like to understand better why it is that latent
variable (z) reconstruction gives rise to better x-space reconstruction.
I did not find the claims of better sample quality of AIM over ALI to be
well supported by the data. In this context, it is not entirely clear what
the significant difference in inception scores represents, though on this, the
results are consistent with those previously published
I really liked the experiment shown in Figure 4 (esp. 4b), it makes the
differences between AIM and ALI very clear. It shows that relative to ALI,
AIM sacrifices coherence between the "marginal" posterior (the distribution
of latent variables encoded from data samples) and the latent space
prior, in favor of superior reconstructions. AIM's choice of trade-off is
one that, in many contexts, one would happy to take as it ensures that
information about x is not lost -- as discussed elsewhere in the paper.
I view this aspect of the paper by far the most interesting.
Summary,
Overall, the proposed AIM model is interesting and shows promise, but I'm
not sure how much impact it will have in light of the existing literature
in this area. Perhaps more ambitious applications would really show off the
power of the model and make it standout from the existing crowd.


Learning to Remember More with Less Memorization
Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.

This paper deals with Memory Augmnted Neural Networks (MANN) and introduces an algorithm which allows full writes to the dense memory to be only exectued every L timesteps. The controller produces a hidden output at most timestps, whih is appended to a cache. Every L steps, soft attention is used to combine this cache of N hidden states to a single one, and then this is used as the input hidden state for the controller, with the outputs performing a write in the full memory M, along with clearing the cache.
The authors first derive "Uniform Writing" (UW) which updates the memory at regular intervals instead of every timestep. The derivation is based on the "contribution" which is norm of the gradient of some input timestep to some hidden state (potentially at a different timestep). I am not clear on whether this terminology for the quantity is novel, if this is the case maybe the authors should state this more clearly. UW says that if all timesteps are equally important, and only D writes can be made in a sequence of length T, then writes should be done every T/(D+1) steps. I have not checked the proof in detail but this seems reasonable that it would maximise the contribution quantity introduced. I am less clear on whether this is obviously the right thing to do - sometimes this value is referred to in relation to information, but that term does not strictly seem to be being used in the information theory sense (no mention of bits or nats anywhere). Regardless, as the authors point out, in real problems there are obviously timesteps which have less or no useful information, and clearly UW is mostly defined in order to build towards CUW.
CUW expands on UW by adding the cache of different hidden states, and using soft attention over them. This feels like a reasonable step, although I would presume there are times when the L hidden states were collected over timesteps with no information, and so the resulting write is not that useful, and times when all of hte L timesteps contain different useful information. In these circumstances it seems like the problem of getting the *useful* information into the memory is still present, as the single write done with the averaged hidden state will need to contain lots of information, which may be more ideal written with several timesteps.
The experiments are well described and overall the paper seems reproducable. The standard toy datasets of copy / reverse / sinusoid are used. The results are interesting - regular DNC with memory size 50 performs surprisingly badly on clean Sinusoid, my guess would be that with hyperparameter tuning this could be improved upon. I'm not sure that using exactly the same hyperparameters for a wide variety of models is appropriate - even with optimizers like Adam and RMSProp, I would want to see at least some sweeping for the best hyperparams, and then graphs like figure 3 should show error bars averaged across multiple runs with the best per-model hyperparameters. However, The DNC with CUW seems to perform well across all synthetic tasks.
There is no mention of Adaptive Computation Time/ACT (Graves, https://arxiv.org/abs/1603.08983) throughout the paper, which is surprising considering Alex Graves' models form two of the baselines used throughout the paper. ACT aims to execute an RNN a variable number of times, usually to do >1 timestep of processing for a single timestep of input. In the context of this paper, I believe it could be adapted to do either zero or one steps of computation per timestep, and that would yield a very comparable network where the LSTM controller always executes, and writes to the memory only happen sometimes. Given that it allows a learned process to decide whether to write, as opposed to having a fixed L which separates full writes, this should have the potential to outperform CUW, as it could learn that at certain times, writes must happen at every step. In my view ACT is attempting to solve essentially the same problem as this paper, so it should either be included as a baseline, or the manuscript should be updated to explain why this is not an appropriate comparison.
I think this is an interesting paper, trying to make progress on an important problem. The results look good, but I can only give a borderline score due to missing ACT numbers, and a few other unclear points. The addition of ACT experiments, and error bars on certain results, would change my mind here.
Notes:
"No solution has been proposed to help MANNs handle ultra long sequence" - (Rae et al 2016) is an attempt to do this, by improving the complexity of reads / writes. This allows bigger memory and longer sequences to be processed.
"Current MANNS only support dense writing" - presumably this means dense as in 'every timestep', but this terminology is overloaded - you could consider NTM / DNC as doing dense writing, and then work of Rae et al 2016 doing sparse writing.
In my experience training these kind of RNNs can have reasonably high variance across seeds - figures 2 & 3 should have error bars, and especially Table 4 as that contains the most important results. Getting 99 percent accuracy when previous SOTA is only 0.1% lower is only really meaningful if the standard deviation across seeds is very small.
Appendix A: the 'by induction' result - I believe there is an error, it should be:
h_t = \sigma_{i=1}^t U_{t-i}W x_i + C
As W is applied to inputs, before the repeated applications of U? I believe the rest of the derivation still holds the same, after the correction.


We propose a novel method for incorporating conditional information into a generative adversarial network (GAN) for structured prediction tasks. This method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data. This method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree. The proposed method is conceptually simpler than joint convolutional neural network -conditional Markov random field (CNN-CRF) models and enforces higher-order consistency without being limited to a very specific class of high-order potentials. Experimental results demonstrate that this method leads to improvement on a variety of different structured prediction tasks including image synthesis, semantic segmentation, and depth estimation.

This paper presents an a particular architecture for conditional discriminators in the cGAN framework. Different to the conventional approach of concatenating the conditioning information to the input, the authors propose to process them separately with two distinct convolutional networks fusing (by element-wise addition) intermediate features of the conditioning branch into the input branch at each layer.
Pros:
+ The writing is mostly clear and easy to follow.
+ I feel that exploring better conditioning strategies is an important direction. Quite often the discriminator discards additional inputs if no special measures against this behaviour are taken.
+ The proposed method seem to outperform the baselines
Cons:
- I’m generally not excited about the architecture as it seems a slight variation of the existing methods. See, for example, the PixelCNN paper [van den Oord et al., 2016] and FiLM [Perez et al., 2017].
- Theoretical justification of the approach is quite weak. The paper shows that the proposed fusion method may result in higher activation values (in case of the ReLU non-linearity, other cases are not considered at all) but this is not linked properly to the performance of the entire system. Paragraph 3 of section 3.1 (sentence 3 and onward) seems to contain a theoretical claim which is never proved.
- It seems that the authors never compare their results with the state-of-the-art. The narrative would be much more convincing if the proposed way of conditioning yielded superior performance compared to the existing systems. From the paper it’s not clear how bad/good the baselines are.
Notes/questions:
* Section 3.1, paragraph 1: Needs to be rephrased. It’s not totally clear what the authors mean here.
* Section 3.1, paragraph 4: “We observed that the fusion …” -  Could you elaborate on this? I think you should give a more detailed explanation with examples because it’s hard to guess what those “important features” are by looking at the figure.
* Figure 4: I would really want to see the result of the projection discriminator as it seems to be quite strong according to the tables. The second row of last column (which is the result of the proposed system) suspiciously resembles the ground-truth - is it a mistake?
* Figure 5: It seems that all the experiments have not been run until convergence. I’m wondering if the difference in performance is going to be as significant when the model are trained fully.
In my opinion, the proposed method is neither sufficiently novel nor justified properly. On top of that, the experimental section is not particularly convincing. Therefore, I would not recommend the paper in its present form for acceptance.


Due to their wide field of view, omnidirectional cameras are frequently used by autonomous vehicles, drones and robots for navigation and other computer vision tasks. The images captured by such cameras, are often analyzed and classified with techniques designed for planar images that unfortunately fail to properly handle the native geometry of such images. That results in suboptimal performance, and lack of truly meaningful visual features. In this paper we aim at improving popular deep convolutional neural networks so that they can properly take into account the specific properties of omnidirectional data. In particular we propose an algorithm that adapts convolutional layers, which often serve as a core building block of a CNN, to the properties of omnidirectional images. Thus, our filters have a shape and size that adapts with the location on the omnidirectional image. We show that our method is not limited to spherical surfaces and is able to incorporate the knowledge about any kind of omnidirectional geometry inside the deep learning network. As depicted by our experiments, our method outperforms the existing deep neural network techniques for omnidirectional image classification and compression tasks.

This paper proposed to use graph-based deep learning methods to apply deep learning techniques to images coming from omnidirectional cameras. It solves the problem of distorsions introduced by the projection of such images by replacing convolutions by graph-based convolutions, with in particular a combinaison of directed graphs which makes the network able to distinguish between orientations.
The paper is fairly well written and easy to follow, and the need for treating omnidirectional images differently is well motivated. However, since the novelty is not so much in the graph convolution method, or in the use of graph methods for treating spherical signals, but in the combined application of the particular graph method proposed to the domain of omnidirectional images, I would expect a more thorough experimental study of the merits of the method and architectural choices.
1. The projected MNIST dataset looks very localized on the sphere and therefore does not seem to leverage that much of the global connectivity of the graph, although it can integrate deformations. Since the dataset is manually projected, why not cover more of the sphere and allow for a more realistic setting with respect to omnidirectional images?
More generally, why not use a realistic high resolution classification dataset and project it on the sphere? While it wouldn't allow for all the characteristics of omnidirectional images such as the wrapping around at the borders, it would lead to a more challenging classification problem. Papers such as [Khasanova & Frossard, 2017a] have at least used two toy-like datasets to discuss the merits of their classification method (MNIST-012, ETH-80), and a direct comparison with these baselines is not offered in this work.
2. The method can be applied for a broad variety of tasks but by evaluating it in a classification setting only, it is difficult to have an estimate of its performance in a detection setting, where I would see more uses for the proposed methods in such settings (in particular with respect to rotationally invariant methods, which do not allow for localization).
3. I fail to see the relevance of the experiments in Section 4.2 for a realistic application. Supposing a good model for spherical deformations of a lens is known, what prevents one from computing a reasonable inverse mapping and mapping the images back to a sphere? If the mapping is non-invertible (overlaps), then at least using an approximate inverse mapping would yield a competitive baseline.
I am surprised at the loss of accuracy in Table 2 with respect to the spherical baseline. Can you identify the source of this loss? Did you retrain the networks for the different deformations, or did you only change the projection of the network trained on a sphere?
4. While the papers describes what happens at the level of the first filters, I did not find a clear explanation of what happens in upper layers, and find this point open to interpretation. Are graph convolutions used again based on the previous polynomial filter responses, sampling a bigger region on the sphere? Could you clarify this?
5. I would also like to see a study of the choice of the different scales used (in particular, size of the neighborhood).
Overall, I find that the paper introduces some interesting points but is too limited experimentally in its current form to allow for a fair evaluation of the merits of the method. Moreover, it leaves some important questions open as to how exactly it is applied (impact of sampling/neighborhood size, design of convolutions in upper layer...) which would need to be clarified and tested.
Additional small details:
- please do not use notation $\mathbb{N}_p$ for the neighborhood, it suggests integers
- p. 4 "While effective, these filters ... as according to Eq. (2) filter..." -> article missing for the word "filter"


Some real-world domains are best characterized as a single task, but for others this perspective is limiting. Instead, some tasks continually grow in complexity, in tandem with the agent's competence. In continual learning there are no explicit task boundaries or curricula. As learning agents have become more powerful, continual learning remains one of the frontiers that has resisted quick progress. To test continual learning capabilities we consider a challenging 3D domain with an implicit sequence of tasks and sparse rewards. We propose a novel agent architecture called Unicorn, which demonstrates strong continual learning and outperforms several baseline agents on the proposed domain. The agent achieves this by jointly representing and efficiently learning multiple policies for multiple goals, using a parallel off-policy learning setup.

Summary
The authors aim to do continual learning to solve dependent tasks using "single-stage end-to-end learning". The resulting "Unicorn" agent trains on all tasks simultaneously. The idea is to use multi-task "off-policy learning", which uses (old) trajectories (experience) from task A to help learning on a related task B. Authors further distinguish between goals (inputs to Q) and tasks (different reward functions). A goal might a color/shape of an object to pick up.
The core model is a UVFA that learns a goal-conditioned Q-function Q(s,a,g).
Some technical aspects:
- use n-step returns.
- when training Q on goal g_i, authors also trajectories that were generated using Q conditioned on another goal(s) g_j. They then truncate the returns for task i when an action taken is not optimal under Q(s,a,g_j) conditioned on goal j. The intuition (seems) to be that this
- authors do not use experience replay or a target Q-function, since the parallelized implementation is reported to be stable enough.
- unicorn sees all train task reward functions during training (but not hold out task rewards).
- unicorn is tested on several 3d maze environments with key-lock etc semantics. The tasks / goals seem simple, and the dependency is defined by changing colors / shapes of objects to be picked up. Authors argue unicorn has to learn to relate task rewards to these goal features.
- unicorn is compared against baselines that 1) do single-task learning (expert) 2) learn on a sum of task rewards (glutton), 3) uniformly random baseline.
- authors show that 1) unicorn performs better on train tasks 2) performs better on hold-out tasks. Also, authors show results for zero-shot transfer learning, with adding abstract tasks (extra reward for picking up any object) improving performance,
Pro
- Simple approach (e.g., no experience replay etc), and uses only a limited set of techniques (e.g., reward truncation).
- Reward performance suggests the model has more properly related goal features to different payoffs.
- Analysis of qualitative behavior is nice.
Con
- The writing is a bit dense in places, e.g., the discussion of baselines is a bit hard to read.
- Description of algorithm is wrapped in long text, a clear algorithm box would make the approach much clearer.
- Not clear what kind of hyperparameters are introduced / used / tuned for Unicorn.
- Authors say "deep dependency", but this seems to just refer to different colors / shapes between objects in the env used in the paper. How is "dependency" between goals and tasks defined in general?
- The experimental setting seems a bit limited, authors only show results on a single domain, and do not offer rigorous definitions. This makes the scope of the paper rather limited.
Reproducibility:
- It's not clear what the variance in the baseline performance is (variance only shown for unicorn).


We describe a DNN for video classification and captioning, trained end-to-end, with shared features, to solve tasks at different levels of granularity, exploring the link between granularity in a source task and the quality of learned features for transfer learning. For solving the new task domain in transfer learning, we freeze the trained encoder and fine-tune an MLP on the target domain. We train on the Something-Something dataset with over 220, 000 videos, and multiple levels of target granularity, including 50 action groups, 174 fine-grained action categories and captions. Classification and captioning with Something-Something are challenging because of the subtle differences between actions, applied to thousands of different object classes, and the diversity of captions penned by crowd actors. Our model performs better than existing classification baselines for SomethingSomething, with impressive fine-grained results. And it yields a strong baseline on the new Something-Something captioning task. Experiments reveal that training with more fine-grained tasks tends to produce better features for transfer learning.

Paper Summary - This paper presents an approach for fine-grained action recognition and video captioning. The authors train a model using both classification and captioning tasks and show that this improves performance on transfer learning tasks. The method is evaluated on the Something-Something v2 dataset as well as a new dataset (proposed in this paper). The authors also evaluate the benefit of using fine-grained action categories vs. coarse-grained action categories on transfer learning.
Paper Strengths
-  Comparing fine-grained vs. coarse-grained action categories for transfer learning is well motivated. Evaluating just this aspect in the context of video classification is helpful (Section 5.1). Establishing the baseline using linear classifiers for feature transfer makes the feature transfer result more robust. The authors have also done a good job of evaluating their method in the coarse-grained and fine-grained settings (Table 1, 2).
- The architectural and experimental design in this paper is well illustrated.
- The 20bn kitchen dataset has interesting categories about intention - pretending to use, using, and using & failing.
- The ablation in Table 1 is helpful in understanding the contribution of 3D vs. 2D convolutions.
Paper Weaknesses
- I believe this paper tries to do too much and as a result fails to show results convincingly. There are too many results and not much focus on analyzing them. In my opinion, the experimental setup in the paper is weak to fully support the authors' claims.
- I now analyze the main contributions of this paper as outlined by the authors in Section 1.
    - Label granularity and feature quality: To me this is the most interesting part of this paper and most related to its title. However, this is also the most under-analyzed aspect. The only result that the authors show is in Sec 5.1 and Fig 5. Apart from using the provided fine-grained vs. coarse-grained labels for evaluation, the authors do not perform many experiments in this domain and neither do they analyze these results. For example, the gain using fine-grained labels is not significant in Figure 5 (2Channel - CG vs. 2Channel - FG). The authors do not explain this aspect. Another missing baseline from Figure 5 is "2Channel - Captions & CG actions". This baseline is needed to understand the contribution of FG vs CG actions when also using captioning as additional supervision.
    - Baselines for captioning: The authors do not provide any details for this task. If the intent is to establish baselines there needs to be more effort on analyzing design decisions - e.g. decoding, layers in LSTM. Captioning metrics such as CIDER and SPICE are missing.
    - Captions as a source for transfer learning: This is poorly analyzed in this paper. 1) Can the captions be converted to "tags" and then used for supervision? What is the benefit of producing the full sequential text description over this simple approach? 2) Captions for transfer learning are only analyzed in Figure 5 without much explanation. It is hard to claim that captioning is the reason for performance gains without really analyzing it completely.
    - 20bn-kitchenware dataset - This dataset is explained in just one paragraph in Section 6. What is the motivation behind collecting this dataset as opposed to showing transfer learning on some other dataset?
- Missing references
        - There has been work in understanding the effect of fine-grained categories in ImageNet transfer learning - What makes ImageNet good for transfer learning? Huh et al. What is the insight provided over this work?
- Minor comments
    - Section 1: Figure 4 is referenced in points 1 & 3. I think you mean Figure 5.


In this work we propose a method to compute continuous embeddings for kmers from raw RNA-seq data, in a reference-free fashion. We report that our model captures information of both DNA sequence similarity as well as DNA sequence abundance in the embedding latent space. We confirm the quality of these vectors by comparing them to known gene sub-structures and report that the latent space recovers exon information from raw RNA-Seq data from acute myeloid leukemia patients. Furthermore we show that this latent space allows the detection of genomic abnormalities such as translocations as well as patient-specific mutations, making this representation space both useful for visualization as well as analysis.

The stated contribution of the paper is the development of a model to learn continuous representations of k-mers from RNA sequencing experiments in an annotation-free manner. The paper motivates this model by considering analysis challenges faced in cancer genomics. This introduction serves well to frame the paper towards addressing these challenges. In particular, the authors highlight challenges faced in recognizing and quantifying patient/tumor-specific RNASeq based expression estimates involving structural variants and indels, which have and continue to be a challenge for existing tools.
Despite this, we are not enthusiastic about the paper for the following reasons:
Narrow and incomplete view of commonly used modern RNA-seq tools/pipelines and their application/use in biomedical research.
The proposed computational method is computationally intractable and is unlikely to ever scale to the genome-wide context.
Described experiments are without context to the existing literature of tools designed to address the biological challenge and by construction are not annotation free.
We further describe these reasons in the following subsections. Overall, we do not believe that the described model/experiments demonstrate utility for either the specific problems in cancer genomics that motivate the paper or the biomedical research field in general.
Narrow and incomplete view of RNA-seq tools/pipelines/application:
------------------
“Reducing this rich data to only the detection of annotated genes [...] is not appropriate for analysis”. Modern RNA-seq pipelines also perform quantification and differential analysis at a minimum.
The description of the standard RNA-seq experiment is problematic:
Sequencing is of cDNA after reverse transcription, not RNA.
Ignores paired end reads (especially with longer insert sizes for fusion detection)
Shredder poor analogy given multiple distinct reads from same sequence, known biases in process
“[...] multiple sequence alignment is an NP-hard problem”. This is true but irrelevant.
Chromosomal translocations are indeed hard to detect by RNA-seq, but not impossible. There are strategies implemented in commonly used tools such as STAR, kallisto, and others to detect these and other structural variants. Individual reads do not have to themselves cover the sequence where translocation occurs, instead read pairs can imply that the insert contains a translocation -- in this case, sequence similarity is much higher. Regardless, cheaper orthogonal assays exist that can detect these events.
“The standard RNA-Seq analysis pipeline has a mean processing times of 28 core hours for mapping with software TopHat, followed by an additional 14 hours of quantification”. See “Please stop using TopHat” (https://twitter.com/lpachter/status/937055346987712512?lang=en) by one of the authors of TopHat and the senior author of the cited paper. Standard pipelines use newer aligners like STAR which are substantially faster.
“Reference based methods yield a relative abundance measurement of genes, which are by definition, hand crafted features”. Relating results back to genes is important to be able to connect sequencing results back to known biology. We see the fact that there is no obvious map from the proposed method back to genetic information as a weakness.
Proposed computational method computationally intractable and unlikely to scale to genome-wide context
------------------------
Plus:
Paper does acknowledge that scalability is a limitation.
Minus:
Lower bound of range of unique kmers per sample without pre-filtering is 10 billion; note that storing counts uncompressed as 32-bit integers corresponds to over 37GB per sample.
Experiments are for only four genes, two at a time with a 2-dimensional embedding. Unclear how patterns will hold when considering k-mers from more genes simultaneously or how embedding could scale. Model formulation suggests that k-mers from co-expressed genes will have similar embeddings, which could complicate visual inspection.
Abstract states that “learned representation both useful for visualization as well as analysis”. Unclear what is done with model/embeddings besides visualization -- non-visual analyses are performed with k-mer counts before embedding. Identification of abnormalities are described only by visual inspection, which is unlikely to scale as more k-mers are added and/or if the dimensionality of the embedding increases.
Experiments without context to existing literature and are not annotation free
----------------------------------------
The paper describes that existing methods are limited by their dependence on annotations. The paper does not describe existing methods using annotations designed to address tasks/applications suggested for the new model. The paper does not compare developed model to these existing methods. To make the experiments performed in the paper computationally tractable, RNA-seq reads are aligned to the reference genome (annotations of sequence) and sequences in specific regions, defined with respect to genes of interest (annotations of genes). The experiments are therefore dependent on annotation although they lose their information/interpretability in this context. The paper notes that many kmers are not included in exonic sequences according to exact matches to annotations of coding sequence. Nonetheless, these reads are in this dataset, which means they are also identified by standard tools/annotations despite their differences. In Figure 3, embedding overlap between kmers in the annotated coding sequence of ZFX and ZFY are illustrated. It would be helpful to show the proportions of reads that were used for this analysis that mapped uniquely vs not to genomic intervals in these genes (reads rather than kmers). In this regard, the absence of any note or methods describing how reads were mapped (what method, with which parameters) to reference (which reference genome, which gene annotations) is a serious limitation.
Other notes
---------------------
The heatmap in Figure 5A is ambiguous and could be improved by better annotating what is on rows/columns, perhaps also including numeric information textually in addition to by color.
Identification of kmers spanning translocation region (illustrated in embedding space by 5B) was done entirely without kmer2vec (identifying exclusive kmers, assembly of kmers, BLAST alignment to two chromosomes). Thus, claim that, as a consequence, “kmer2vec captures real genomic abnormalities and allows to extract directly from the kmer embedding space the abnormal sequence” is unsubstantiated.


EFFICIENT TRAINING ON VERY LARGE CORPORA VIA GRAMIAN ESTIMATION
We study the problem of learning similarity functions over very large corpora using neural network embedding models. These models are typically trained using SGD with random sampling of unobserved pairs, with a sample size that grows quadratically with the corpus size, making it expensive to scale. We propose new efficient methods to train such models without sampling unobserved pairs. Inspired by matrix factorization, our approach relies on adding a global quadratic penalty and expressing this term as the inner-product of two Gramians. We show that the gradient of this term can be efficiently computed by maintaining estimates of the Gramians, and develop variance reduction schemes to improve the quality of the estimates. We conduct large-scale experiments that show a significant improvement in training time and generalization performance compared to sampling methods.

Summary of the paper:
This work presents a novel method for similarity function learning using non-linear model. The main problem with the similarity function learning models is the pairwise component of the loss function which grows quadratically with the training set. The existing stochastic approximations which are agnostic to training set size have high variance and this in-turn results in poor convergence and generalisation. This paper presents a new stochastic approximation of the pairwise loss with reduced variance. This is achieved by exploiting the dot-product structure of the least-squares loss and is computationally efficient provided the embedding dimensions are small. The core idea is to rewrite the least-squares as the matrix dot product of two PSD matrices (Grammian). The Grammian matrix is the sum of the outer-product of embeddings along the training samples. The authors present two algorithms for training the model, 1)SAGram: By maintaining a cache of all embedding vectors of training points (O(nk) space)$, whenever a point is encountered it's cache is replaced with it's embedding vector. 2) SOGram: This algorithm keeps a moving average of the Grammian estimate to reduce the variance. Experimental results shows that this approach reduces the variance in the Grammian estimates, results in faster convergence and better generalisation.
Review:
The paper is well written with clear contribution to the problem of similarity  learning.  My only complain is that, I think the evaluation is a bit weak and does not support the claim that is applicable all kinds of problems e.g. nlp and recommender systems. This task in Wikipedia does not seem to be standard (kind of arbitrary) — there are some recommendation results in the appendix but I think it should have been in the main paper.
Overall interesting but I would recommend evaluating in standard similarity learning for nlp and other tasks (perhaps more than one)
There are specific similarity evaluation sets for word embeddings. It can be found in following papers: https://arxiv.org/pdf/1301.3781.pdf
http://www.aclweb.org/anthology/D15-1036


HIERARCHICAL GENERATIVE MODELING FOR CONTROLLABLE SPEECH SYNTHESIS
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model based on the variational autoencoder (VAE) framework, with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, we train a high-quality controllable TTS model on real found data, which is capable of inferring speaker and style attributes from a noisy utterance and use it to synthesize clean speech with controllable speaking style.

The authors describe the conditioned GAN model to generate speaker conditioned Mel spectra. They augment the z-space corresponding to the identification with latent variables that allow a richer set of produced audio. In a way this is like a partially conditioned model that has "extra" degrees of freedom. It looks that the "latent" variables are just concaneted to the "original" set of z-values (altough with particular conditions to maximize independence). The conditioning of the z-space has originality in it and may provide interesting to the audience. Ultimately one coud think about z-space direction being totally mapped to specific features of the produced signal.
Also, I am curious to know how the Mel spectra are used to produce the actual sound wave - as the phase information is not present if utilizing only the spectral amplitude. Very often this leads to suboptimal generation, and the remedy is to use the time domain like in ( https://arxiv.org/ftp/arxiv/papers/1810/1810.05319.pdf).  However, in this case the audio samples show a pretty nice generation of sound.  However, it is not really end to end.
The manuscript has some curious decisios in its concepts - I do not see the architecture really hiearchial, nor end to end. I would prefer modifications on the paper that concentrate on the truly novel features.
The paper is clear, well written and done with high ambition, from data set utilization  to novel architetures to human quality panels. Results are good and interesting.
NEW:
The authors have addressed the concerns I had with the manuscript.


It is challenging to disentangle an object into two orthogonal spaces of structure and appearance since each can influence the visual observation in a different and unpredictable way. It is rare for one to have access to a large number of data to help separate the influences. In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner. We address this problem in a two-branch Variational Autoencoder framework. For the structure branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge. This encourages the branch to distill geometry information. Another branch learns the complementary appearance information. The two branches form an effective framework that can disentangle object's structure-appearance representation without any human annotation. We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data. We are able to generate photo-realistic images with 256 × 256 resolution that are clearly disentangled in structure and appearance.

Summary: The authors present an autoencoding strategy for disentangling structure and appearance in image data. They achieve this using a learned, spatial prior in the VAE framework.
Writing: The paper contains grammatical errors and I found Section 2 (2.1 and 2.2, specifically) to be a bit confusing as many ideas were described with words when they could have been outlined more precisely mathematically.
Major comments:
The paper takes ideas from Zhang et al and Jakab et al and put them in a VAE context. The paper, however, constructs the ELBO in such a way that distance it from many key ideas of the VAE. Particularly, the paper decomposes the ELBO into three terms and proceed to define these terms as they wish. In this way, the novel contributions of this paper are left unclear and many decisions left unjustified.
- Incorporating landmark/spatial information into autoencoders is not a new idea. Zhang et al and Jakab et al both train autoencoders with disentangled structure, and Finn, 2016 [1] uses a similar spatial landmark strategy when learning representations. Incorporating structural information as a prior distribution (as in this paper) is an interesting idea. However, it is not clear that the defined prior log p(y) is a proper density (integrates to 1). Given the importance of a prior distribution in VAEs, this choice should be precisely justified (maybe relate it to beta-VAE or just use a properly normalizable distribution)
- The paper chooses variational distributions in such a way that removes the entropy term from the KL divergences. Specifically, both q(z | x, y) and p(z | y) have fixed, identity covariance, resulting in the KL divergences equating to L_2 distance. An important part of the VAE is the explicit incorporation of uncertainty by means of learned variances. Although this can sometimes be problematic (see beta-VAE), neglecting to include these variances at all removes an important aspect of VAE.
- The paper includes a likelihood model which also throws away key ideas from VAE. Although some neural likelihood models as in the VAE have issues with blurriness, the reasons are still not completely understood. However, there are well explored alternatives to what the authors propose. For example, many image-based VAEs use Bernoulli likelihoods [2, 3] or autoregressive likelihoods [4]. Autoregressive models, especially, can produce sharp images. The authors introduce a unnormalizable likelihood which combines L1 loss with a function that incorporates L1 distance in VGG space. Using VGG in the likelihood model is unjustified and seems unnecessarily complicated, given the existence of powerful decoders that already exist in VAE literature.
- The authors incorporate various connections and concatenations between neural networks and distributions that further complicate the variational lower bound. For example, p(z | y) is concatenated to q(z | x, y) in addition to acting as a prior on q(z | x, y). This introduces a dependency between the likelihood model and the variational posterior which normally does not exist. Furthermore skip connections are introduced between E_\theta and D_\theta, which complicate the ELBO further. The authors should explicitly write out the loss function they are optimizing at this point or describe how they are modifying ELBO to justify these chocise.
Overall, I find it difficult to call this a variational autoencoder given the liberal modifications to the evidence lower-bound. However, even if I was to interpret this work as an autoencoder with a custom loss function, this model ends up very similar to that in Zhang et al with the main differences being the inclusion of an equivariance constraint in Zhang that is not present in this paper and that Zhang et al use a feature map that’s multiplied by landmarks to incorporate appearance information whereas this paper uses the z representation as appearance information.
The qualitative results of this paper, although good looking, are very similar to qualitative results in Zhang et al and Jakab et al. A quick comment: in Figure 6, you should use the same celebrity faces when comparing Jakab against your own work. The quantitative results only compare to the same model trained without the KL loss term; this, in my mind, is more of a sanity check than a fair baseline. The authors should be comparing against alternate strategies that incorporate spatial information, such as Zhang et al and Jakab et al.
[1] Finn, Chelsea, et al. "Deep spatial autoencoders for visuomotor learning." 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2016.
[2] Chen, Xi, et al. "Variational lossy autoencoder." arXiv preprint arXiv:1611.02731 (2016).
[3] http://ruishu.io/2018/03/19/bernoulli-vae/
[4] van den Oord, Aaron, et al. "Conditional image generation with pixelcnn decoders." Advances in Neural Information Processing Systems. 2016.


The effectiveness of deep neural architectures has been widely supported in terms of both experimental and foundational principles. There is also clear evidence that the activation function (e.g. the rectifier and the LSTM units) plays a crucial role in the complexity of learning. Based on this remark, this paper discusses an optimal selection of the neuron non-linearity in a functional framework that is inspired from classic regularization arguments. A representation theorem is given which indicates that the best activation function is a kernel expansion in the training set, that can be effectively approximated over an opportune set of points modeling 1-D clusters. The idea can be naturally extended to recurrent networks, where the expressiveness of kernel-based activation functions turns out to be a crucial ingredient to capture long-term dependencies. We give experimental evidence of this property by a set of challenging experiments, where we compare the results with neural architectures based on state of the art LSTM cells.

The scope of the paper is interesting: to additionally learn the nonlinear activation function of the neuron.
The insights provided in section 2 with eqs (2)-(5) are interesting and naturally build on the previous work of Poggio & Girosi (1990) and Smola (1998). I found this a nice new insight and the strongest part of the paper. It is e.g. revealing to see to which P and L the rectifier nonlinearity is corresponding.
On the other hand I also have a number of suggestions for further improvement:
- Section 1: related to the overall function to be learned, the authors state "this general problem has been already solved". I think this statement is not completely correct, because depending on the choice of the stabilizer one obtains different optimal representations (e.g. Gaussian RBF or thin plate splines) as explained in Poggio & Girosi (1990). The theory does not tell what the best stabilizer is.
Additional relevant work that would be good to mention at this point, in the area of kernel methods, is e.g. learning the kernel.
- It seems that no other existing work on deep kernel machines has been mentioned in the paper, while in the conclusions the authors state "In this paper we have introduced Kernel-based deep neural networks".
- Related to the training set T_N the notation e^kappa is not explained. It is not clear how this is related to eq (1).
- It would be good to comment on the difference between (3)(4) and Poggio & Girosi (1990).
- unnumbered eq after (5): are there multiple solutions to the problem (non-convex)?
- The explanation of the recurrent network at the end of section 2 is too limited. Moreover, LSTM is not just a neuron nonlinearity, but a recurrent network with a particular structure. To which P and L would LSTM correspond?
- Fig.2: some of the nonlinearities look quite complicated and some of them are oscillatory (is this desirable? it reminds us of overfitting). Often one is interested in activation functions with a "simple shape" like sigmoid, tanh, relu. A more complicated nonlinearity may reduce the interpretability of the model.
- The examples given are rather conceptual (though nice) examples of the proposed method. However, no comparisons with other methods have been made yet in terms of generalization performance, e.g. on a few standard classification benchmark data sets, in comparison with other deep or shallow models.
A possible drawback of the proposed method might be (or maybe not) that additional unknown parameters need to be learned, which could possibly lead to worse generalization. It might be good to further investigate this.


In this paper, we extend the persona-based sequence-to-sequence (Seq2Seq) neural network conversation model to a multi-turn dialogue scenario by modifying the state-of-the-art hredGAN architecture to simultaneously capture utterance attributes such as speaker identity, dialogue topic, speaker sentiments and so on. The proposed system, phredGAN has a persona-based HRED generator (PHRED) and a conditional discriminator. We also explore two approaches to accomplish the conditional discriminator: (1) phredGAN a , a system that passes the attribute representation as an additional input into a traditional adversarial discriminator, and (2) phredGAN d , a dual discriminator system which in addition to the adversarial discriminator, collaboratively predicts the attribute(s) that generated the input utterance. To demonstrate the superior performance of phredGAN over the persona SeqSeq model, we experiment with two conversational datasets, the Ubuntu Dialogue Corpus (UDC) and TV series transcripts from the Big Bang Theory and Friends. Performance comparison is made with respect to a variety of quantitative measures as well as crowd-sourced human evaluation. We also explore the trade-offs from using either variant of phredGAN on datasets with many but weak attribute modalities (such as with Big Bang Theory and Friends) and ones with few but strong attribute modalities (customer-agent interactions in Ubuntu dataset).

===============================
I have read the authors' response and other reviewers' comments carefully. Thank you for taking great efforts to improve the paper, including providing additional results on human evaluation. (Btw, Table 1 and Table 2 are also much nicer now.)
However, from the reviews it seems that all the reviewers agree that the novelty of this paper is limited, and the contribution is incremental.  I understand that this paper is the first and only work using adversarial framework for persona multi-turn conversation models. However, from the modeling perspective, I still think the novelty is limited.
As a summary, I have updated the score from 4 to 5 to reflect the efforts that the authors have been taken to improve the paper. However, due to reasons above, I still prefer a rejection recommendation.
===============================
Contributions:
The main contribution of this paper is the proposed phredGAN, which is a persona-based GAN framework for multi-turn dialogue modeling. Specifically, a persona-based HRED generator is developed, with two different kinds of discriminator design. Experiments are conducted on both the UDC and the TV series transcript datasets.
Weaknesses:
(1) Novelty: I would say the novelty of this paper is rather limited. This paper heavily rely on the previous hredGAN work (Olabiyi et al., 2018), and extends it by injecting attributes into the system, borrowing ideas from the persona-based Seq2seq model (Li et al, 2016b).
phredGAN_a is a straightforward extension of hredGAN, while phredGAN_d further introduces a collaborative discriminator that tries to predict the attribute that generated the input utterance. However, in summary, I think this paper is not novel enough.
(2) Presentation: The paper is generally easy to follow and understand. However, I would say the paper is poorly written, and needs further polishing. For example, Table 1 & 2 are pretty ugly.
(3) Evaluation: Generally, I think the experiments are not convincing and also not well-executed, with detailed comments listed below.
Questions:
(1) In phredGAN_a, as shown in Eqn. (4), the attribute is used as input of the discriminator, while in phredGAN_d, as shown in Eqn. (5) & (6), the attribute is used as the target of the discriminator. My question is: why not use the attribute as both input & output? That is, why not combine (4) & (6), instead of using (5) & (6)? Please clarify this.
(2) In experiments, Section 3.1, the authors mention that the generator and the discriminator use a shared encoder. However, the generator and discriminator has a different role. Since the encoder is shared, then: in one step, we update the encoder to minimize the GAN objective, in the alternative step, we update the encoder again to maximize the GAN objective. So, how to deal with this conflicting role of encoder during the training? Please clarify this.
(3) From Table 2, it seems that it is difficult to see that phredGAN is better than hredGAN. Can you provide some explanations here?
(4) In Table 4, if the responses generated by hredGAN can be provided, that would be better to demonstrate the advantage of phredGAN. How does phredGAN compare with hredGAN qualitatively?
(5) From Table 1 & 2, it seems to me there is no metric that is specifically designed to evaluate whether the model captures the attribute information. Is there a way to quantitatively evaluate this? For example, pretrain an attribute classifier, or use the collaborative discriminator in the phredGAN_d model to measure how the generated response reflect the attribute. If we can observe the performance of phredGAN is better than that of hredGAN, that would be helpful for the paper.
(6) Since the task is challenging, and the automatic metrics designed for this task is not perfect, like other papers, I think human evaluation is essential and desired for this task. However, such human evaluation is lacked in this paper.


It is challenging to train deep neural networks robustly on the industrial-level data, since labels of such data are heavily noisy, and their label generation processes are normally agnostic. To handle these issues, by using the memorization effects of deep neural networks, we may train deep neural networks on the whole dataset only the first few iterations. Then, we may employ early stopping or the small-loss trick to train them on selected instances. However, in such training procedures, deep neural networks inevitably memorize some noisy labels, which will degrade their generalization. In this paper, we propose a meta algorithm called Pumpout to overcome the problem of memorizing noisy labels. By using scaled stochastic gradient ascent, Pumpout actively squeezes out the negative effects of noisy labels from the training model, instead of passively forgetting these effects. We leverage Pumpout to upgrade two representative methods: MentorNet and Backward Correction. Empirical results on benchmark vision and text datasets demonstrate that Pumpout can significantly improve the robustness of representative methods.

A new method for defending against label noise during training of deep neural networks is presented. The main idea is to “forget” about wrongly labeled examples during training by an ascending step in the gradient direction. This is a meta-approach that can be combined with other methods for noise robustness; the detection of the noisy examples is specific of the base method utilised. Experimental results are promising.
In general, I find the idea original, and of potential practical use, but I believe the contribution of paper to be limited and not well supported by experiments. Moreover, I think that some claims made in this work are poorly justified.
== Method
The paper would greatly benefit from some theoretical backing of the proposed optimization scheme, even on a simplified scenario. An idea would be to prove that, given a dataset with noisy labels, PumpOut converges close to the best model (= the one learned without noise), for certain hyperparameters. I think this would be new and interesting. A result of similar fashion was proven in [A].
I found the following arguments not well or only heuristically supported:
* section 2: the scaling factor \gamma. Why using \gamma=1 is suboptimal? One could claim that as much as you want to memorize the true image-label patterns, you also want to forget the image-noise ones.
* Why MentorNet + PumpOut does not suffer from the selection bias effect of CoTraining? This is unclear to me
* The non-negative version of the BackwardCorrection, and its appeal to Kiryo et al 17 is interesting, but it sidesteps its justification. A loss that can be negative does not necessarily means that it not lower-bounded. In fact, for a minimization problem to be well-defined, all you need is a lower bounded objective. Then, adding the lower bound makes your loss non-negative. Notice that Patrini et al 17 did not state that BC is unbounded, but only that it can be negative. Can you show more that that -- maybe, at least experimentally?
The statement of Theorem 2 is trivial. In fact, no proof is given as it would be self-evident. Moreover, the Theorem is not used by PumpOut. Algorithm 3 uses a if-else conditional on the scale of the backward correction, without the max(0, .). I suggest to remove this part. I have notice later that the non-negative version of BC is used as a baseline in the experiment, but I think that is the only use.
Regarding the presentation, in section 3, I suggest to move the explanation of MentorNet and BackwardCorrection before their upgrade by PumpOut.
== Experiments
Table 1 can be removed as these are extremely common datasets.
The experimental results look very promising for applications. As a side effect of this analysis, I can also notice an improvement over BC given to the nnBC, which is nice per se. Although, I would have strengthen the empirics as follow.
* SET2 is only run on MNIST. Why not even on CIFAR10 which is used in SET1? Any future reader will wonder “did it work on CIFAR10?"
* A much harder instance of noise, for instance open set [B] or from a real dataset [Xiao et al 15] would have more clearly supported the use of PumpOut for real applications.
* Can the authors elaborate on “the choices of \beta and \gamma follows Kirkyo et al 2017” ? And how assuming their knowledge gives a fair comparison to BC which does not require them? I believe this is a critical point for the validity of the experiments.
Minor:
* “LRELU active function’ -> activation function. What is a LRELU? LeakyReLU?
[A] Malach, Eran, and Shai Shalev-Shwartz. "Decoupling" when to update" from" how to update"." Advances in Neural Information Processing Systems. 2017.
[B] Veit, Andreas, et al. "Learning From Noisy Large-Scale Datasets With Minimal Supervision." CVPR. 2017.


We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds very little computational overhead during learning and is simple to implement generically in standard deep learning frameworks. Our experiments provide strong evidence that structured gradient regularization can act as an effective first line of defense against attacks based on long-range correlated signal corruptions.

Short paper summary: This work proposes a novel method of gradient regularization (SGR) which utilizes the covariance structure of adversarial examples generated during training. The authors propose simple techniques to reduce the computational overhead of SGR. Empirically, the authors compare their method to standard adversarial training and gradient norm regularization.
Brief review summary: There are some interesting ideas in this work but I feel that the some practical aspects lack formal justification and the comparison to existing work is inconclusive.
Detailed comments:
In addition to some minor comments, I have two concerns. First, with the SGR algorithm itself. And second with the empirical analysis. While I suspect that the first concern may be clarified with discussion I think that the second is more serious and is the primary factor behind my review score.
1) As the SGR algorithm is written I wonder whether the regularization term may be computed more efficiently using something like a Hutchinson trace estimation trick. I suspect that if the random vector used to estimate the trace was the xi from Algorithm 1 then the same Mahalanobis gradient norm would be recovered. This would hold only in the case beta=1, bringing me to my second point.
2) What is the purpose of the running average of the covariance? A relatively small beta value is used in practice but I do not see any strong justification for this. Is there a good reason why we do not want the covariance matrix to be a close approximation for the local gradient landscape? This seems like an important part of the algorithm, especially as it may shed light on my next note.
3) In practice, Algorithm 1 uses adversarial attack schemes to generate the perturbations. In simple cases like FGM, this would give the covariance of the input-output gradient which seems that it would have a direct interpretation as a form of classical gradient regularization. To this extent, I also wonder how the SGR algorithm could be related to interpretations of adversarial training as gradient smoothing (when using small perturbations).
I recognize that the above points are (so far as I could tell) not directly addressed in the work, and some may be fairly considered out of scope. However, due to the direct comparison to adversarial training later and the need to tie SGR to adversarial attacks I feel that it would be important to distinguish these cases.
Overall, I felt that the first three sections did well to introduce the motivation and techniques used and were was easy to follow. The derivation of the SGR algorithm was clear and concise but I believe that some of the practical details (covariance running average, computational efficiency [at first glance, it looks like the full Jacobian must be computed, but practically the sum over K reduces this to a single backprop call]) could have been elaborated on.
For the empirical evaluation the authors provided ample detail on the experimental set up and have performed a fairly thorough investigation in terms of existing defenses and attacks. I felt that the bulk of the study which is contained in Table 1 is fairly inconclusive or at the very least, difficult to interpret completely. Additional comments:
4) I felt that Figure 1 and 2 are a little difficult to interpret at first. It would help to clearly define what is meant by short- and long-range signal corruptions. However, they do suggest some interesting findings. As these covariance matrices depend directly on the model itself, I think it is worth investigate (or commenting on) how this structure may change when introducing things like SGR (or GN). The authors claim that unregularized classifiers give too much weight to short range correlations but they should show that SGN (or other methods) correct this.
5) My biggest concern with this work is with the results presented in Table 1. In terms of how they are presented: first I think that the fool column requires further explanation, or perhaps more simply the column could show accuracy instead of the average perturbation size. Second, I am not sure why the reported accuracies are averaged over attack strengths in a range. So far as I am aware, this is not standard and makes it difficult to interpret the performance of the models in this way. Figure 4 in the appendix does a better job of describing the behavior over a range of attack strengths.
6) From the table, it is not obvious to me that SGR provides any improvements to robustness over existing techniques. Indeed, the authors write that SGR achieves white-box accuracies which are between those of the clean and adversarially trained models and claim that SGR improves on the clean accuracy for CIFAR-10. But in the table the gap between FGSM and GN/SGR clean accuracies seem fairly small with FGSM providing better robustness (for most source attacks). Even more concerning, is the fact that GN seems to outperform SGR. I do not find these results substantial enough to motivate SGR as a robustness defense compared with adversarial training (or even GN), especially as SGR has the same computational limitations involved with expensive adversarial perturbations.
I felt that the study into the covariance structure of adversarial perturbations was interesting but as it stands was not complete enough to be informative in general. In the conclusion the authors write that they provide evidence that current adversarial attacks act by perturbing the short-range correlations of signals but this has only been confirmed for unregularized classifiers. Despite these issues, I thought that the paper was well written and hope that the empirical study can be improved and clarified.
Minor comments:
- Section 2.1, set of transformations only introduced briefly then forgotten. Leaving output invariant confused me, as this does not apply to adversarial examples.
- Section 2.3, second paragraph l3: In Maaten et al. should be citet.
- Section 3.1, should  make clear that derivative is with respect to the data.
- Section 3.1, define delta as the Hessian clearly (it is used for the simplex in the previous section). Though this is easy to figure out.
- Section 7.1, starts with (iii), is this intentional? Perhaps an introductory sentence could make this clearer.
- Section 7.3, for label leaking, I'm not convinced by this argument alone. Assuming the covariance structure is still computed from a particular adversarial example, I see no compelling reason that this would not occur.
Clarity: The paper is very clearly written and is easy to follow.


HARMONIC UNPAIRED IMAGE-TO-IMAGE TRANSLA- TION
The recent direction of unpaired image-to-image translation is on one hand very exciting as it alleviates the big burden in obtaining label-intensive pixel-to-pixel supervision, but it is on the other hand not fully satisfactory due to the presence of artifacts and degenerated transformations. In this paper, we take a manifold view of the problem by introducing a smoothness term over the sample graph to attain harmonic functions to enforce consistent mappings during the translation. We develop HarmonicGAN to learn bi-directional translations between the source and the target domains. With the help of similarity-consistency, the inherent selfconsistency property of samples can be maintained. Distance metrics defined on two types of features including histogram and CNN are exploited. Under an identical problem setting as CycleGAN, without additional manual inputs and only at a small training-time cost, HarmonicGAN demonstrates a significant qualitative and quantitative improvement over the state of the art, as well as improved interpretability. We show experimental results in a number of applications including medical imaging, object transfiguration, and semantic labeling. We outperform the competing methods in all tasks, and for a medical imaging task in particular our method turns CycleGAN from a failure to a success, halving the mean-squared error, and generating images that radiologists prefer over competing methods in 95% of cases.

This paper adds a spatial regularization loss to the well-known CycleGAN loss for unpaired image-to-image translation (Zhu et al., ICCV17).  Essentially, the regularization loss (Eq. 6) is similar to imposing a CRF (Conditional Random Field) term on the network outputs, encouraging spatial consistency between patches within each generated image.
The paper is clear and well written.
Unpaired Image-to-Image translation is an important problem.
The way the smoothness loss (Eq. 6) is presented gives readers the impression that spatial pairwise regularization is new, ignoring its long history (e.g., CRFs) in computer vision (not a single classical paper on CRFs is cited). Putting aside classical spatial regularization works, imposing pairwise regularization on the outputs of modern deep networks has been investigated in a very large number of works recently, particularly in the context of weakly-supervised semantic CNN segmentation, e.g.,  [Tang et al., On Regularized Losses for Weakly-supervised CNN Segmentation, ECCV 18 ], [Lin et al. : Scribblesup: Scribble-supervised convolutional networks for semantic segmentation, CVPR 2016], among  many other works. Very similar in spirit to this ICLR submission, these works impose within-image pairwise regularization (e.g., CRF) on the latent outputs of deep networks, with the main difference that these works use CNN semantic segmentation classifiers whereas here we have a CycleGAN for image generation.
Also, in the context of supervised CNN segmentation, CRFs have made a significant impact when used as post-processing step, e.g., very well known works such as [DeepLab by Chen et al. ICLR15] and [CRFs as recurrent Neural Networks by Zheng et al., ICCV 2015].
It might be a valid contribution to evaluate spatial regularization (e.g., CRFs losses) on image generation tasks (such as CycleGAN), but the paper really needs to acknowledge very related prior works on regularization (at least in the context of deep networks).
There are also related pioneering semi-supervised deep learning works based on graph Laplacian regularization, e.g., [Westen et al., Deep Learning via Semi-supervised embedding, ICML 2008], which the paper does not acknowledge/discuss.
The manifold regularization terminology is misleading. The regularization is not over the feature space of image samples. It is within the spatial domain of each generated image (patch or pixel level); so, in my opinion, CRF (or spatial) regularization (instead of manifold regularization) is a much more appropriate terminology.
Also, I would not call this approach HarmonicGan. I would call it CRF-GAN or Spatially-Regularized GAN. The computation of harmonic functions is just one way, among many other (potentially better) ways to optimize pairwise smoothness terms (including the case of the used smoothness loss). And, by the way, I did not get how the loss in (9) gives a harmonic function. Could you please clarify and give more details? In my understanding, the harmonic solution in [ Zhu and Ghahramani, ICML 2013] comes directly as a solution of the graph Laplacian (and it assumes some labeled points, i.e., a semi-supervised setting). Even, if the solution is correct (which I do not see how), I do not think it is an efficient way to handle pairwise-regularization problems in image processing, particularly when matrix  W = [w_{ij}] is dense (which might be the case here, unless you are truncating the Gaussian kernel with some heuristics). In this case, back-propagating the proposed loss would be of quadratic complexity w.r.t the number of image patches. Again, there is a long tradition in optimizing efficiently pairwise regularizers in vision/learning (even in the case of dense affinity matrices), and one very well-known work, which is currently being used a lot in the context imposing CRF structure on the outputs of deep networks, is  [Krahenbuhl and Koltun, Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials], NIPS 2011. This highly related and widely used inference work for dense pairwise regulation is not cited/discussed neither. The Gaussian filtering ideas of the work of Krahenbuhl and Koltun, which ease optimizing dense pairwise terms (from quadratic to linear) are applicable here (as a Gaussian kernel is used), and are widely used in computer vision, including closely related works imposing spatial regularization losses on the outputs of deep networks, e.g., [Tang et al., On Regularized Losses for Weakly-supervised CNN Segmentation, ECCV 18], among many others.

When using feature from pre-training (VGG) in the CRF loss, the comparison with unsupervised CycleGAN is not fair. In Table 2 (Label translation on Cityscapes), CycleGAN outperforms the proposed method in all metrics when only unsupervised histogram features are used, which makes me doubt about the practical value of the proposed regularization in the context of image-translation tasks. Having said that, the histogram-based regularization is helping in the medical-imaging application (Table 1). By the way, the use of histograms (of patches or super-pixels) as unsupervised features in pairwise regularization is not new neither; see for instance [Lin et al.: Scribblesup: Scribble-supervised convolutional networks for semantic segmentation, CVPR 2016]. Also, it might be better to use super-pixels instead of patches.
So, in summary, the technical contribution is minor, in my opinion (imposing pairwise regularization on the outputs of deep networks has been done in many works, but not for CycleGAN); optimization of the proposed loss as a harmonic function is not clear to me; using VGG in the comparisons with CycleGAN is not fair; and the long history of closely-related spatial regularization terms (e.g., CRFs) in computer vision is completely ignored.
Minor: please use ‘term’ instead of ‘constraint’. These are unconstrained optimization problems and there are no equality or inequality constraints here.


IDENTIFYING AND CONTROLLING IMPORTANT NEURONS IN NEURAL MACHINE TRANSLATION
Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

The authors propose a number of methods to identify individual important neurons in a machine translation system. The crucial assumption, drawn from the computer vision literature, is that important neurons are going to be correlated across related models (e.g. models that are trained on different subsets of the data). This hypothesis is validated to some extent: erasing the neurons that scored highly on these measures reduced BLEU score substantially. However, it turns out that most of the activation of the important neurons can be explained using sentence position. Supervised classification experiments on the important neurons revealed neurons that tracked properties such as the span of parentheses or word classes (e.g., auxiliary verbs, plural nouns, etc).
Strengths:
* The paper is very well written and provides solid intuitions for the methods proposed.
* The methods seem promising, and the degree of localist representation is striking.
* The methods may be able to address the question of *how* localist the representations are (though no numerical measure of localism is proposed).
* There is a correlation between the neuron importance metrics proposed in the paper and the effect on BLEU score of erasing those neurons from the network (of course, it’s not clear what particular linguistic properties are affected by this erasure - the decrease BLEU may reflect inability to track specific word tokens more than any higher-level linguistic property).
Weaknesses:
* It wasn't clear to me why the neurons that track particular properties (e.g., being inside a parentheses) couldn't be identified using a supervised classifier to begin with, without first identifying "important" neurons using the unsupervised methods proposed in the paper. The unsupervised methods do show their strength in the more exploratory visualization-based analyses -- as the authors point out (bottom of p. 6), the neuron that activates on numbers but only at the beginning of the sentence does not correspond to a plausible a-priori hypothesis. Still, most of the insight in the paper seems to be derived from the supervised experiments.
* The particular linguistic properties that are being investigated in the classification experiments are fairly limited. Are there neurons that track syntactic dependencies, for example?
* I wasn't sure how the GMMs (Gaussian mixture models) for predicting linguistic properties from neuron activations were set up.
* It's nice to see that individual neurons function as knobs that can change the gender or tense of the output (with varying accuracy). At the same time, I was unable to follow the authors' argument that this technique could be used to reduce gender bias in MT.
* I wasn't sure what insight was gained from the SVCCA analyses -- this method seems to be a bit of a distraction given the general focus on localist vs. distributed representation. In general, I didn’t come away with an understanding of the pros and cons of each of the methods.


main.pdf
We propose CAML, a meta-learning method for fast adaptation that partitions the model parameters into two parts: context parameters that serve as additional input to the model and are adapted on individual tasks, and shared parameters that are meta-trained and shared across tasks. At test time, the context parameters are updated with one or several gradient steps on a task-specific loss that is backpropagated through the shared part of the network. Compared to approaches that adjust all parameters on a new task (e.g., MAML), our method can be scaled up to larger networks without overfitting on a single task, is easier to implement, and saves memory writes during training and network communication at test time for distributed machine learning systems. We show empirically that this approach outperforms MAML, is less sensitive to the task-specific learning rate, can capture meaningful task embeddings with the context parameters, and outperforms alternative partitionings of the parameter vectors.

Summary
CAML is a gradient-based meta-learning method closely related to MAML. It divides model parameters into disjoint sets of task-specific parameters $\phi$ which are adapted to each task and task-independent parameters $\theta$ with are meta-learned across tasks. $\phi$ are then interpreted as an embedding and fed as input to the model (parameterized by $\theta$). Experiments demonstrate that this approach performs on par with MAML while adapting far fewer parameters. An additional benefit is that this approach is less sensitive to the adaptation learning rate and is easier to implement and faster to compute.
Strengths
While not really explained in the paper, this work connects gradient-based to embedding-based meta-learning approaches. Adaptation is via gradient descent, but the adapted parameters are then re-interpreted as an embedding.
The method has the potential to perform on par with MAML while being simpler and faster.
The paper is well-written.
Weaknesses
The field of meta-learning variants is crowded, and this paper struggles to carve out its novelty.
Rusu et al (LEO) optimize a context vector, which is used to generate model parameters. Reducing the generative model to a point estimate, how is this different from generating the FiLM parameters as a function of context as done in CAML?
Lee and Choi (MT-nets) propose a general formulation for learning which model parameters to adapt. CAML is simpler in that the model parameters to adapt are chosen beforehand to be inputs.
Snell et al. / Oreshkin et al. are prototype-based methods infer context via a neural network rather than optimizing for it.
In this context, CAML appears to be yet another point drawn from the convex hull of choices already explored in episodic meta-learning (these choices can be broadly grouped into task encoding and conditional inference). The paper must then rest on its experimental results, which are at present unconvincing.
On the whole, the experimental results seem weak and analysis results largely uninformative. The method is benchmarked on the toy tasks of sinusoid regression and a 2-D point mass, as well as mini-ImageNet few-shot classification. The sinusoid and point mass navigation are toy and compared only to MAML, so it is hard to draw conclusions from those experiments. For mini-ImageNet, while CAML outperforms MAML, it seems that the pertinent comparison is with MT-NET (which CAML does not outperform) and LEO (missing fair comparison?).
Questions regarding experiments
 - CAML is robust to the adaptation learning rate, but isn’t this true of any scheme that separates meta-learned and adapted parameters into disjoint sets? (e.g. also true of Lee and Choi?)
 - The visualizations of the context parameters are nice, but interpreting much higher dimensional context vectors (which would be necessary for harder tasks) is more difficult, so I’m not sure what to take away from this? It’s very unsurprising that the 2-D context vector encodes x and y position in the point mass experiment, for example.
 - I am confused by the comparison between adapting input parameters versus subsets of nodes at each layer or entire layers for the sinusoid regression task. Adapting subsets of nodes at each layer roughly corresponds to Lee and Choi, yet the reported numbers are quite different?
 - In Table 3, which CAML is a fair comparison (in terms of network size and architecture) to MT-NET?
Editorial Notes
Intro paragraph 3: fine-tuning image classification features for a semantic segmentation task is not a good example of task independent parameters, since fine-tuning end-to-end gives significant improvements.
Related work paragraph 2: Initializing context parameters to zero is not the only difference with Rei et al (2015), and seems a strange thing to highlight?
Tables 1 and 2: state what the task is in the caption


We develop a probabilistic extension of state-of-the-art embedding models for link prediction in relational knowledge graphs. Knowledge graphs are collections of relational facts, where each fact states that a certain relation holds between two entities, such as people, places, or objects. We argue that knowledge graphs should be treated within a Bayesian framework because even large knowledge graphs typically contain only few facts per entity, leading effectively to a small data problem where parameter uncertainty matters. We introduce a probabilistic reinterpretation of the DistMult  and ComplEx (Trouillon et al., 2016) models and employ variational inference to estimate a lower bound on the marginal likelihood of the data. We find that the main benefit of the Bayesian approach is that it allows for efficient, gradient based optimization over hyperparameters, which would lead to divergences in a non-Bayesian treatment. Models with such learned hyperparameters improve over the state-of-the-art by a significant margin, as we demonstrate on several benchmarks.

Summary:
The paper presents a probabilistic treatment of knowledge graph embeddings, motivating it in parameter uncertainty estimation and easier hyperparameter optimisation. The authors present density-based DistMult and ComplEx variants, where the posterior parameter distributions for entity and relation embeddings are approximated by diagonal Gaussians q_\gamma. Variational EM is used to infer the variational parameters \gamma as well as the per-entity/per-relation precision (\lambda) hyperparameters. The training process proposed by the authors consists of three phases: (1) pretraining a MAP estimate that’s used as initial means of the posterior approximating Gaussians, (2) variational EM (see above) to find better hyperparameters and (3) another MAP training phase that uses the updated per-entity/per-relation hyperparameters. Finally, experimental results indicate a slight improvement in MRR and HITS@10 across FB and WN datasets.
Originality:
To the best of the reviewer’s knowledge, the presented approach is novel for knowledge graph embeddings.
Discussion:
While the task is relevant, it is unclear how significant the improvements are. While overall, the proposed method seems to indicate small improvement upon a very strong baseline, in some cases it’s very close (96.2 vs 96.4 HITS@10 on WN18, 36.4 vs 36
.5 MRR on FB15K237), or worse (85.8 vs 85.4 MRR on FB15K).
It is unclear how adequate some details in the experimental setup are for verifying the main hyperparameter optimization claim. In particular, what is “a reasonable choice of hyperparameters” in the first training phase? From figure 3b it seems the initial lambda’s are set proportionately to the frequency, as in the baseline. Are the initial hyperparameter values in EM set the same as the hyperparameter values used for MAP in the reported results? If the claim is to optimize hyperparameters, shouldn’t their initial values be set as uninformed as possible? How do different initial hyperparameter values affect final performance?
The authors claim that the improvement is most notable for entities with fewer training points, however, this is only investigated by using a balanced MRR, where the results are again very close, the same (WN18) or worse (FB15K) for ComplEx. Wouldn’t it be clearer to perform a separate evaluation only considering low-frequency entities to verify this claim?
Parameter uncertainty is not further handled in the paper, the final approach is a point estimate, which discards the uncertainties obtained by VI. Authors mention (last paragraph of Sec. 4) that for a large embedding dimension, bayesian predictions are worse, while for small dimension, they are better. The author’s hypothesis is that a more flexible posterior approximation could solve this issue. No concrete numbers or further analysis are provided.
Clarity and presentation:
The result tables should be merged and formatted better.
Figures need some work (Fig. 2 looks poorly scaled, all figures should be in vector format for scalability, typos in Fig. 1)
Questions:
- How much additional computation is needed to achieve the reported results?
- Would it be possible to group the entities in bins by frequencies (say 6-10 bins) and assign each bin a hyperparameter, and run grid search over just 6-10 hyperparameters, and then interpolate between the bins to set hyperparameters per entity as a function of its frequency?


High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents a general technique toward 8-bit low precision inference of convolutional neural networks, including 1) channel-wise scale factors of weights, especially for depthwise convolution, 2) Winograd convolution, and 3) topology-wise 8-bit support. We experiment the techniques on top of a widely-used deep learning framework. The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining. We perform a systematical and comprehensive study on 18 widely-used convolutional neural networks and demonstrate the effectiveness of 8-bit low precision inference across a wide range of applications and use cases, including image classification, object detection, image segmentation, and super resolution. We show that the inference throughput and latency are improved by 1.6X and 1.5X respectively with minimal within 0.6% 1 to no loss in accuracy from FP32 baseline. We believe the methodology can provide the guidance and reference design of 8-bit low precision inference for other frameworks. All the code and models will be publicly available soon.

This paper designs a system to automatically quantize the CNN pretrained models. This system contains three main components: 1) different scale factors for channel-wise network; 2) Winograd 8bit quantization; 3) topology wise 8bit operation support. All these three techniques are standard ways to perform model quantization. The work is solid in the sense that 1) as far as I know there is no work actually using all of these quantization schemes, and designs a system to automatically do quantization with additional algorithm support(retrain strategy). 2)  Significantly amount of experiment on quantizing 18 existing widely used CNN models for different applications, e.g., image classification, image segmentation, etc. 3) it reports the actually inference speed up comparing INT 32 to INT 8, although most of the speed up is less than 2.0.
Several questions:
1) What is the difference between retrain and calibration? As mentioned in the paper, the system does not require retrain, but it seems to me that calibration step, e.g., sampling to find the maximum values, etc, is a form of retrain. I think if that is the case, maybe some quantized models with retrain is worthy comparing with.
2) Many quantization techniques are used in the system, are there any conclusions for what techniques are most important for a particular CNN network/application?
3) Are there any new/novel quantization algorithms in the system?
4) The inference speed up is mostly less than 2 times, with some are achieving 2.1 speed up while some are without any speedup. Any reason for that? Also what is the overhead of the inference time?