In Variational Auto-Encoder (VAE), the default choice of reconstruction loss function between the decoded sample and the input is the squared L 2 . We propose to replace it with the log hyperbolic cosine (log-cosh) loss, which behaves as L 2 at small values and as L 1 at large values, and differentiable everywhere. Compared with L 2 , the log-cosh loss improves the reconstruction without damaging the latent space optimization, thus automatically keeping a balance between the reconstruction and the generation. Extensive experiments on MNIST and CelebA datasets show that the log-cosh reconstruction loss significantly improves the performance of VAE and its variants in output quality, measured by sharpness and FID score. In addition, the gradient of the log-cosh is a simple tanh function, which makes the implementation of gradient descent as simple as adding one sentence in coding.

The paper proposes a new loss function which can be used in the reconstruction term of various auto-encoder architectures. The pixel-wise cost function \ell(X, X') = f(X - X'; a) is defined for pairs of two input images X and X' and has one positive real-valued hyperparameter a. For small values of t the function f(t; a) behaves like a quadratic function, while for large t it behaves like |t|. As a consequence, it is smooth, everywhere differentiable (like L2) while not penalizing outliers too hard (like L1). The authors present several experiments conducted on MNIST and Celeba datasets, demonstrating that a simple change of a conventional pixel-wise squared L2 distance with the proposed log-cosh cost function improves the FID scores of generated samples as well as the visual quality of reconstructions (including "the sharpness").
I would say this is clearly an empirical study (even though the authors claim they provide "theoretical justifications", they are rather hand wavy), which is not a bad thing in this case. The message of the paper is very clear and I think the authors did a good job in selling their point. The main (and, perhaps, the only) contribution is the proposal to use the log-cosh function as the reconstruction cost. And this proposal is well justified by the set of experiments.
However, there are several major issues:
(1.1) The objective functions reported in appendix A.1 corresponding to WAE have in fact nothing to do with WAE. In WAE the regularizer penalizes the divergence between the prior distribution p(a) and *the aggregated posterior* distribution \int_x q(z|x) p(x) dx. In other words, D_MMD(q(z|x) || p(z)) in Eq. 8 should be replaced with D_MMD(\int_x q(z|x) p(x) dx || p(z)) in order to result in the WAE model. In summary, if the authors indeed used objectives reported in Eq. 8 of Appendix A, they were actually not using WAE but rather some other sort of regularized auto-encoders, which in a way are quite similar to VAEs.
(1.2)  I am surprised to see the reported FID scores for the Celeba data set. Having worked with this data set myself in combination with VAEs and WAEs, I am impressed with the extremely low FID scores: 46 for the vanilla L2 VAE and 30 for the L2 WAE. Note that while in the appendix the authors say they follow the architectural choices provided in [1] while performing the "L2 WAE Celeba" experiment, the authors arrive at FID=30 compared to FID=55 reported in the "Wasserstein Autoencoders" paper. Also, based on my experience, achieving FID=46 on CelebA with a vanilla VAE is very impressive. Note that the authors use 10^4 of samples to evaluate the FID scores, which is exactly the same as in [1]. This size is known to be large enough to reduce the variance of FID, so the difference (55 - 30) can not be explained by the fluctuations of FID. Therefore, I ask the authors to (anonymously) share the code and/or checkpoints of the 2 particular trained models: L2 VAE and L2 WAE trained on Celeba.
Other comments:
(2.1) Note that the reconstruction cost function in VAE should be normalized for every value of the code Z, as it corresponds to the logarithm of the likelihood (density) function -log p(X|Z). L2 and L1 costs both correspond to the well known likelihood (decoder) models (Gaussian and Laplace). However, it is hard to say what decoder model (what type of conditional distribution p(X|Z) ) would give rise to the proposed log-cosh function. In particular, the normalizing constant is not known and may depend on Z. In other words, by exchanging the L2 cost with the log-cosh loss in the VAE one looses the theoretical guarantees supporting VAE, including the fact that the objective is the lower bound on the marginal log likelihood. While this is not necessarily a problem (unless one uses the value of the objective as the bound on the marginal log likelihood, which is not the case in this paper), I would suggest mentioning it. Notice that, for instance, in WAE this problem does not appear, as the reconstruction term there does not involve any likelihood functions and thus does not need to be normalized.
(2.2) In Figure 2 I don't see why the authors did not highlight bad samples in the second row corresponding to their proposed method? I see many badly looking images there. Say, (4, 9) in VAE (MLP) and (8, 9) in VAE (Conv) and (6, 1) in WAE (MLP) and (2, 10) in WAE (Conv), where (i, j) means i-th row, j-th column, indexing starting from 1.
(2.3) How would the Huber loss perform and how does it compare to the proposed loss?
[1] Wasserstein Autoencoders. Tolstikhin et al., ICLR, 2018.


UNDERSTANDING STRAIGHT-THROUGH ESTIMATOR IN TRAINING ACTIVATION QUANTIZED NEURAL NETS
Training activation quantized neural networks involves minimizing a piecewise constant function whose gradient vanishes almost everywhere, which is undesirable for the standard back-propagation or chain rule. An empirical way around this issue is to use a straight-through estimator (STE) (Bengio et al., 2013)  in the backward pass only, so that the "gradient" through the modified chain rule becomes non-trivial. Since this unusual "gradient" is certainly not the gradient of loss function, the following question arises: why searching in its negative direction minimizes the training loss? In this paper, we provide the theoretical justification of the concept of STE by answering this question. We consider the problem of learning a two-linear-layer network with binarized ReLU activation and Gaussian input data. We shall refer to the unusual "gradient" given by the STE-modifed chain rule as coarse gradient. The choice of STE is not unique. We prove that if the STE is properly chosen, the expected coarse gradient correlates positively with the population gradient (not available for the training), and its negation is a descent direction for minimizing the population loss. We further show the associated coarse gradient descent algorithm converges to a critical point of the population loss minimization problem. Moreover, we show that a poor choice of STE leads to instability of the training algorithm near certain local minima, which is verified with CIFAR-10 experiments.

Summary:
The paper presents an analysis of training single-layer hard-threshold (binary activation) networks for regression with a mean-squared loss function using two different straight-through estimators: the original identity-function STE and a ReLU-based STE. The paper demonstrates that training with the latter against the population loss is guaranteed to converge to a critical point, whereas using the former can cause instability in the training.
    Pros:
        - Interesting analysis that provides a novel method for determining which gradient estimators are effective for training single-layer binarized networks and which are not.
        - The paper is fairly clear, despite being quite technical; however, I did find myself jumping around a lot to refer back to previous results or definitions so the ordering and layout could definitely be improved.
    Cons:
        - Related work is missing and some claims in the paper are wrong as a result.
        - A single-layer binarized network is essentially just a perceptron, which we know how to learn already, so it’s not clear how this analysis will benefit analysis of multi-layer binarized networks (however, since it seems like a novel analysis approach, it’s possible that it can be extended). This connection is not made in the paper.
        - The paper does not analyze the most common and successful straight-through estimator: the saturated straight-through estimator, which uses the derivative of the hard_tanh activation (e.g., see [2]) and is a shifted and scaled version of the clipped ReLU STE.
Overall, I like the paper but it has too many issues currently for me to give it a high score. However, if my questions and comments are addressed sufficiently, I would be happy to improve my score.
Detailed questions and comments:
1.	The claim that “we make the first theoretical justification for the concept of STE” is wrong and should be significantly toned down and clarified. Bengio et al. (2013), which this paper cites, provides some theoretical justification already, as do papers on target propagation, such as [1] and [2]. These, as well as additional papers cited in [2] are quite relevant and should also be cited and discussed.
2.	The claim that “it is not the gradient of any function” is also wrong. Each STE is the gradient of a particular function, but is not the gradient of the function used in the forward pass. Please clarify.
3.	The single-layer binarized network architecture studied in this paper can equivalently be framed as a linear function of a collection of single-layer perceptrons with shared weights. Obviously, much work has been done on analyzing the perceptron architecture. Why is none of it discussed in this paper? How does that work relate to the work done in this paper? How does the convolutional layer used here change the results of that related work?
4.	(a) Is there an intuition for why the derivative of the ReLU performs better (i.e., converges) better than the identity? Why does clipping the bottom make it work better? I do not see this explained in the text anywhere and it would be helpful to include this.
(b) Further, depending on the reasoning given, it seems that clipping the top may also be useful (as in the clipped ReLU, which is a shifted and scaled version of the saturated STE discussed in Hubara et al. and [2]). Does your analysis extend to this STE? This would be very useful, as the SSTE/clipped ReLU is the most commonly used STE and the most empirically successful (as validated by your own experiments, as well as in previous work on training binary networks). Also, the SSTE/clipped ReLU is an even better approximation of the step function.
(c) Cai et al. (2017) is not the first use of the clipped ReLU activation function, since it is equivalent to the SSTE when using sign(x) \in {-1, +1} instead of your activation function (\sigma(x) \in {0, 1}) (i.e., you can shift and scale everything to get equivalent results).
5.	In section 3.1, you mention that when using the derivative of the ReLU for the STE then \mu`(x) = \sigma(x). Is this just a coincidence or does this fact help with convergence?
6.	Why did you choose to train your networks initialized with the weights from their full-precision counterparts? When you train using different initializations, does this significantly affect your results?
7.	The improved empirical performance of the clipped ReLU / SSTE is unsurprising but why does the vanilla ReLU STE perform so poorly on CIFAR-10 with ResNet-20 with 2 bit quantization?
8.	In the end, it’s not clear that training single-layer hard-threshold networks is particularly important. Instead, the goal of quantization, etc. is to train multi-layer hard-threshold networks. Can this analysis be extended to such networks? Does it say anything about training such networks currently?
9.	The acknowledgments section is just the text from the style file.
10.	The capitalization is wrong in a number of places in your references.
[1] Difference Target Propagation. Lee, Zhang, Fischer, and Bengio. ECML/PKDD (2015).
[2] Deep Learning as a Mixed Convex-Combinatorial Optimization Problem. Friesen and Domingos. ICLR (2018).
------------------------
After reading the author response, they have sufficiently addressed my main concerns. I think that this is a good paper that will be of interest to those concerned with understanding the training of activation-quantized / hard-threshold neural networks. I have thus increased my score.


MULTILINGUAL NEURAL MACHINE TRANSLATION WITH SOFT DECOUPLED ENCODING
Multilingual training of neural machine translation (NMT) systems has led to impressive accuracy improvements on low-resource languages. However, there are still significant challenges in efficiently learning word representations in the face of paucity of data. In this paper, we propose Soft Decoupled Encoding (SDE), a multilingual lexicon encoding framework specifically designed to share lexicallevel information intelligently without requiring heuristic preprocessing such as pre-segmenting the data. SDE represents a word by its spelling through a character encoding, and its semantic meaning through a latent embedding space shared by all languages. Experiments on a standard dataset of four low-resource languages show consistent improvements over strong multilingual NMT baselines, with gains of up to 2 BLEU on one of the tested languages, achieving the new state-of-the-art on all four language pairs 1 .

This paper focuses on the problem of word representations in multilingual NMT system. The idea of multilingual NMT is to share data among multiple language pairs. Crucially this requires some way to tie the parameters of words from different languages, and one popular method is to share subword units among languages. The problem is that subword units in different languages may not be semantically equivalent, and many semantically-equivalent concepts are not represented by the same subwords. This paper proposes an alternative way to share word representation, in particular by proposing a common set of "semantic" concept vectors across languages which are then folded into the word representations via attention.
The problem is well-motivated and the proposed solution is reasonable. Previous works such as (Gu et. al. 2018) have been motivated in a similar fashion, and the proposed solution seems to outperform it on the TED dataset of Qi et. al. 2018.
The experiments are informative. The main open questions I have are:
(a) Varying the latent embedding size. It seems like only 10,000 is tried. Since this is the main contribution of the work, it will be desirable to see results for different sizes. Is the method sensitive to this hyperparameter? Also suggestions on how to pick the right number based on vocabulary size, sentence size, or other language/corpus characteristics will be helpful.
(b) What do the latent embeddings look like? Intuitively will they be very different from those from Gu et. al. 2018 because you are using words rather than subwords as the lexical unit?
(c) The explanation for why your model outperforms Gu et. al. 2018 seems insufficient -- it would be helpful to provide more empirical evidence in the ablation studies in order really understand why your method, which is similar to some extent, is so much better.
The paper is generally clear. Here are few suggestions for improvement:
- Table 1: Please explain lex unit, embedding, encoding in detail. For example, it is not clear what is joint-Lookup vs. pretrain-Lookup. It can be inferred if one knows the previous works, but to be self-contained, I would recommend moving this table and section to Related Works and explaining the differences more exactly.
- Sec 4.2: Explain the motivation for examining the three different lexical units.
- Table 3: "Model = Lookup (ours)" was confusing. Do you mean "our implementation of Neubig & Hu 2018? Or ours=SDE? I think the former?
- Are the word representions in Eq 4 defined for each word type or word token? In other words, for the same word "puppy" in two different sentences in the training data, do they have the same attention and thus the same e_SDE(w)? You do not have different attentions depending on the sentence, correct? I think so, but please clarify. (Actually, Figure 2 has a LSTM which implies a sentential context, so this was what caused the potential confusion).
- There are some inconsistencies in the terms: e.g. latent semantic embedding vs latent word embedding. Lexical embedding vs Character embedding. This makes it a bit harder to line up Sec 4.4 results with Sec 3.2 methods.
- Minor spelling mistakes. e.g. dependant -> dependent. Please double-check for others.


Modern applications from Autonomous Vehicles to Video Surveillance generate massive amounts of image data. In this work we propose a novel image outlier detection approach (IOD for short) that leverages the cutting-edge image classifier to discover outliers without using any labeled outlier. We observe that although intuitively the confidence that a convolutional neural network (CNN) has that an image belongs to a particular class could serve as outlierness measure to each image, directly applying this confidence to detect outlier does not work well. This is because CNN often has high confidence on an outlier image that does not belong to any target class due to its generalization ability that ensures the high accuracy in classification. To solve this issue, we propose a Deep Neural Forest-based approach that harmonizes the contradictory requirements of accurately classifying images and correctly detecting the outlier images. Our experiments using several benchmark image datasets including MNIST, CIFAR-10, CIFAR-100, and SVHN demonstrate the effectiveness of our IOD approach for outlier detection, capturing more than 90% of outliers generated by injecting one image dataset into another, while still preserving the classification accuracy of the multi-class classification problem.

Pros
----
[Originality/Clarity]
The manuscript presents a novel technique for outlier detection in a supervised learning setting where something is considered an outlier if it is not a member of any of the "known" classes in the supervised learning problem at hand. The proposed solution builds upon an existing technique (deep neural forests). The authors clearly explain the enhancements proposed and the manuscript is quite easy to follow.
[Clarity/Significance]
The enhancements proposed are empirically evaluated in a manner that clearly shows the impact of the proposed schemes over the existing technique. For the data sets considered, the proposed schemes have demonstrated significant improvements for this scoped version of outlier detection.
[Significance]
The proposed scheme for improving the performance of the ensemble of the neural decision trees could be of independent interest in the supervised learning setting.
Limitations
-----------
[Significance]
Based on my familiarity with the traditional literature on outlier detection in an unsupervised setting, it would be helpful for me to have some motivation for this problem of outlier detection in a supervised setting. For example, the authors mention that this outlier detection problem might allow us to identify images which are incorrectly labelled as one of the "known" classes even though the image is not a true member of any of the known classes, and might subsequently require (manual) inspection. However, if this technique would actually be used in such a scenario, the parameters of the empirical evaluation, such as a threshold for outliers that considers 5000 images as outliers, seem unreasonable. Usually number of outliers (intended for manual inspection) are fairly low. Empirical evaluations with a smaller number of outliers is more meaningful and representative of a real application in my opinion.
[Significance]
Another somewhat related question I have is the applicability of this proposed outlier detection scheme in the unsupervised scheme where there are no labels and no classification task in the first place. Is the proposed scheme narrowly scoped to the supervised setting?
[Comments on empirical evaluations]
- While the proposed schemes of novel inlier-ness score (weighted sum vs. max route), novel regularization scheme and ensemble of less correlated neural decision trees are extremely interesting and do show great improvements over the considered existing schemes, it is not clear to me why the use of something like Isolation Forest (or other more traditional unsupervised outlier detection schemes such as nearest/farthest neighbour based) on the learned representations just before the softmax is not sufficient. This way, the classification performance of the network remains the same and the outlier detection is performed on the learned features (since the learned features are assumed to be a better representation of the images than the raw image features). The current results do not completely convince me that the proposed involved scheme is absolutely necessary for the considered task of outlier detection in a supervised setting.
- [minor] Along these lines, considering existing simple baselines such as auto-encoder based outlier detection should be considered to demonstrate the true utility of the proposed scheme. Reconstruction error is a fairly useful notion of outlier-ness. I acknowledge that I have considered the authors' argument that auto-encoders were formulated for dimensionality reduction.
[Minor questions]
- In Equation 10, it is not clear to me why (x,y) \in \mathcal{T}. I thought \mathcal{T} is the set of trees and (x,y) was the sample-label pair.
- It would be good understand if this proposed scheme is limited to the multiclass classification problem or is it also applicable to the multilabel classification problem (where each sample can have multiple labels).


CAPSULE GRAPH NEURAL NETWORK
The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance. However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representations may not suffice to preserve the node/graph properties efficiently, resulting in sub-optimal graph embeddings. Inspired by the Capsule Neural Network (CapsNet) (Sabour et al., 2017), we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms. By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level. As a result, our model generates multiple embeddings for each graph to capture graph properties from different aspects. The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs. Our extensive evaluations with 10 graph-structured datasets demonstrate that CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven. It outperforms other SOTA techniques on several graph classification tasks, by virtue of the new instrument.

The authors provide an architecture that applies recent advances in the field of capsule networks in the graph neural network domain. First, hierarchical node level capsules are extracted using GCN layers. Second, after weighting each capsule by the output of a proposed attention module, graph level capsules are computed by performing a global dynamic routing. These graph level capsules are used for training a capsule classifier using a margin loss and a reconstruction loss.
The general architecture seems to be a reasonable application of the capsule principle in the graph domain, following the proof of concept MNIST architecture proposed by Sabour et al.
My main concern is that I have problems grasping the motivation behind using capsules in the given scenario. Besides an unprecise motivation in the introduction, there is no clear reason why the routing mechanism helps with solving the given tasks. Capsule networks capture pose covariances by applying a linear, trainable transformation to pose vectors and computing the agreement of the resulting votes. It is not clear to me how discrete information like graph connectivity can be encoded in a pose vector so that linear transformations are able to match different "connectivity poses".
Is there a more formal argument that explains why capsules should be able to capture more information about the input graph than other GCNNs?
Also, some design choices seem to be quite arbitrary. One example is using the last feature maps of the GCN as positions for coordinate addition. Is there a theoretical/intuitive motivation for this?
Results for the given experiments show improvement on some graphs. However, the authors proposed several concepts: a global pooling method using dynamic routing, an attention mechanism, a novel reconstruction loss, interpreting deep node embeddings as spatial positions. It is not clear to what extent the individual aspects of the method contribute to the gains. The qualitative capsule embedding analysis is interesting. However, this part needs a comparison to standard global graph embeddings to see if there is a significant difference.
In my opinion, the paper needs:
1) a clear experimental evaluation showing that capsules and the dynamic routing lead to improved results (i.e. by providing an ablation study to show which gains result from the attention-based global pooling mechanism, the reconstruction loss, the dynamic routing and from the coordinate addition), or
2) a more precise motivation for the use of dynamic routing to capture correlation between pose vectors in graphs in general (i.e. formal arguments why the method is stronger in capturing statistics or for what types of graphs it provides more discriminative power).
Overall, the paper does not convince me that capsules and dynamic routing provide advantages if used like the authors propose. Therefore, I tend to voting for rejecting the paper as long as points 1) and 2) are not addressed properly.
Minor remarks:
- There are quite a lot of grammatical errors (especially missing articles).
--------------------------
Update:
The authors addressed some of the weak points mentioned above adequately. The experimental evaluation was significantly improved and the results are a nice contribution. However, the theoretical contribution and the poor motivation of capsules in the graph context remain weak points. I have updated my rating accordingly.


LEARNING TO INFER AND EXECUTE 3D SHAPE PROGRAMS
Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higherlevel shape structure such as the repetition and reflective symmetry of object parts. In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships. In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes. Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction. After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner. Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories. It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.

This paper presents an approach to infer shape programs given 3D models. The programs include placing and arranging predefined primitives in layouts and can be written as a program over a domain-specific language (DSL).
The architecture consists of a recurrent network that encodes a 3D shape represented as a voxel grid and outputs the instructions using a LSTM decoder. The generation is two-step where the first step predicts a program ID and the second step predicts instructions within the program ID. This aspect wasn't completely clear to me, see questions below. A second module that renders the program to 3D is also implemented as a neural network in order to optimize the model parameter in a end-to-end manner by minimizing a reconstruction loss.
The method is evaluated on 3D shape reconstruction tasks for chairs and tables categories of the ShapeNet dataset. The approach compares favorably to Tulsiani et al., which considers a shape to be composed of a fixed number of cuboids.
The paper is well written and investigates an important problem. But it is hard to tease of the contributions and the relative importance of various steps in the paper:
1. Structure search vs. prediction. How does the model perform relative to a search-based approach for program generation. That would be slower but perhaps more accurate. The prediction model can be thought of an amortized inference procedure for search problems. What advantages does the approach offer?
2. Choice of the DSL. Compared to CSG modeling instructions of Sharma et al. the proposed DSL is more targeted to the shape categories. While this restricts the space of programs (e.g., no intersection, subtraction operations are used) leading to better generation of chairs and tables, it also limits the range and generalization of the learned models to new categories. Some discussion and comparison with the choice of DSL would be useful.
3. Is the neural render necessary -- Wouldn't it be easier to simply use automatic differentiation to compute gradients of the rendering engine?
4. It is also not clear to me how having a differentiable renderer allows training in an end-to-end manner since the output space is discrete and variable length. In CSGNet (Sharma et al.) policy-gradient techniques were used to optimize the LSTM parameters. The details of the guided adaptation were unclear to me (Section 4.3).
5. Is the neural renderer reliable -- Is is not clear if the neural renderer can provide accurate gradients when the generated programs are incorrect since the model is trained on a clean samples. In practice this means that the encoder has to initialized well. Since the renderer is also learned, would it generalize to new programs within the same DSL but different distribution over primitives -- e.g., a set of tables that have many more legs. Some visualizations of the generated shapes from execution traces could be added, sampling programs from within and outside the program distributions used to train.
6. All the above points give an impression that the choice of DSL and careful initialization are important to get the model to work. Some discussion on how robust the model is to these choices would be useful. In other words how meaningful is the generalization from the supervised training set of templates chairs and tables?
7. Missing baselines: The model is trained on 100,000 chairs and tables with full supervision. What is the performance of a nearest neighbor prediction algorithm? This is an important baseline that is missing. A comparison with a simplified CSGNet with shape primitives and union operations is also important. Tulsiani et al. consider unions but constrain that all instances have the same number of primitives which can lead to poor reconstruction results. Furthermore the training sets are likely different making evaluations unclear. I suggest training the following decoders on the same training set used in this approach (1) fixed set of cuboids (e.g., Tulsiani et al.), (2) A recurrent decoder with cuboids, (3) CSGNet (different primitives and operations), (4) a nearest neighbor predictor with the Hamming or Chamfer distance metric.


In this paper we show strategies to easily identify fake samples generated with the Generative Adversarial Network framework. One strategy is based on the statistical analysis and comparison of raw pixel values and features extracted from them. The other strategy learns formal specifications from the real data and shows that fake samples violate the specifications of the real data. We show that fake samples produced with GANs have a universal signature that can be used to identify fake samples. We provide results on MNIST, CIFAR10, music and speech data.

The primary purpose of this paper, from what I understand, is to show that fake samples created with common generative adversarial network (GAN) implementations are easily identified using various statistical techniques. This can potentially be useful in helping to identify artificial samples in the real world.
I think that the reviewers did an excellent job of probing into different statistical perspectives, such as looking at the continuity of the distribution of pixel intensities and the various higher moments of spectral data. I also must applaud the fact that they did not relegate themselves to image data, but branched out to speech and music data as well.
One of the first findings is that, with MNIST and CIFAR, the pixel intensities of fake samples are noticeably different when viewed from the perspective of a Kolgomorov-Smirnov Test or Jensen-Shannon Divergence comparison. This is an interesting observation, but less so than it would be if compared to something such as a variational autoencoder (VAE), which fits a KL distribution explicitly. IWGAN and LSGAN are using different metrics in their loss functions (such as Wassertein and least squares), and thus the result is not surprising or novel. I think if the authors had somehow shown how they worked their metrics into IWGAN or LSGAN to achieve better results, this could have been interesting.
Another observation the authors make is about the smoothness of the GAN distributions. This may not be so easily wrapped into the loss function, but it seems easily remedied as a post-processing step, or perhaps even a smoothing layer in the network itself. Nevertheless, this is an observation that I have not seen discussed in the literature so there is merit to at least noting the difference. It is confusing that on page 4, the authors state that they hypothesized that the smoothness was due to the pixel values themselves, and chose to alter the distribution of the original pixels in [0,1]. However, they state that in Figure 5, the smoothness remained "as expected." Did the authors misspeak here?
I found the music and speech experiments very interesting. The authors note that the synthetic Bach chorales, for instance, introduce many transitions that are not seen in the training and testing set of real Bach chorales. This, again, is interesting to note, but not surprising as the authors are judging the synthetic chorales on criteria for which they were not explicitly optimized. I do not believe these observations to be paper-worthy by themselves. However, the authors I believe have a good start on creating papers in which they specifically address these issues, showing how they can create better synthetic samples by incorporating their observations.
As to the writing style, there are many places where the writing is not quite clear. I would suggest getting an additional party to help proofread to avoid grammatical mistakes. I do not believe that the mistakes are so egregious as to impede understanding. However, it could distract from the importance on the authors future innovations if not corrected.
One last note. The title of the paper is "TequilaGAN: How to Easily Identify GAN Samples." This makes it seem as if the authors were introducing another type of GAN, like LSGAN or DCGAN. However, they are not. As a matter of fact, nowhere else in the paper is the word "TequilaGAN" mentioned. This title seems a bit sensational and misleading.
In the end, although I did find this paper to be an interesting read, I cannot recommend it for publication in ICLR.
----
Edit - November 29, 2018: Increasing my rating from a 4 to a 5 after discussion with the authors. Though their insights are not unknown, I think the authors are right in the fact that this is not explicitly discussed, at least not in the peer-review research with which I am familiar. But I don't think this by itself merits an ICLR publication.


This paper presents a novel deep learning architecture for classifying structured objects in ultrafine-grained datasets, where classes may not be clearly distinguishable by their appearance but rather by their context. We model sequences of images as linear-chain CRFs, and jointly learn the parameters from both localvisual features and neighboring class information. The visual features are learned by convolutional layers, whereas class-structure information is reparametrized by factorizing the CRF pairwise potential matrix. This forms a context-based semantic similarity space, learned alongside the visual similarities, and dramatically increases the learning capacity of contextual information. This new parametrization, however, forms a highly nonlinear objective function which is challenging to optimize. To overcome this, we develop a novel surrogate likelihood which allows for a local likelihood approximation of the original CRF with integrated batch-normalization. This model overcomes the difficulties of existing CRF methods to learn the contextual relationships thoroughly when there is a large number of classes and the data is sparse. The performance of the proposed method is illustrated on a huge dataset that contains images of retail-store product displays, and shows significantly improved results compared to linear CRF parametrization, unnormalized likelihood optimization, and RNN modeling.

Summary:
This paper introduces a new dataset consisting of images of various objects placed on store shelves that are labeled with object boundaries and what are described as “ultrafine-grained” class labels. The accompanying task is to predict the labels of each object given the individual images as well as their spatial layout relative to each other. To solve this task, a deep structured model is used consisting of CNN features for each image which are fed into a linear-chain CRF. To better deal with the large number of classes, pairwise potentials are represented as the multiplication of two lower-rank matrices which represent a sort of “class embedding” for each potential label. Training efficiency is improved by considering an objective based on a form of piecewise pseudolikelihood, which allows for training-time inference to be conducted with linear complexity relative to the number of labels. This objective also allows for easy use of batch normalization for the input features to the CRF model. This model/training procedure are compared against a number of models/training procedures to demonstrate its utility.
Comments:
Arguably, the primary contribution of this paper is the introduction of a new “ultrafine-grained” classification dataset which additionally allows for context to be utilized during prediction. This an interesting task, and it’s clear where being able to make such classifications is useful. The task is somewhat limited in scope, however. It’s unclear to me how models developed for this specific task would contain insights or be useful for other tasks - the utility of any models developed for this task seem limited to this exact task. If you have any other examples where inputs might be structured in this way, this would be good to add to the paper.
The model introduced is interesting, but its novelty is limited. It’s mostly a synthesis of ideas from previous work - CNN-based features, using a CRF to model correlations among labels, and approximating the full likelihood with pseudolikelihood. The interesting additions to these ideas are the fact that an “embedding” is learned for each class and that using the pseudolikelihood during training allows for batch norm to be applied in an easy way. Neither of these is a ground-breaking insight, but they are interesting nonetheless. I am somewhat surprised that the use of batch norm during training but not during testing did not hurt performance - a discussion of why this is the case would be good to have. For the most part, I think the experimentation is sufficiently rigorous - comparisons are made against a variety of baselines, and the new model trained with the specified training procedure outperforms the other alternatives. The one additional comparison I would have liked to see would have been against a model that pairwise potentials from the input features using a neural network-based model (for example, the one used in [1] - this seems like a rather glaring omission.
Other Comments:
-Since you ran a cross-validation, you should add confidence intervals to your reported numbers
-One additional dataset detail I was hoping to see that you didn’t provide is the mean/standard deviation of the number of instances per class,
-Your appendix contains a number of interesting ablation studies - you really should report the numbers for these as well
-The title of your paper is somewhat misleading - it’s hard to argue that the form of class embedding you use is a “deep” class embedding since it’s just a matrix of parameters that are learned during training.
Overall, I’m not convinced the model/training procedure by themselves would be fully worthy of publication, but the fact that a new dataset is introduced with a challenging variant of standard classification tasks adds merit to this work.
[1] Ma, Xuezhe, and Eduard Hovy. "End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF."
REVISION:
The other reviewers raised some concerns that I had overlooked (especially regarding novelty of using matrix factorization to generate your potentials). Given these, I do not think that this paper is in a state where it is ready to be accepted. Proper citations and analysis of your approach will be needed first.


ENVIRONMENT PROBING INTERACTION POLICIES
A key challenge in reinforcement learning (RL) is environment generalization: a policy trained to solve a task in one environment often fails to solve the same task in a slightly different test environment. A common approach to improve interenvironment transfer is to learn policies that are invariant to the distribution of testing environments. However, we argue that instead of being invariant, the policy should identify the specific nuances of an environment and exploit them to achieve better performance. In this work, we propose the "Environment-Probing" Interaction (EPI) policy, a policy that probes a new environment to extract an implicit understanding of that environment's behavior. Once this environmentspecific information is obtained, it is used as an additional input to a task-specific policy that can now perform environment-conditioned actions to solve a task. To learn these EPI-policies, we present a reward function based on transition predictability. Specifically, a higher reward is given if the trajectory generated by the EPI-policy can be used to better predict transitions. We experimentally show that EPI-conditioned task-specific policies significantly outperform commonly used policy generalization methods on novel testing environments.

The submission presents a reinforcement learning method for exploring/probing the environment to determine an environment’s properties and exploit these during later tasks. The method relies on jointly learning an embedding that simplifies prediction of future states and a policy that maximises a curiosity/intrinsic motivation like reward to learn to explore areas where the prediction model underperforms. In particular, the reward is based on the difference between prediction based on the learned embedding and prediction based on a prior collected dataset, such that the reward optimises to collect data with a large difference between the prediction accuracy of both models. The subsequently frozen policy and embedding are then used in other domains in a system identification like manner with the embedding utilised as input for a final task policy. The method is evaluated on a striker and hopper environment with varying dynamics parameters and shown to outperform a broad set of baselines.
In particular the broad set of baselines and small performed ablation study on the proposed method are quite interesting and beneficial for understanding the approach. However, the ablation study could be in more detail with respect to the additional training variations (Section 4.1.3; e.g. without all training tricks). Additionally, information about the baselines should be extended in the appendix as e.g. different capacities alone could have an impact where the performances of diff. algorithms are comparably similar. In particular, additional information about the training procedure for the UP-OSI (Yu et al 2017) baseline is required as the original approach relies on iterative training and it is unclear if the baseline implementation follows the original implementation (similar to Section 4.1.3.).
Overall the submission provides an interesting new direction on learning system identification approaches, that while quite similar to existing work (Yu et al 2017), provides increased performance on two benchmark tasks. The contribution of the paper focuses on detailed evaluation and, overall, beneficial details of the proposed method. The novelty of the submission is however limited and highly similar to current methods.
Minor issues:
- Related work on learning system identification:
Learning to Perform Physics Experiments via Deep Reinforcement Learning
Misha Denil, Pulkit Agrawal, Tejas D Kulkarni, Tom Erez, Peter Battaglia, Nando de Freitas


This paper proposes a sparse sequence clustering algorithm based on hierarchical Bayesian Modeling. This algorithm can cluster very sparse non-negative integer sequences of unequal length which do not necessarily belong to well-defined classes. Such Sequences are generated from the vast majority of normal human actions, for example, user behavior data for Wikipedia contributors expressed as a count of updates per day. Thus this algorithm and modeling technique is very useful for modeling non-negative integer sequences generated by real-life human actions, such as user-visits to websites or shops. This data model is a mixture model where every sequence is generated by a mixture of distributions associated to several clusters but does not need the data to be represented by a Gaussian mixture, which is the most commonly used representation of sequences and that gives significant modeling freedom. This algorithm also generates a very interpretable profile for the discovered Clusters,i.e, the latent groups of users who visited a food joint over a two year period. The Cluster profile, in this case, would contain the representative visit-counts of the cluster or group. The data that was used for the work has been contributed by a digital marketing company. The data is a collection of sparse sequences of a number of user visits to a food joint, where each entry of a user's sequence is the number of visits of that user per week to a given restaurant.

This paper propose a hierarchical Bayesian model to cluster sparse sequences data. The observations are modeled as Poisson distributions, whose rate parameter \lambda_i is written as the summation of \lambda_{ik}, a Gamma distribution with rate equal to the mixture proportion \alpha_{ik}. The model is implemented in Pystan. Experimental results on a real-world user visit dataset were presented.
The format of this paper, including the listing in the introduction section, the long url in section 2.3, and the model specification in section 3.2, can be improved. In particular, the presentation of the model would be more clear if the graphical model can be specified.
The motivation of choosing the observation model and priors is not clear. In section 3, the author described the details of model specification without explaining why those design choices were appropriate for modeling sparse sequence data.
Experimental results on a real-world dataset is presented. However, to demonstrate how the model works, it would be best to add synthetic experiments as sanity check. Results using common baseline approaches should also be presented. The results should also be properly quantified in order to compare the relative advantage of different approaches.


Encoding images as a series of high-level constructs, such as brush strokes or discrete shapes, can often be key to both human and machine understanding. In many cases, however, data is only available in pixel form. We present a method for generating images directly in a high-level domain (e.g. brush strokes), without the need for real pairwise data. Specifically, we train a "canvas" network to imitate the mapping of high-level constructs to pixels, followed by a high-level "drawing" network which is optimized through this mapping towards solving a desired image recreation or translation task. We successfully discover sequential vector representations of symbols, large sketches, and 3D objects, utilizing only pixel data. We display applications of our method in image segmentation, and present several ablation studies comparing various configurations.

This paper presents an unsupervised method for generating images in a high-level domain (brush strokes and geometric primitives). The proposed system is comprised of two neural networks: the drawer D and a forward model C of an external renderer R. The latter is trained on the rollouts produced by sending random actions to R. The forward model is then freezed and used to train D, i.e., the network that repeatedly interacts (sends commands) with the C to produce a desired image. Since everything is differentiable, D can be optimized via regular gradient descent.
Pros:
+ The paper is written clearly and relatively easy to read
+ The idea to replace the non-differentiable renderer with a differentiable approximation makes sense. Pure RL setups (i.e., in [Ganin et al., 2018]) are quite sample inefficient and hard to train due to high variance of REINFORCE.
+ The proposed method is tested both in 2D (drawings and floor plans) and 3D (prisms) domains and yields relatively good results.
Cons:
- The datasets used in the paper are quite simplistic. I’m wondering how hard it is to train a forward model for more complex data. At the very least, I would want to see how the method handles the 3D experiment when the view is not axis-aligned and there is more variety in the number of primitives and their types.
- The performance of the method especially on drawings and floor plans is not excellent. The drawer tends to reproduce line art with small disjoint strokes - very different from how humans would accomplish this task. A similar observation can be made for floor plans (the system outputs small pixel-like boxes that tile bigger rooms). This suggests that recovered commands do not really correspond to the higher-level structure of the input. Unlike in RL approaches, injection of prior knowledge about the data (e.g., the floor plan should be reproduced using the minimum possible number of rectangles) seems problematic within the proposed framework.
- It’s unclear how to use the approach for non-continuous actions (e.g., choosing types of primitives in 3D or different instruments in music).
- It seems the method may suffer from significant exploration problems in more complex settings. Consider an image of a rectangle that the system should reproduce. Say, it initially outputs a box that doesn’t intersect with the target one. The gradient of l^2-distance between those two images in the pixel space is non-zero but it is zero w.r.t. the action taken since no small change of the action parameters would make the generated box intersect with the target (assuming that the target is far from the model’s output) so l^2 will stay the same.
- I would love to see some comparison (preferably quantitative - speed of training, quality of reconstructions, etc.) to an RL system. So far, in the paper, there is only one figure showing a couple of images produced by such system.
Notes/questions:
* Section 2, paragraph 2: The systems by Xie et al. and Ganin et al. are very distinct. The former models the appearance of a single stroke while the latter is more similar to the present paper and synthesizes the entire image using strokes of a predefined appearance.
* Section 3.3, paragraph 3: “pixel-wise maximum” - it seems to be a fairly restrictive setup which only works when the model increases intensity of pixels.
* Section 3.4: This is a straightforward idea and is not novel (e.g., already used in some demonstrations of the method in [Ganin et al., 2018])
* Section 4.2, paragraph 2: During training, do you use all the patches or randomly sample them? Is your loss computed per patch or for the entire image?
In summary, the paper presents an interesting idea but the execution needs some improvement (especially, in terms of evaluation) before the paper is ready to be accepted to the conference.
After going through the authors' comments and the revised version of the paper, I keep the rating as is. The paper needs a more convincing evaluation section as well as some clean up (e.g., references to figures and tables in the text)


Our goal is to infer reward functions from demonstrations. In order to infer the correct reward function, we must account for the systematic ways in which the demonstrator is suboptimal. Prior work in inverse reinforcement learning can account for specific, known biases, but cannot handle demonstrators with unknown biases. In this work, we explore the idea of learning the demonstrator's planning algorithm (including their unknown biases), along with their reward function. What makes this challenging is that any demonstration could be explained either by positing a term in the reward function, or by positing a particular systematic bias. We explore what assumptions are sufficient for avoiding this impossibility result: either access to tasks with known rewards which enable estimating the planner separately, or that the demonstrator is sufficiently close to optimal that this can serve as a regularizer. In our exploration with synthetic models of human biases, we find that it is possible to adapt to different biases and perform better than assuming a fixed model of the demonstrator, such as Boltzmann rationality.

This paper has proposed algorithms for inferring reward functions from demonstrations with unknown biases. To achieve this, the authors have proposed to learn planners from demonstrations in multiple tasks via value iteration networks to learn the reward functions.
This paper has provided an excellent motivation of their work in Sections 1 & 2 with references being made to human behaviors and heuristics, though the authors can choose a more realistic running example that is less extreme than making orthogonal decisions/actions. The paper is well-written, up till Section 4.
On the flip side, there does not seem to be any significant technical challenges, perhaps due to some of the assumptions that they have made. Like the authors have mentioned, I do find assumption 3 to be overly strong and restrictive, as empirically demonstrated in Section 5.2. Arguably, is it really weaker than that of noisy rationality? At this moment, it is difficult to overlook this, even though the authors have argued that it may not be as restrictive in the future when more sophisticated differentiable planners are developed.
The experimental results are not as convincing as I would have liked. In particular, Algorithm 2 (learning a demonstrator's model) does not seem to outperform that of assuming an optimal demonstrator for the noiseless case and a Boltzmann demonstrator for the noisy case (Fig. 3). This was also highlighted by the authors as well: "The learning methods tend to perform on par with the best of two choices." It begs the question whether  accounting for unknown systematic bias can indeed outperform the assumption of a particular inaccurate bias when we know a priori whether the demonstrations are noisy or not.
Other detailed comments are provided below:
I would have preferred that the authors present their technical formulations in Section 4 using the demonstrator's trajectories instead of policies.
The authors say that "In some cases, like they naive and sophisticated hyperbolic discounters, especially the noisy ones, the learning methods outperform both optimal and Boltzmann assumptions." But, Fig. 3 shows that Algorithm 2 does not perform better than either that of the optimal or Boltzmann demonstrator.
In Section 5.2, the authors have empirically demonstrated the poor approximate planning performance of VIN, as compared to an exact model the demonstrator. What then would its implications be on the adaptivity of Algorithms 1 and 2 to biases?
The following references on IRL with noisy demonstration trajectories would be relevant:
Benjamin Burchfiel, Carlo Tomasi, and Ronald Parr. Distance Minimization for Reward Learning from Scored Trajectories. In Proc. AAAI, 2016.
J. Zheng, S. Liu, and L. M. Ni. Robust Bayesian inverse reinforcement learning with sparse behavior noise. In Proc. AAAI, 2014.
Minor issues:
On page 4, the expression D : W × R -> S -> A -> [0, 1] can be more easily understood with the use of parentheses.
For Assumption 2b, you can italicize "some".
In the first paragraph of section 4.1, what are you summing over?
Line 3 of Algorithm 1: PI_W?
Page 7: For the learning the bias setting?
Page 7: figure 3 shows?
Page 7: they naive?
Page 7: so as long as?
Page 8: adaption?
Page 8: predicated?
Page 8: figure 4 shows?


IMPROVING SEQUENCE-TO-SEQUENCE LEARNING VIA OPTIMAL TRANSPORT
Sequence-to-sequence models are commonly trained via maximum likelihood estimation (MLE). However, standard MLE training considers a word-level objective, predicting the next word given the previous ground-truth partial sentence. This procedure focuses on modeling local syntactic patterns, and may fail to capture long-range semantic structure. We present a novel solution to alleviate these issues. Our approach imposes global sequence-level guidance via new supervision based on optimal transport, enabling the overall characterization and preservation of semantic features. We further show that this method can be understood as a Wasserstein gradient flow trying to match our model to the ground truth sequence distribution. Extensive experiments are conducted to validate the utility of the proposed approach, showing consistent improvements over a wide variety of NLP tasks, including machine translation, abstractive text summarization, and image captioning.

====== Final Comments =======
I thank the authors for updating the manuscript with clarifications and for clear replies to my concerns.
I agree with R2 to some extent that the empirical performance of the method, as well as the formulation,  is interesting. In general, the authors addressed my concerns regarding how optimal transport training model interfaces with MLE training and the choice of using scaled softmax for computing the Wasserstein distances. However, I still find myself agreeing with R3 that the choice of sets to compute Wasserstein distances (as opposed to sequences is somewhat unjustified); and it is not clear how the theory in Sec. 3 justifies using sets instead of words, as the data distribution p_d is over sequences in both the W-2 term as well as the MLE term. This would be good to clarify further, or explain more clearly how Sec. 3 justifies this choice.
Also, I missed this in the original review, but the assumption that KL(\mu| p_d) = KL(\p_d| \mu) since \mu is close to p_d for MLE training does not seem like a sensible assumption under model misspecification (as MLE is mode covering). I would suggest the authors discuss this in a revision/ camera-ready version.
In light of these considerations, I am updating the rating to 6 (to reflect the points that have been addressed), but I still do not believe that the method is super-well justified, despite an interesting formulation and strong empirical results (which are aspects the paper could still improve upon).
=====================
**Summary**
The paper proposes a regularization / fine-tuning scheme in addition to maximum likelihood sequence training using optimal transport. The basic idea is to match a sampled sentence from a model with a ground truth sentence using optimal transport. Experimental results show that the proposed modification improves over MLE training across machine translation, summarization and image captioning domains.
**Strengths**
+ The proposed approach shows promising results across multiple domains.
+ The idea of using optimal transport to match semantics of words intuitively makes sense.
**Weaknesses**
1. Generally, one would think that for optimal transport we would use probabilities which come from the model, i.e. given a set of words and probabilities for each of the words, one would move mass around from one word to another word (present in the ground truth) if the words were semantically relevant to each other and vice versa. However, it seems the proposed algorithm / model does not get the marginal distributions for the words from the model, and indeed does not use any beliefs from the model in the optimal transport formulation / algorithm [Alg. 1, Line 2]. This essentially then means, following objective 2, that optimal transport has no effect on the model’s beliefs on the correct word, but merely needs to ensure that the cosine similarity between words which have a high transport mass is minimized, and has no direct relation to the belief of the model itself. This strikes as a somewhat odd design choice. (*)
2. In general, another issue with the implementation as described in the paper could be the choice of directly using the temperature scaled softmax as an estimate of the argmax from the model, instead of sampling utterances from the model. It seems worth reporting results, even if as a baseline what sampling from something like a concrete distribution [A] yeilds in terms of results. (*)
3. As a confirmation, Is the differentiable formulation for sentences also used for MLE training part of the objective, or does MLE still use ground truth sentences (one-hot encoded) as input. Does this mean that the model is forwarded twice (once with ground truth, and once with softmax outputs)? In general, instead of/ in addition to an Algorithm box that expains the approach of Xie et.al., it would be good to have a clear algorithmic explanation of the training procedure of the proposed model itself. Further, the paper is not clear if the model is always used for finetuning the an MLE model (as Section 3) seems to suggest or if the optimal transport loss is used in conjunction with the model as the bullet ``complementary MLE loss’’ seems to suggest. (*)
4. Section 3: It is not clear that the results / justifications hold for the proposed model since the distance optimized in the current paper is not a wasserstein-2 distance. Sure, it computes cosine distance, which is L-2 but it appears wasserstein-2 distance is defined for continuous probability measures, while the space on which the current paper computes distances is inherently the (discrete) space of word choices. (*)
Experiments
5. The SPICE metric for image captioning takes the content of the sentences (and semantics) into account, instead of checking for fluency. Prior work on using RL techniques for sequence predictions have used SPICE + CIDEr [B] to alleviate the problem with RL methods mentioned in page. 5. Would be good to weaken the claim. (*)
Minor Points
1. It might be good to be more clear about the objective from Xie et.al. by also stating the modified formulation they discuss, and explaining what choice of the function yeilds the bregman divergence of the form discussed in Eqn. 3.
2. It would be nice to be consistent with how \mathcal{L}_{ot} is used in the paper. Eqn. 4 lists it with two embedding matrices as inputs, while Alg. 1, Line 11 assigns it to be the inner product of two matrices.
**Preliminary Evaluation**
In general, the paper is an interesting take on improving MLE for sequence models. However, while the idea of using optimal transport is interesting and novel for training sequence models, I have questions about the particular way in which it has been implemented, which seems somewhat unjustified. I also have further clarifications about a claimed justification for the model. Given convincing responses for these, and other clarification questions (marked with a *) this would be a good paper.
References
[A]: Maddison, Chris J., Andriy Mnih, and Yee Whye Teh. 2016. “The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1611.00712.
[B]: Liu, Siqi, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2016. “Improved Image Captioning via Policy Gradient Optimization of SPIDEr.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1612.00370.


DEEP FRANK-WOLFE FOR NEURAL NETWORK OPTIMIZATION
Learning a deep neural network requires solving a challenging optimization problem: it is a high-dimensional, non-convex and non-smooth minimization problem with a large number of terms. The current practice in neural network optimization is to rely on the stochastic gradient descent (SGD) algorithm or its adaptive variants. However, SGD requires a hand-designed schedule for the learning rate. In addition, its adaptive variants tend to produce solutions that generalize less well on unseen data than SGD with a hand-designed schedule. We present an optimization method that offers empirically the best of both worlds: our algorithm yields good generalization performance while requiring only one hyper-parameter. Our approach is based on a composite proximal framework, which exploits the compositional nature of deep neural networks and can leverage powerful convex optimization algorithms by design. Specifically, we employ the Frank-Wolfe (FW) algorithm for SVM, which computes an optimal step-size in closed-form at each time-step. We further show that the descent direction is given by a simple backward pass in the network, yielding the same computational cost per iteration as SGD. We present experiments on the CIFAR and SNLI data sets, where we demonstrate the significant superiority of our method over Adam, Adagrad, as well as the recently proposed BPGrad and AMSGrad. Furthermore, we compare our algorithm to SGD with a hand-designed learning rate schedule, and show that it provides similar generalization while often converging faster. The code is publicly available at https://github.com/oval-group/dfw.

Dual Block-Coordinate Frank-Wolfe (Dual-BCFW) has been widely used in the literature of non-smooth and strongly-convex stochastic optimization problems, such as (structural) Support Vector Machine. To my knowledge, the submission is the first sound attempt to adapt this type of Dual-based algorithm for optimization of Deep Neural Network, which employs a proximal-point method that linearizes not the whole loss function but only the DNN (up to the logits) to form a convex subproblem and then deal with the loss part in the dual.
The attempt is not perfect (actually with a couple of issues detailed below), but the proposed approach is inspiring and I personally would love it published to encourage more development along this thread.  The following points out a couple of items that could probably help further improve the paper.
*FW vs BCFW*
The algorithm employed in the paper is actually not Frank-Wolfe (FW) but Block-Coordinate Frank-Wolfe (BCFW), as it minimizes w.r.t. a block of dual variables belonging to the min-batch of samples.
*Batch Size*
Though the algorithm can be easily extended to the min-batch case, the author should discuss more how the batch size is interpreted in this case (i.e. minimizing w.r.t. a larger block of dual variables belonging to the batch of samples) and the algorithmic block (Algorithm 1) should be presented in a way reflecting the batch size since this is the way people use an algorithm in practice (to improve the utilization rate of a GPU).
*Convex-Conjugate Loss*
The Dual FW algorithm does not need to be used along with the hinge loss (SVM loss). All convex loss function can derive a dual formulation based on its convex-conjugate. See [1,2] for examples. It would be more insightful to compare SGD vs dual-BCFW when both of them are optimizing the same loss functions (either hinge loss or cross-entropy loss) in the experimental comparison.
[1] Shalev-Shwartz, Shai, and Tong Zhang. "Stochastic dual coordinate ascent methods for regularized loss minimization." JMLR (2013)
[2] Tomioka, Ryota, Taiji Suzuki, and Masashi Sugiyama. "Super-linear convergence of dual augmented Lagrangian algorithm for sparsity regularized estimation." JMLR (2011).
*BCFW vs BCD*
Actually, (Lacoste-Julien, S. et al., 2013) proposes Dual-BCFW to optimize structural SVM because the problem contains exponentially many number of dual variables. For typical multiclass hinge loss problem the Dual Block-Coordinate Descent that minimizes w.r.t. all dual variables of a sample in a closed-form update converges faster without extra computational cost. See the details in, for example, [3, appendix for the multiclass hinge loss case].
[3] Fan, Rong-En, et al. "LIBLINEAR: A library for large linear classification." JMLR (2008).
*Hyper-Parameter*
The proposed dual-BCFW still contains a hyperparameter (eta) due to the need to introduce a convex subproblem, which makes its number of hyperparameters still the same to SGD.


h-DETACH: MODIFYING THE LSTM GRADIENT TO- WARDS BETTER OPTIMIZATION
Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps. We introduce a simple stochastic algorithm (h-detach) that is specific to LSTM optimization and targeted towards addressing this problem. Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed. Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them. Our algorithm 1 prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better. We show significant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modification of LSTM gradient on various benchmark datasets.

The results are intriguing. However, similar methods like BN-LSTM [3] and Variational RNNs [4] achieve arguably the same with very similar mechanisms. We do not think they can be considered as orthogonal. This should be addressed by the authors. Also, hard long-term experiments like sequentially predicting pixels (like through MDLSTM-based PixelRNN) or language modelling should be favoured over short sentence image captions.
It is possible that we will improve our ratings once our concerns are addressed.
Paper Summary:
The authors claim that the gradient along the computational path that goes through the cell state (the linear temporal path or A gradient) of an LSTM carries information about long-term dependencies. Those gradients can be corrupted by the gradient of all other computational paths (i.e. the B gradient). They claim that this makes it hard to learn long-term dependencies and has, therefore, significant negative effects on the convergence speed, training stability, and generalisation performance. They propose a method called h-detach and run experiments on the delayed copy task, sequential MNIST, permuted sequential MNIST (pMNIST), and caption generation on the MS COCO dataset. All show either somewhat improved performance or much more stable learning curves. At every step, h-detach randomly drops all gradients that flow through the h of the standard LSTM, the B gradients, and only keeps the ones from the linear temporal path, the A gradients. Experiments also suggest that the A gradients carry more long-term information than B gradients and that LSTMs with h-detach do not need gradient clipping for successful training.
Positive:
The paper is written clearly. It is well structured and well motivated. H-detach is simple, effective, and somewhat novel (see below). Experiments indicate that its main benefit is training stability as well as minor performance improvements.
Negative:
We are not sure how significant these results are for the following reasons:
- MS COCO image caption generation is the only more challenging dataset, but it seems a bit misplaced as it has very short sentences, while the authors motivate their work through a focus on long-term dependencies. Why not apply h-detach to a language model such as [1] with official online implementations, e.g., [2]. A setting with PixelRNN [6] based on MD-LSTM [7] would also be a great testbed for h-detach.
- The purpose of h-detach is to scale down the B gradients. However, methods which apply e.g. BatchNorm to the hidden state learn a scale parameter which could be learned by the network explicitly. For the backward pass, this has the effect of scaling down the B gradient. Consider e.g. [3] which also achieves similar training stability on sequential MNIST and pMNIST with little overhead.
- Another very related method is [4] which properly applies a random dropout mask over the recurrent inputs that is shared across timesteps of an RNN. We think that h-detach is essentially achieving the same in a similar way.
Problems with Introduction and Related Work Section:
- The vanishing gradient problem was first described by Hochreiter in 1991 [5] (not by Bengio in 1994).
- Intro mentions GRU as if it was separate from LSTM. Clarify that GRU is essentially a variant of vanilla LSTM with forget gates [8]. Since one gate is missing, GRU is less powerful than the original LSTM [9].
[1] Zaremba et al. "Recurrent neural network regularization." arXiv:1409.2329 (2014).
[2] https://www.tensorflow.org/tutorials/sequences/recurrent
[3] Cooijmans et al. "Recurrent batch normalization." arXiv:1603.09025 (2016).
[4] Gal et al. "A theoretically grounded application of dropout in recurrent neural networks." NIPS 2016.
[5] Hochreiter, Sepp. "Untersuchungen zu dynamischen neuronalen Netzen." Diploma thesis, TUM (1991)
[6] Oord et al. "Pixel recurrent neural networks." arXiv preprint arXiv:1601.06759 (2016).
[7] Graves et al. "Multi-Dimensional Recurrent Neural Networks" arXiv preprint arXiv:0705.2011 (2011).
[8] Gers et al. “Learning to Forget: Continual Prediction with LSTM.“ Neural Computation, 12(10):2451-2471, 2000.
[9] Weiss et al. On the Practical Computational Power of Finite Precision RNNs for Language Recognition. arXiv:1805.04908.
Comments after rebuttal:
The  paper has clearly improved.
It leaves a few questions open though. For example, it is surprising that h-detach doesn't work on language modelling since Dropout-LSTM and BN-LSTM clearly improve over vanilla LSTM in this case (if not every case). In the new version, the authors only reference it in one or two sentences but don't discuss this in detail.
When dropout is mentioned, one should also mention that dropout is a variant of the old stochastic delta rule:
Hanson, S. J. (1990). A Stochastic Version of the Delta Rule, PHYSICA D,42, 265-272.  See also arXiv:1808.03578
Nevertheless, we now think that this is a very interesting LSTM regularization paper that people who study this field should probably know. We are increasing the score by 2 points!


To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational heteroencoder (VHE) that decodes text via a deep probabilisitic topic model, the variational posterior of whose local latent variables is encoded from an image via a Weibull distribution based inference network. To further improve VHE and add an image generator, we propose VHE randomized generative adversarial net (VHE-GAN) that exploits the synergy between VHE and GAN through their shared latent space. After training with a hybrid stochastic-gradient MCMC/variational inference/stochastic gradient descent inference algorithm, VHEGAN can be used in a variety of settings, such as text generation/retrieval conditioning on an image, image generation/retrieval conditioning on a document/image, and generation of text-image pairs. The efficacy of VHEGAN is demonstrated quantitatively with experiments on both conventional and generalized ZSL tasks, and qualitatively on (conditional) image and/or text generation/retrieval.

Paper Summary: This paper studies the zero-shot learning problem with deep generative models. More specifically, it proposed a hybrid framework that combines VAEs (more precisely, the variational hetero-encoder or VHE) and GANs all together. The entire model is composed of an image encoder (Weibull upward-downward variational encoder), a text decoder (Poisson Gamma belief network), and an image generator (generative adversarial network). Once learned, the generative models can be directly used for zero-shot classification and various image generation applications. In the experiments, two benchmark datasets CUB and Oxford-Flowers are used.
==
Novelty/Significance:
Zero-shot learning is a challenging task and he main motivation of the paper (using generative model) is interesting. The text representation in the paper is simply bag-of-words which limits the application to some extent. In a broader context, image captioning using generative model seems quite relevant.
Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space, Wang et al. In NIPS 2017.
==
Quality:
Overall, reviewer feels this is a very interesting work. However, the results from the paper is quite mixed. It is not yet convincing whether the proposed approach is the state-of-the-art in zero-shot learning or text-to-image generation.
First, this paper demonstrates the power of generative models in text-to-image generation and other applications. However, reviewer feels that the zero-shot classification result is weak. In Table 1 and Table 2, it seems that GAZSL (Zhu et al. 2018) outperforms the proposed approach.
Q1: In Table 2, is it possible to report the top-5 accuracy on CUB-easy and top-1 accuracy on Oxford-Flower dataset? Otherwise, it is not very convincing that proposed approach is better than the state-of-the-art approach GAZSL.
Second, the text-to-image generation results look reasonably good. But the resolution and quality of generated images are far from state-of-the-art. One suggestion is to train the VHE model with an improved image generator.
StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, Zhang et al. In CVPR 2017.
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks, Xu et al. In CVPR 2018.
Also, reviewer would expect to see an improved image generator can lead to a better ZSL performance.
Typo: In the title: Zero-Short → Zero-Shot.


LEARNING WHAT AND WHERE TO ATTEND
Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs). Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels. Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition. We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps. Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization. As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers.

This paper proposes a new approach to use more informative signals (than only class labels), specifically, regions humans deem important on images, to improve deep convolutional neural networks. They collected a large dataset by implementing a game on clickme.ai and showed that using this information results in both i) improved classification accuracy and ii) more interpretable features.
I think this is good work and should be accepted. The main contribution is three fold: i) a publicly available dataset that many researchers can use, ii) a network module to incorporate this human information that might be inserted into many networks to improve performance, and iii) some insights on the effect of such human supervision and the relation between features that humans deem important to those that neural nets deem important.
Some suggestions on how to improve the paper:
1. I find Sections 3 & 4 hard to track - some missing details and notation issues. Several variables are introduced without detailing the proper dimensions, e.g., the global feature attention vector g (which is shown in the figure actually). The relation between U and u_k isn't clear. Also, it will help to put a one-sentence summary of what this module does at the beginning of Section 3, like the last half-sentence in the caption of Figure 3. I was quite lost until I see that. Some more intuition is needed, on W_expand and W_shrink; maybe moving some of the "neuroscience motivation" paragraph up into the main text will help. Bold letters are used to denote many different things - in  Section 4 as a set of layers, in other places a matrix/tensor, and even an operation (F).
2. Is there any explanation on why you add the regularization term to every layer in a network? This setup seems to make it easy to explain what happens in Figure 4. One interesting observation is that after your regularization, the GALA features with ClickMe maps exhibit minimal variation across layers (those shown). But without this supervision the features are highly different. What does this mean? Is this caused entirely by the regularization? Or there's something else going on, e.g., this is evidence suggesting that with proper supervision like human attention regions, one might be able to use a much shallower network to achieve the same performance as a very deep one?
3. Using a set of 10 images to compute the correlation between ClickMe and Clicktionary maps isn't ideal - this is even less than the number of categories among the images. I'm also not entirely convinced that "game outcomes from the first and second half are roughly equal" says much about humans not using a neural net-specific strategy, since you can't rule out the case that they learned to play the game very quickly (in the first 10 of the total 380 rounds).
4. Title - this paper sound more "human feedback" to me than "humans-in-the-loop", because the loop has only 1 iteration.  Because you are collecting feedback from humans but not yet giving anything back to them. Maybe change the title?


With interest in program synthesis and similarly flavored problems rapidly increasing, neural models optimized for tree-domain problems are of great value. In the sequence domain, transformers can learn relationships across arbitrary pairs of positions with less bias than recurrent models. Under the intuition that a similar property would be beneficial in the tree domain, we propose a method to extend transformers to tree-structured inputs and/or outputs. Our approach abstracts transformer's default sinusoidal positional encodings, allowing us to substitute in a novel custom positional encoding scheme that represents node positions within a tree. We evaluated our model in tree-to-tree program translation and sequenceto-tree semantic parsing settings, achieving superior performance over the vanilla transformer model on several tasks.

The paper describes an interesting idea for using Vashwani's transformer with tree-structured data, where nodes' positions in the tree are encoded using unique affine transformations. They test the idea in several program translation tasks, and find small-to-medium improvements in performance.
Overall the idea is promising, but the work isn't ready for publication. The implementation details weren't easy to follow, the experiments were narrow, and there are key citations missing. I would recommend trying some more diverse tasks, and putting this approach against other graph neural network techniques.
REVISED:
I've revised by review upwards by 1, though I still recommend rejection. The authors improved the scholarship by adding many more citations and related work. They also made the model details and implementation more clear.
The remaining problem I see is that the results are just not that compelling, and the experiments do not test any other graph neural network architectures.
Specifically, in Table 1 (synthetic experiments) the key result is that their tree-transformer outperforms seq-transformer on structured input. But seq-transformer is best on raw programs. I'm not sure what to make of this. But I wouldn't use tree-transformer in this problem. I'd use seq-transformer.
In Table 2 (CoffeeScript-JavaScript experiments), no seq-transformer results are presented. That seems... suspicious. Did the authors try those experiments? What were the results? I'd definitely like to see them, or an explanation of why they're not shown. This paper tests whether tree-transformers are better than seq-transformer and other seq/tree models, but this experiment's results do not address that fully. Of the 8 tasks tested, tree-transformer is best on 5/8 while tree2tree is best on 3/8.
In Table 3, there's definitely a moderate advantage to using tree-transformer over seq-transformer, but in 5/6 of the tasks tree-transformer is worse than other approaches. The authors write, "Transformer architectures in general, however, do not yet compete with state-of-the-art results.".
Finally, no other graph neural network/message-passing/graph attention architectures are tested (eg. Li et al 2016 was cited but not tested, and Gilmer et al 2017 and Veličković et al 2017 weren't cited or tested), but there's a reasonable chance they'd outperform the tree-transformer.
So overall the results are intriguing, and I believe there's something potentially valuable here. But I'm not sure there's sufficient reason presented in the paper to use tree-transformer over seq-transformer or other seq/tree models. Also, while the basic idea is nice, as I understand it is restricted to trees, so other graphical structures wouldn't be handled.