Generative Adversarial Networks (GANs) have shown remarkable success as a framework for training models to produce realistic-looking data. In this work, we propose a Recurrent GAN (RGAN) and Recurrent Conditional GAN (RCGAN) to produce realistic real-valued multi-dimensional time series, with an emphasis on their application to medical data. RGANs make use of recurrent neural networks (RNNs) in the generator and the discriminator. In the case of RCGANs, both of these RNNs are conditioned on auxiliary information. We demonstrate our models in a set of toy datasets, where we show visually and quantitatively (using sample likelihood and maximum mean discrepancy) that they can successfully generate realistic time-series. We also describe novel evaluation methods for GANs, where we generate a synthetic labelled training dataset, and evaluate on a real test set the performance of a model trained on the synthetic data, and vice-versa. We illustrate with these metrics that RCGANs can generate time-series data useful for supervised training, with only minor degradation in performance on real test data. This is demonstrated on digit classification from 'serialised' MNIST and by training an early warning system on a medical dataset of 17,000 patients from an intensive care unit. We further discuss and analyse the privacy concerns that may arise when using RCGANs to generate realistic synthetic medical time series data, and demonstrate results from differentially private training of the RCGAN.

The authors propose to use synthetic data generated by GANs as a replacement for personally identifiable data in training ML models for privacy-sensitive applications such as medicine. In particular it demonstrates adversarial training of a recurrent generator for an ICU monitoring multidimensional time series, proposes to evaluate such models by the performance (on real data) of supervised classifiers trained on the synthetic data ("TSTR"), and empirically analyzes the privacy implications of training and using such a model.
This paper touches on many interesting issues -- deep/recurrent models of time series, privacy-respecting ML, adaptation from simulated to real-world domains. But it is somewhat unfocused and does not seem make a clear contribution to any of these.
The recurrent GAN architecture does not appear particularly novel --- the authors note that similar architectures have been used for discrete tasks such language modeling (and fail to note work that uses convolutional or recurrent generators for video prediction, a more relevant continuous task, see e.g.  http://carlvondrick.com/tinyvideo/, or autoregressive approaches to deep models of time series, e.g. WaveNet https://arxiv.org/abs/1609.03499) and there is no obvious new architectural innovation.
I also find it difficult to assess whether the proposed model is actually generating reasonable time series. It may be true that "one plot showing synthetic ICU data would not provide enough information to evaluate its actual similarity to the real data" because it could not rule out that case that the model has captured the marginal distribution in each dimension but not joint structure. However producing marginal distributions that look reasonable is at least a *necessary* condition and without seeing those plots it is hard to rule out that the model may be producing highly unrealistic samples.
The basic privacy paradigm proposed seems to be:
1. train a GAN using private data
2. generate new synthetic data, assume this data does not leak private information
3. train a supervised classifier on the private data
so that the GAN training-sampling loop basically functions as an anonymization procedure. For this to pan out, we'd need to see that the GAN samples are a) useful for a range of supervised tasks, and b) do not leak private information. But the results  in Table 2 show that the TSTR results are quite a lot worse than real data in most cases, and it's not obvious that the small set of tasks evaluated are representative of all tasks people might care about. The attempts to demonstrate empirically that the GAN does not memorize training data aren't particularly convincing; this is an adversarial setting so the fact that a *particular* test doesn't reveal private data doesn't imply that a determined attacker wouldn't succeed. In this vein, the experiments with DP-SGD are more interesting, although a more direct comparison would be helpful (it is frustrating to flip back and forth between Tables 2 and 3 in an attempt to tease out relative performance) and and it is not clear how the settings (ε = 0.5 and δ ≤ 9.8 × 10−3) were selected or whether they provide a useful level of privacy. That said I agree this is an interesting avenue for future work.
Finally it's worth noting that discarding patients with missing data is unlikely to be innocuous for ICU applications; data are quite often not missing at random (e.g., a patient going into a seizure may dislocate a sensor). It appears that the analysis in this paper threw out more than 90% of the patients in their original dataset, which would present serious concerns in using the resulting synthetic data to represent the population at large. One could imagine coding missing data in various ways (e.g. asking the generator to produce a missingness pattern as well as a time series and allowing the discriminator to access only the masked time series, or explicitly building a latent variable model) and some sort of principled approach to missing data seems crucial for meaningful results on this application.


Deep learning models require extensive architecture design exploration and hyperparameter optimization to perform well on a given task. The exploration of the model design space is often made by a human expert, and optimized using a combination of grid search and search heuristics over a large space of possible choices. Neural Architecture Search (NAS) is a Reinforcement Learning approach that has been proposed to automate architecture design. NAS has been successfully applied to generate Neural Networks that rival the best human-designed architectures. However, NAS requires sampling, constructing, and training hundreds to thousands of models to achieve well-performing architectures. This procedure needs to be executed from scratch for each new task. The application of NAS to a wide set of tasks currently lacks a way to transfer generalizable knowledge across tasks. In this paper, we present the Multitask Neural Model Search (MNMS) controller. Our goal is to learn a generalizable framework that can condition model construction on successful model searches for previously seen tasks, thus significantly speeding up the search for new tasks. We demonstrate that MNMS can conduct an automated architecture search for multiple tasks simultaneously while still learning well-performing, specialized models for each task. We then show that pre-trained MNMS controllers can transfer learning to new tasks. By leveraging knowledge from previous searches, we find that pre-trained MNMS models start from a better location in the search space and reduce search time on unseen tasks, while still discovering models that outperform published human-designed models.

The paper proposes an extension of the Neural Architecture Search approach, in which a single RNN controller is trained with RL to select hyperparameters for child networks that must perform different tasks. The architecture includes the notion of a "task embedding", that helps the controller keeping track of similarity between tasks, to facilitate transfer across related tasks.
The paper is very well written, and based on a simple but interesting idea. It also deals with core issues in current machine learning.
On the negative side, there is just one experiment, and it is somewhat limited. In the experiment, the proposed model is trained on two very different tasks (English sentiment analysis and Spanish language detection), and then asked to generalize to another English sentiment analysis task and to a Spanish sentiment analysis task. The models converge faster to high accuracy in the proposed transfer learning setup than when trained one a single task with the same architecture search strategy. Moreover, the task embedding for the new English task is closer to that of the training English task, and the same for the training/test Spanish tasks.
My main concern with the experiment is that the approach is only tested in a setup in which there is a huge difference between two classes of tasks (English vs Spanish), so the model doesn't need to learn very sophisticated task embeddings to group the tasks correctly for transfer. It would be good to see other experiments where there is less of a trivial structure distinguishing tasks, to check if transfer helps.
Also, I find it surprising that the Corpus Cine sentiment task embedding is not correlated at all with the SST sentiment task. If the controller is really learning something interesting about the nature of the tasks, I would have expected a differential effect, such that IMDB is only correlated with SST, but Corpus Cine is correlated to both the Spanish language identification task and SST. Perhaps, this is worth some discussion.
Finally, it's not clear to me why the multitask architecture was used in the experiment even when no multi-task pre-training was conducted: shouldn't the simple neural architecture search method be used in this case?
Minor points:
"diffferentiated": different?
"outputted actions": output actions
"the aim of increase the training stability": the aim of increasing training stability
Insert references for Polyak averaging and Savitzky-Golay filtering.
Figure 3: specify that the Socher 2013 result is for SST
Figure 4: does LSS stand for SST?
I'm confused by Fig. 6: why aren't the diagonal values 100%?
MNMS is referred to as MNAS in Figure 5.
For architecture search, the neuroevolution literature should also be cited (https://www.oreilly.com/ideas/neuroevolution-a-different-kind-of-deep-learning).


We present Cross-View Training (CVT), a simple but effective method for deep semi-supervised learning. On labeled examples, the model is trained with standard cross-entropy loss. On an unlabeled example, the model first performs inference (acting as a "teacher") to produce soft targets. The model then learns from these soft targets (acting as a "student"). We deviate from prior work by adding multiple auxiliary student prediction layers to the model. The input to each auxiliary student layer is a sub-network of the full model that has a restricted view of the input (e.g., only seeing one region of an image). The students can learn from the teacher (the full model) because the teacher sees more of each example. Concurrently, the students improve the quality of the representations used by the teacher as they learn to make predictions with limited data. When combined with Virtual Adversarial Training, CVT improves upon the current state-of-the-art on semi-supervised CIFAR-10 and semi-supervised SVHN. We also apply CVT to train models on five natural language processing tasks using hundreds of millions of sentences of unlabeled data. On all tasks CVT substantially outperforms supervised learning alone, resulting in models that improve upon or are competitive with the current state-of-the-art.

This paper presents a so-called cross-view training for semi-supervised deep models. Experiments were conducted on various data sets and experimental results were reported.
Pros:
* Studying semi-supervised learning techniques for deep models is of practical significance.
Cons:
* The novelty of this paper is marginal. The use of unlabeled data is in fact a self-training process. Leveraging the sub-regions of the image to improve performance is not new and has been widely-studied in image classification and retrieval.
* The proposed approach suffers from a technical weakness or flaw. For the self-labeled data, the prediction of each view is enforced to be same as the assigned self-labeling. However, since each view related to a sub-region of the image (especially when the model is not so deep), it is less likely for this region to contain the representation of the concepts (e.g., some local region of an image with a horse may exhibit only grass); enforcing the prediction of this view to be the same self-labeled concepts (e.g,“horse”) may drive the prediction away from what it should be ( e..g, it will make the network to predict grass as horse). Such a flaw may affect the final performance of the proposed approach.
* The word “view” in this paper is misleading. The “view” in this paper is corresponding to actually sub-regions in the images
* The experimental results indicate that the proposed approach fails to perform better than the compared baselines in table 2, which reduces the practical significance of the proposed approach.


Published as a conference paper at ICLR 2018 TD OR NOT TD: ANALYZING THE ROLE OF TEMPORAL DIFFERENCING IN DEEP REINFORCEMENT LEARNING
Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (MC). How do these results hold up in deep RL, which deals with perceptually complex environments and deep nonlinear models? In this paper, we re-examine the role of TD in modern deep RL, using specially designed environments that control for specific factors that affect performance, such as reward sparsity, reward delay, and the perceptual complexity of the task. When comparing TD with infinite-horizon MC, we are able to reproduce classic results in modern settings. Yet we also find that finite-horizon MC is not inferior to TD, even when rewards are sparse or delayed. This makes MC a viable alternative to TD in deep RL.

This paper includes several controlled empirical studies comparing MC and TD methods in predicting of value function with complex DNN function approximators. Such comparison has been carried out both in theory and practice for simple low dimensional environments with linear (and RKHS) value function approximation showing how TD methods can have much better sample complexity and overall performance compared to pure MC methods. This paper shows some results to the contrary when applying RL to complex perceptual observation space.
The main results include:
(1) In a rollout update a mix of MC and TD update (i.e. a rollout of > 1 and < horizon) outperforms either extreme. This is inline with TD-lambda analysis in previous work.
(2) Pure MC methods can outperform TD methods when the rewards becomes noisy.
(3) TD methods can outperform pure MC methods when the return is mostly dominated by the reward in the terminal state.
(4) MC methods tend to degrade less when the reward signal is delayed.
(5) Somewhat surprising: MC methods seems to be on-par with TD methods when the reward is sparse and even longer than the rollout horizon.
(6) MC methods can outperform TD methods with more complex and high dimensional perceptual inputs.
The authors conjecture that several of the above observations can be explained by the fact that the training target in MC methods is "ground truth" and do not rely on bootstrapping from the current estimates as is done in a TD rollout. They suggest that training on such signal can be beneficial when training deep models on complex perceptual input spaces.
The contributions of the paper are in parts surprising and overall interesting. I believe there are far more caveats in this analysis than what is suggested in the paper and the authors should avoid over-generalizing the results based on a few domains and the analysis of a small set of algorithms. Nonetheless I find the results interesting to the RL community and a starting point to further analysis of the MC methods (or adaptations of TD methods) that work better with image observation spaces. Publishing the code, as the authors mentioned, would certainly help with that.
Notes:
- I find the description of the Q_MC method presented in the paper very confusing and had to consult the reference to understand the details. Adding a couple of equations on this would improve the readability of the paper.
- The first mention of partial observability can be moved to the introduction.
- Adding results for m=3 to table 2 would bring further insight to the comparison.
- The results for the perceptual complexity experiment seem contradictory and inconclusive. One would expect Q_MC to work well in Grid Map domain if the conjecture put forth by the authors was to hold universally.
- In the study on reward sparsity, although a prediction horizon of 32 is less than the average steps needed to get to a rewarding state, a blind random walk might be enough to take the RL agent to a close-enough neighbourhood from which a greedy MC-based policy has a direct path to the goal. What is missing from this picture is when a blind walk cannot reach such a state, e.g. when a narrow corridor is present in the environment. Such a case cannot be resolved by a short horizon MC method. In other words, a sparse reward setting is only "difficult" if getting into a good neighbourhood requires long term planning and cannot be resolved by a (pseudo) blind random walk.
- The extrapolation of the value function approximator can also contribute to why the limited horizon MC method can see beyond its horizon in a sparse reward setting. That is, even if there is no way to reach a reward state in 32 steps, an MC value function approximation with horizon 32 can extrapolate from similar looking observed states that have a short path to a rewarding state, enough to be better than a blind random walk. It would have been nice to experiment with increasing model complexity to study such effect.


Published as a conference paper at ICLR 2018 FRATERNAL DROPOUT
Recurrent neural networks (RNNs) form an important class of architectures among neural networks useful for language modeling and sequential prediction. However, optimizing RNNs is known to be harder compared to feed-forward neural networks. A number of techniques have been proposed in literature to address this problem. In this paper we propose a simple technique called fraternal dropout that takes advantage of dropout to achieve this goal. Specifically, we propose to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions. In this way our regularization encourages the representations of RNNs to be invariant to dropout mask, thus being robust. We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout. We evaluate our model and achieve state-of-the-art results in sequence modeling tasks on two benchmark datasets -Penn Treebank and Wikitext-2. We also show that our approach leads to performance improvement by a significant margin in image captioning (Microsoft COCO) and semi-supervised (CIFAR-10) tasks.

The authors present Fraternal dropout as an improvement over Expectation-linear dropout (ELD) in terms of convergence and demonstrate the utility of Fraternal dropout on a number of tasks and datasets.
At test time, more often than not, people apply dropout in deterministic mode while at training time masks are sampled randomly. The paper addresses this issue by trying to reduce the gap.
I have 1.5 high level comments:
- Dropout can be applied by averaging results corresponding to randomly sampled masks ('MC eval'). This should not be ignored, and preferrably included in the evaluation.
- It could be made clearer why the proposed regularization would make the aforementioned gap smaller. Intuitively, the bias of the deterministic approximation (compared to the MC eval) should also play a role. It may be worth asking whether the bias changes? A possibility is that MC and deterministic evaluations meet halfway and with fraternal dropout MC eval is worse than without.
Details:
- The notation is confusing: p() looks like a probability distribution, z looks like a latent variable, p^t and l^t have superscripts instead of Y having a subscript, z^t is a function of X. Wouldn't f(X_t) be preferrable to p^t(z_t)?
- The experiments are set up and executed with care, but section 4 could be improved by providings details (as much as in section 5). The results on PTB and Wikitext-2 are really good. However, why not compare to ELD here? Section 5 leads the reader to believe that ELD would be equally good.
- Section 5 could be the most interesting part of the paper. This is where different regularization methods are compared (by the way, this is not "ablation"). It is somewhat unfortunate that due to lack of computational resources the comparisons are made at a single hyperparameter setting.
All in all, the results of section 4 are clearly good, but are they better than those of ELD? Evaluation and interpretation of results in section 5 is made difficult by the omission of the most informative quantity which Fraternal dropout is supposed to be approximating.


GANs have shown how deep neural networks can be used for generative modeling, aiming at achieving the same impact that they brought for discriminative modeling. The first results were impressive, GANs were shown to be able to generate samples in high dimensional structured spaces, like images and text, that were no copies of the training data. But generative and discriminative learning are quite different. Discriminative learning has a clear end, while generative modeling is an intermediate step to understand the data or generate hypothesis. The quality of implicit density estimation is hard to evaluate, because we cannot tell how well a data is represented by the model. How can we certainly say that a generative process is generating natural images with the same distribution as we do? In this paper, we noticed that even though GANs might not be able to generate samples from the underlying distribution (or we cannot tell at least), they are capturing some structure of the data in that high dimensional space. It is therefore needed to address how we can leverage those estimates produced by GANs in the same way we are able to use other generative modeling algorithms.

The main take-away messages of this paper seem to be:
1. GANs don't really match the target distribution. Some previous theory supports this, and some experiments are provided here demonstrating that the failure seems to be largely in under-sampling the tails, and sometimes perhaps in introducing spurious modes.
2. Even if GANs don't exactly match the target distribution, their outputs might still be useful for some tasks.
(I wouldn't be surprised if you disagree with what the main takeaways are; I found the flow of the paper somewhat disjointed, and had something of a hard time identifying what the "point" was.)
Mode-dropping being a primary failure mode of GANs is already a fairly accepted hypothesis in the community (see, e.g. Mode Regularized GANs, Che et al ICLR 2017, among others), though some extra empirical evidence is provided here.
The second point is, in my opinion, simultaneously (i) an important point that more GAN research should take to heart, (ii) relatively obvious, and (iii) barely explored in this paper. The only example in the paper of using a GAN for something other than directly matching the target distribution is PassGAN, and even that is barely explored beyond saying that some of the spurious modes seem like reasonable-ish passwords.
Thus though this paper has some interesting aspects to it, I do not think its contributions rise to the level required for an ICLR paper.
Some more specifics:
Section 2.1 discusses four previous theoretical results about the convergence of GANs to the true density. This overview is mostly reasonable, and the discussion of Arora et al. (2017) and Liu et al. (2017) do at least vaguely support the conclusion in the last section of this paragraph. But this section is glaringly missing an important paper in this area: Arjovsky and Bottou (2017), cited here only in passing in the introduction, who proved that typical GAN architectures *cannot* exactly match the data distribution. Thus the question of metrics for convergence is of central importance, which it seems should be important to the topic of the present paper. (Figure 3 of Danihelka et al. https://arxiv.org/abs/1705.05263 gives a particularly vivid example of how optimizing different metrics can lead to very different results.) Presumably different metrics lead to models that are useful for different final tasks.
Also, although they do not quite fit into the framing of this section, Nowozin et al.'s local convergence proof and especially the convergence to a Nash equilibrium argument of Heusel et al. (NIPS 2017, https://arxiv.org/abs/1706.08500) should probably be mentioned here.
The two sample testing section of this paper, discussed in Section 2.2 and then implemented in Section 3.1.1, seems to be essentially a special case of what was previously done by Sutherland et al. (2017), except that it was run on CIFAR-10 as well. However, the bottom half of Table 1 demonstrates that something is seriously wrong with the implementation of your tests: using 1000 bootstrap samples, you should reject H_0 at approximately the nominal rate of 5%, not about 50%! To double-check, I ran a median-heuristic RBF kernel MMD myself on the MNIST test set with N_test = 100, repeating 1000 times, and rejected the null 4.8% of the time. My code is available at https://gist.github.com/anonymous/2993a16fbc28a424a0e79b1c8ff31d24 if you want to use it to help find the difference from what you did. Although Table 1 does indicate that the GAN distribution is more different from the test set than the test set is from itself, the apparent serious flaw in your procedure makes those results questionable. (Also, it seems that your entry labeled "MMD" in the table is probably n * MMD_b^2, which is what is computed by the code linked to in footnote 2.)
The appendix gives a further study of what went wrong with the MNIST GAN model, arguing based on nearest-neighbors that the GAN model is over-representing modes and under-representing the tails. This is fairly interesting; certainly more interesting than the rehash of running MMD tests on GAN outputs, in my opinion.
Minor:
In 3.1.1, you say "ideally the null hypothesis H0 should never be rejected" – it should be rejected at most an alpha portion of the time.
In the description of section 3.2, you should clarify whether the train-test split was done such that unique passwords were assigned to a single fold or not: did 123456 appear in both folds? (It is not entirely clear whether it should or not; both schemes have possible advantages for evaluation.)


The fundamental task of general density estimation has been of keen interest to machine learning. Recent advances in density estimation have either: a) proposed a flexible model to estimate the conditional factors of the chain rule, p(x i x i−1 , . . .); or b) used flexible, non-linear transformations of variables of a simple base distribution. Instead, this work jointly leverages transformations of variables and autoregressive conditional models, and proposes novel methods for both. We provide a deeper understanding of our methods, showing a considerable improvement through a comprehensive study over both real world and synthetic data. Moreover, we illustrate the use of our models in outlier detection and image modeling tasks.

This paper is well constructed and written. It consists of a number of broad ideas regarding density estimation using transformations of autoregressive networks. Specifically, the authors examine models involving linear maps from past states (LAM) and recurrence relationships (RAM).
The critical insight is that the hidden states in the LAM are not coupled allowing considerable flexibility between consecutive conditional distributions. This is at the expense of an increased number of parameters and a lack of information sharing. In contrast, the RAM transfers information between conditional densities via the coupled hidden states allowing for more constrained smooth transitions.
The authors then explored a variety of transformations designed to increase the expressiveness of LAM and RAM. The authors importantly note that one important restriction on the class of transformations is the ability to evaluate the Jacobian of the transformation efficiently. A composite of transformations coupled with the LAM/RAM networks provides a highly expressive model for modelling arbitrary joint densities but retaining interpretable conditional structure.
There is a rich variety of synthetic and real data studies which demonstrate that LAM and RAM consistently rank amongst the top models demonstrating potential utility for this class of models.
Whilst the paper provides no definitive solutions, this is not the point of the work which seeks to provide a description of a general class of potentially useful models.


MASKGAN: BETTER TEXT GENERATION VIA FILLING IN THE
Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined by validation perplexity even though this is not a direct measure of the quality of the generated text. Additionally, these models are typically trained via maximum likelihood and teacher forcing. These methods are well-suited to optimizing perplexity but can result in poor sample quality since generating text requires conditioning on sequences of words that may have never been observed at training time. We propose to improve sample quality using Generative Adversarial Networks (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation. GANs were originally designed to output differentiable values, so discrete language generation is challenging for them. We claim that validation perplexity alone is not indicative of the quality of text generated by a model. We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context. We show qualitatively and quantitatively, evidence that this produces more realistic conditional and unconditional text samples compared to a maximum likelihood trained model.

Generating high-quality sentences/paragraphs is an open research problem that is receiving a lot of attention. This text generation task is traditionally done using recurrent neural networks. This paper proposes to generate text using GANs. GANs are notorious for drawing images of high quality but they have a hard time dealing with text due to its discrete nature. This paper's approach is to use an actor-critic to train the generator of the GAN and use the usual maximum likelihood with SGD to train the discriminator. The whole network is trained on the "fill-in-the-blank" task using the sequence-to-sequence architecture for both the generator and the discriminator. At training time, the generator's encoder computes a context representation using the masked sequence. This context is conditioned upon to generate missing words. The discriminator is similar and conditions on the generator's output and the masked sequence to output the probability of a word in the generator's output being fake or real. With this approach, one can generate text at test time by setting all inputs to blanks.
Pros and positive remarks:
--I liked the idea behind this paper. I find it nice how they benefited from context (left context and right context) by solving a "fill-in-the-blank" task at training time and translating this into text generation at test time.
--The experiments were well carried through and very thorough.
--I second the decision of passing the masked sequence to the generator's encoder instead of the unmasked sequence. I first thought that performance would be better when the generator's encoder uses the unmasked sequence. Passing the masked sequence is the right thing to do to avoid the mismatch between training time and test time.
Cons and negative remarks:
--There is a lot of pre-training required for the proposed architecture. There is too much pre-training. I find this less elegant.
--There were some unanswered questions:
            (1) was pre-training done for the baseline as well?
            (2) how was the masking done? how did you decide on the words to mask? was this at random?
            (3) it was not made very clear whether the discriminator also conditions on the unmasked sequence. It needs to but
                  that was not explicit in the paper.
--Very minor: although it is similar to the generator, it would have been nice to see the architecture of the discriminator with example input and output as well.
Suggestion: for the IMDB dataset, it would be interesting to see if you generate better sentences by conditioning on the sentiment as well.


Published as a conference paper at ICLR 2018 SKIP RNN: LEARNING TO SKIP STATE UPDATES IN RECURRENT NEURAL NETWORKS
Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc. github.io/skiprnn-2017-telecombcn/.

UPDATE: Following the author's response I've increased my score from 5 to 6. The revised paper includes many of the additional references that I suggested, and the author response clarified my confusion over the Charades experiments; their results are indeed close to state-of-the-art on Charades activity localization (slightly outperformed by [6]), which I had mistakenly confused with activity classification (from [5]).
The paper proposes the Skip RNN model which allows a recurrent network to selectively skip updating its hidden state for some inputs, leading to reduced computation at test-time. At each timestep the model emits an update probability; if this probability is over a threshold then the next input and state update will be skipped. The use of a straight-through estimator allows the model to be trained with standard backpropagation. The number of state updates that the model learns to use can be controlled with an auxiliary loss function. Experiments are performed on a variety of tasks, demonstrating that the Skip-RNN compares as well or better than baselines even when skipping nearly half its state updates.
Pros:
- Task of reducing computation by skipping inputs is interesting
- Model is novel and interesting
- Experiments on multiple tasks and datasets confirm the efficacy of the method
- Skipping behavior can be controlled via an auxiliary loss term
- Paper is clearly written
Cons:
- Missing comparison to prior work on sequential MNIST
- Low performance on Charades dataset, no comparison to prior work
- No comparison to prior work on IMDB Sentiment Analysis or UCF-101 activity classification
The task of reducing computation by skipping RNN inputs is interesting, and the proposed method is novel, interesting, and clearly explained. Experimental results across a variety of tasks are convincing; in all tasks the Skip-RNNs achieve their goal of performing as well or better than equivalent non-skipping variants. The use of an auxiliary loss to control the number of state updates is interesting; since it sometimes improves performance it appears to have some regularizing effect on the model in addition to controlling the trade-off between speed and accuracy.
However, where possible experiments should compare directly with prior published results on these tasks; none of the experiments from the main paper or supplementary material report any numbers from any other published work.
On permuted MNIST, Table 2 could include results from [1-4]. Of particular interest is [3], which reports 98.9% accuracy with a 100-unit LSTM initialized with orthogonal and identity weight matrices; this is significantly higher than all reported results for the sequential MNIST task.
For Charades, all reported results appear significantly lower than the baseline methods reported in [5] and [6] with no explanation. All methods work on “fc7 features from the RGB stream of a two-stream CNN provided by the organizers of the [Charades] challenge”, and the best-performing method (Skip GRU) achieves 9.02 mAP. This is significantly lower than the two-stream results from [5] (11.9 mAP and 14.3 mAP) and also lower than pretrained AlexNet features averaged over 30 frames and classified with a linear SVM, which [5] reports as achieving 11.3 mAP. I don’t expect to see state-of-the-art performance on Charades; the point of the experiment is to demonstrate that Skip-RNNs perform as well or better than their non-skipping counterparts, which it does. However I am surprised at the low absolute performance of all reported results, and would appreciate if the authors could help to clarify whether this is due to differences in experimental setup or something else.
In a similar vein, from the supplementary material, sentiment analysis on IMDB and action classification on UCF-101 are well-studied problems, but the authors do not compare with any previously published results on these tasks.
Though experiments may not show show state-of-the-art performance, I think that they still serve to demonstrate the utility of the Skip-RNN architecture when compared side-by-side with a similarly tuned non-skipping baseline. However I feel that the authors should include some discussion of other published results.
On the whole I believe that the task and method are interesting, and experiments convincingly demonstrate the utility of Skip-RNNs compared to the author’s own baselines. I will happily upgrade my rating of the paper if the authors can address my concerns over prior work in the experiments.
References
[1] Le et al, “A Simple Way to Initialize Recurrent Networks of Rectified Linear Units”, arXiv 2015
[2] Arjovsky et al, “Unitary Evolution Recurrent Neural Networks”, ICML 2016
[3] Cooijmans et al, “Recurrent Batch Normalization”, ICLR 2017
[4] Zhang et al, “Architectural Complexity Measures of Recurrent Neural Networks”, NIPS 2016
[5] Sigurdsson et al, “Hollywood in homes: Crowdsourcing data collection for activity understanding”, ECCV 2016
[6] Sigurdsson et al, “Asynchronous temporal fields for action recognition”, CVPR 2017


The maximum mean discrepancy (MMD) between two probability measures P and Q is a metric that is zero if and only if all moments of the two measures are equal, making it an appealing statistic for two-sample tests. Given i.i.d. samples from P and Q, Gretton et al. (2012) show that we can construct an unbiased estimator for the square of the MMD between the two distributions. If P is a distribution of interest and Q is the distribution implied by a generative neural network with stochastic inputs, we can use this estimator to train our neural network. However, in practice we do not always have i.i.d. samples from our target of interest. Data sets often exhibit biases-for example, under-representation of certain demographics-and if we ignore this fact our machine learning algorithms will propagate these biases. Alternatively, it may be useful to assume our data has been gathered via a biased sample selection mechanism in order to manipulate properties of the estimating distribution Q.
In this paper, we construct an estimator for the MMD between P and Q when we only have access to P via some biased sample selection mechanism, and suggest methods for estimating this sample selection mechanism when it is not already known. We show that this estimator can be used to train generative neural networks on a biased data sample, to give a simulator that reverses the effect of that bias.

This paper proposes an importance-weighted estimator of the MMD, in order to estimate the MMD between distributions based on samples biased according to a known scheme. It then discusses how to estimate the scheme when it is unknown, and further proposes using it in either the MMD-based generative models of Y. Li et al. (2015) / Dziugaite et al. (2015), or in the MMD GAN of C.-L. Li et al. (2017).
The estimator itself is natural (and relatively obvious), though it has some drawbacks that aren't fully discussed (below).
The application to GAN-type learning is reasonable, and topical. The first, univariate, experiment shows that the scheme is at least plausible. But the second experiment, involving a simple T ratio based on whether an MNIST digit is a 0 or a 1, doesn't even really work! (The best model only gets the underrepresented class from 20% up to less than 40%, rather than the desired 50%, and the "more realistic" setting only to 33%.)
It would be helpful to debug whether this is due to the classifier being incorrect, estimator inaccuracies, or what. In particular, I would try using T based on a pretrained convnet independent of the autoencoder representation in the MMD GAN, to help diagnose where the failure mode comes from.
Without at least a working should-be-easy example like this, and with the rest of the paper's technical contribution so small, I just don't think this paper is ready for ICLR.
It's also worth noting that the equivalent algorithm for either vanilla GANs or Wasserstein GANs would be equally obvious.
Estimator:
In the discussion about (2): where does the 1/m bias come from? This doesn't seem to be in Robert and Casella section 3.3.2, which is the part of the book that I assume you're referring to (incidentally, you should specify that rather than just citing a 600-page textbook).
Moreover, it is worth noting that Robert and Cassela emphasize that if E[1 / \tilde T] is infinite, the importance sampling estimator can be quite bad (for example, the estimator may have infinite variance). This happens when \tilde T puts mass in a neighborhood around 0, i.e. when the thinned distribution doesn't have support at any place that P does. In the biased-observations case, this is in some sense unsurprising: if we don't see *any* data in a particular class of inputs, then our estimates can be quite bad (since we know nothing about a group of inputs that might strongly affect the results). In the modulating case, the equivalent situation is when F(x) lacks a mean, which seems less likely. Thus although this is probably not a huge problem for your case, it's worth at least mentioning. (See also the following relevant blog posts:  https://radfordneal.wordpress.com/2008/08/17/the-harmonic-mean-of-the-likelihood-worst-monte-carlo-method-ever/ and https://xianblog.wordpress.com/2012/03/12/is-vs-self-normalised-is/ .)
The paper might be improved by stating (and proving) a theorem with expressions for the rate of convergence of the estimator, and how they depend on T.
Minor:
Another piece of somewhat-related work is Xiong and Schneider, Learning from Point Sets with Observational Bias, UAI 2014.
Sutherland et al. 2016 and 2017, often referenced in the same block of citations, are the same paper.
On page 3, above (1): "Since we have projected the distributons into an infinite-dimensional space, the distance between the two distributions is zero if and only if all their moments are the same." An infinite-dimensional space isn't enough; the kernel must further be characteristic, as you mention. See e.g. Sriperumbuder et al. (AISTATS 2010) for more details.
Figure 1(b) seems to be plotting only the first term of \tilde T, without the + 0.5.


The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of the stochastic gradient, whereas the update magnitude is solely determined by an estimate of its relative variance. We disentangle these two aspects and analyze them in isolation, shedding light on ADAM's inner workings. Transferring the "variance adaptation" to momentum-SGD gives rise to a novel method, completing the practitioner's toolbox for problems where ADAM fails.

Summary:
The paper is trying to improve Adam based on variance adaption with momentum. Two algorithms are proposed, M-SSD (Stochastic Sign Descent with Momentum) and M-SVAG (Stochastic Variance-Adapted Gradient with Momentum) to solve finite sum minimization problem. The convergence analysis is provided for SVAG for strongly convex case. Numerical experiments are provided for some standard neural network structures with three common datasets MNIST, CIFAR10 and CIFAR100 compared the performance of M-SSD and M-SVAG to two existing algorithms: SGD momentum and Adam.

Comments:
Page 4, line 5: You should define \nu clearly.
Theorem 1: In the strongly convex case, assumption E ||g_t ||^2 \leq G^2 (if G is a constant) is too strong. In this case, G could be equal to infinity. If G is not infinity, you already assume that your algorithm converges, that is the reason why this assumption is not so good for strongly convex. If G is infinity (this is really possible for strongly convex), your proof would get a trouble as eq. (40) is not valid anymore.
Also, to compute \gamma_{t,i}, it requires to compute \nabla f_{t,i}, which is full gradient. By doing this, the computational cost should add the dependence of M, which is very large as you mentioned in the introduction. According to your rate O(1/t), the complexity is worse than that of gradient descent and SGD as well.
As I understand, there is no theoretical results for M-SSG and M-SVAG, but only the result for SVAG with exact \eta_i^2 in the strongly convex case. Also, theoretical results are not strong enough. Hence, the experiments need to make more convincingly, at least for some different complicated architecture of deep neural network. As I see, in some dataset, Adam performs better than M-SSD, some another dataset, Adam performs better than M-SVAG. Same situation for M-SGD. My question is that: When should we use M-SSD or M-SVAG? For a given dataset, why should we not use Adam or M-SGD (or other existing algorithms such as Adagrad, RMSprop), but your algorithms?
You should do more experiments to various dataset and architectures to be more convincing since theoretical results are not strong enough. Would you think to try to use VGG or ResNet to ImageNet?
I like the idea of the paper but I would love if the author(s) could improve more theoretical results to convince people. Otherwise, the results in this paper could not be considered as good enough. At this moment, I think the paper is still not ready for the publication.
Minor comments:
Page 2, in eq. (6): You should mention that “1” is a vector.
Page 4, line 4: Q in R^{d} => Q in R^{d x d}
Page 6, Theorem 1: You should define the finite sum optimization problem with f since you have not used it before.
Page 6, Theorem 1: You should use another notation for “\mu”-strongly convex parameter since you have another “\mu”-momentum parameter in section 3.4
Page 4, Page 7: Be careful with the case when c = 0 (page 4) and mu = 1 (page 7-8) with dividing by 0.


This paper puts forward a new text to tensor representation that relies on information compression techniques to assign shorter codes to the most frequently used characters. This representation is language-independent with no need of pretraining and produces an encoding with no information loss. It provides an adequate description of the morphology of text, as it is able to represent prefixes, declensions, and inflections with similar vectors and are able to represent even unseen words on the training dataset. Similarly, as it is compact yet sparse, is ideal for speed up training times using tensor processing libraries. As part of this paper, we show that this technique is especially effective when coupled with convolutional neural networks (CNNs) for text classification at character-level. We apply two variants of CNN coupled with it. Experimental results show that it drastically reduces the number of parameters to be optimized, resulting in competitive classification accuracy values in only a fraction of the time spent by one-hot encoding representations, thus enabling training in commodity hardware.

The manuscript proposed to use prefix codes to compress the input to a neural network for text classification. It builds upon the work by Zhang & LeCun (2015) where the same tasks are used.
There are several issues with the paper and I cannot recommend acceptance of the paper in the current state.
- It looks like it is not finished.
- the datasets are not described properly.
- It is not clear to me where the baseline results come from.
 They do not match up to the Zhang paper (I have tried to find the matching accuracies there).
- It is not clear to me what the baselines actually are or how I can found more info on those.
- the results are not remarkable.
Because of this, the paper needs to be updated and cleaned up before it can be properly reviewed.
On top of this, I do not enjoy the style the paper is written in, the language is convoluted.
For example: “The effort to use Neural Convolution Networks for text classification tasks is justified by the possibility of appropriating tools from the recent developments of techniques, libraries and hardware used especially in the image classification “
I do not know which message the paper tries to get across here.
As a reviewer my impression (which is subjective) is that the authors used difficult language to make the manuscript look more impressive.
The acknowledgements should not be included here either.


A HIERARCHICAL MODEL FOR DEVICE PLACEMENT
We introduce a hierarchical model for efficient placement of computational graphs onto hardware devices, especially in heterogeneous environments with a mixture of CPUs, GPUs, and other computational devices. Our method learns to assign graph operations to groups and to allocate those groups to available devices. The grouping and device allocations are learned jointly. The proposed method is trained with policy gradient and requires no human intervention. Experiments with widely-used computer vision and natural language models show that our algorithm can find optimized, non-trivial placements for TensorFlow computational graphs with over 80,000 operations. In addition, our approach outperforms placements by human experts as well as a previous state-of-the-art placement method based on deep reinforcement learning. Our method achieves runtime reductions of up to 60.6% per training step when applied to models such as Neural Machine Translation.

The paper seems clear enough and original enough. The idea of jointly forming groups of operations to colocate and figure out placement on devices seems to hold merit. Where the paper falls short is motivating the problem setting. Traditionally, for determining optimal execution plans, one may resort to cost-based optimization (e.g., database management systems). This paper's introduction provides precisely 1 statement to suggest that may not work for deep learning. Here's the relevant phrase: "the cost function is typically non-stationary due to the interactions between multiple devices". Unfortunately, this statement raises more questions than it answers. Why are the cost functions non-stationary? What exactly makes them dynamic? Are we talking about a multi-tenancy setting where multiple processes execute on the same device? Unlikely, because GPUs are involved. Without a proper motivation, its difficult to appreciate the methods devised.
Pros:
- Jointly optimizing forming of groups and placing these seems to have merit
- Experiments show improvements over placement by human "experts"
- Targets an important problem
Cons:
- Related work seems inadequately referenced. There exist other linear/tensor algebra engines/systems that perform such optimization including placing operations on devices in a distributed setting. This paper should at least cite those papers and qualitatively compare against those approaches. Here's one reference (others should be easy to find): "SystemML's Optimizer: Plan Generation for Large-Scale Machine Learning Programs" by Boehm et al, IEEE Data Engineering Bulletin, 2014.
- The methods are not well motivated. There are many approaches to devising optimal execution plans, e.g., rule-based, cost-based, learning-based. In particular, what makes cost-based optimization inapplicable? Also, please provide some reasoning behind your hypothesis which seems to be that while costs may be dynamic, optimally forming groups and placing them is learn-able.
- The template seems off. I don't see the usual two lines under the title ("Anonymous authors", "Paper under double-blind review").
- The title seems misleading. ".... Device Placement" seems to suggest that one is placing devices when in fact, the operators are being placed.


We analyze the expressiveness and loss surface of practical deep convolutional neural networks (CNNs) with shared weights and max pooling layers. We show that such CNNs produce linearly independent features at a "wide" layer which has more neurons than the number of training samples. This condition holds e.g. for the VGG network. Furthermore, we provide for such wide CNNs necessary and sufficient conditions for global minima with zero training error. For the case where the wide layer is followed by a fully connected layer we show that almost every critical point of the empirical loss is a global minimum with zero training error. Our analysis suggests that both depth and width are very important in deep learning. While depth brings more representational power and allows the network to learn high level features, width smoothes the optimization landscape of the loss function in the sense that a sufficiently wide network has a well-behaved loss surface with almost no bad local minima.

This paper presents an analysis of convolutional neural networks from the perspective of how the rank of the features is affected by the kinds of layers found in the most popular networks. Their analysis leads to the formulation of a certain theorem about the global minima with respect to parameters in the latter portion of the network.
The authors ask important questions, but I am not sure that they obtain important answers. On the plus side, I'm glad that people are trying to further our understanding our neural networks, and I think that their investigation is worthy of being published.
They present a collection of assumptions, lemmas, and theorems. They have no choice but to have assumptions, because they want to abstract away the "data" part of the analysis while still being able to use certain properties about the rank of the features at certain layers.
Most of my doubts about this paper come from the feeling that equivalent results could be obtained with a more elegant argument about perturbation theory, instead of something like the proof of Lemma A1. That being said, it's easy to voice such concerns, and I'm willing to believe that there might not exist a simple way to derive the same results with an approach more along the line of "whatever your data, pick whatever small epsilon, and you can always have the desired properties by perturbing your data by that small epsilon in a random direction". Have the authors tried this ?
I'm not sure if the authors were the first to present this approach of analyzing the effects of convolutions from a "patch perspective", but I think this is a clever approach. It simplifies the statement of some of their results. I also like the idea of factoring the argument along the concept of some critical "wide layer".
Good review of the literature.
I wished the paper was easier to read. Some of the concepts could have been illustrated to give the reader some way to visualize the intuitive notions. For example, maybe it would have been interesting to plot the rank of features a every layer for LeNet+MNIST ?
At the end of the day, if a friend asked me to summarize the paper, I would tell them :
"Features are basically full rank. Then they use a square loss and end up with an over-parametrized system, so they can achieve loss zero (i.e. global minimum) with a multitude of parameters values."
Nitpicking :
"This paper is one of the first ones, which studies CNNs."
This sentence is strange to read, but I can understand what the authors mean.
"This is true even if the bottom layers (from input to the wide layer) and chosen randomly with probability one."
There's a certain meaning to "with probability one" when it comes to measure theory. The authors are using it correctly in the rest of the paper, but in this sentence I think they simply mean that something holds if "all" the bottom layers have random features.


Survival Analysis (time-to-event analysis) in the presence of multiple possible adverse events, i.e., competing risks, is a challenging, yet very important problem in medicine, finance, manufacturing, etc. Extending classical survival analysis to competing risks is not trivial since only one event (e.g. one cause of death) is observed and hence, the incidence of an event of interest is often obscured by other related competing events. This leads to the nonidentifiability of the event times distribution parameters, which makes the problem significantly more challenging. In this work we introduce Siamese Survival Prognosis Network, a novel Siamese Deep Neural Network architecture that is able to effectively learn from data in the presence of multiple adverse events. The Siamese Survival Network is especially crafted to issue pairwise concordant time-dependent risks, in which longer event times are assigned lower risks. Furthermore, our architecture is able to directly optimize an approximation to the C-discrimination index, rather than relying on well-known metrics of cross-entropy etc., and which are not able to capture the unique requirements of survival analysis with competing risks. Our results show consistent performance improvements on a number of publicly available medical datasets over both statistical and deep learning state-of-the-art methods.

The authors tackle the problem of estimating risk in a survival analysis setting with competing risks. They propose directly optimizing the time-dependent discrimination index using a siamese survival network. Experiments on several real-world dataset reveal modest gains in comparison with the state of the art.
- The authors should clearly highlight what is their main technical contribution. For example, Eqs. 1-6 appear to be background material since the time-dependent discrimination index is taken from the literature, as the authors point out earlier. However, this is unclear from the writing.
- One of the main motivations of the authors is to propose a model that is specially design to avoid the nonidentifiability issue in an scenario with competing risks. It is unclear why the authors solution is able to solve such an issue, specially given the modest reported gains in comparison with several competitive baselines. In other words, the authors oversell their own work, specially in comparison with the state of the art.
- The authors use off-the-shelf siamese networks for their settting and thus it is questionable there is any novelty there. The application/setting may be novel, but not the architecture of choice.
- From Eq. 4 to Eq. 5, the authors argue that the denominator does not depend on the model parameters and can be ignored. However, afterwards the objective does combine time-dependent discrimination indices of several competing risks, with different denominator values. This could be problematic if the risks are unbalanced.
- The competitive gain of the authors method in comparison with other competing methods is minor.
- The authors introduce F(t, D | x) as cumulative incidence function (CDF) at the beginning of section 2, however, afterwards they use R^m(t, x), which they define as risk of the subject experiencing event m before t. Is the latter a proxy for the former? How are they related?


A SIMPLE NEURAL ATTENTIVE META-LEARNER
Deep neural networks excel in regimes with large amounts of data, but tend to struggle when data is scarce or when they need to adapt quickly to changes in the task. In response, recent work in meta-learning proposes training a meta-learner on a distribution of similar tasks, in the hopes of generalization to novel but related tasks by learning a high-level strategy that captures the essence of the problem it is asked to solve. However, many recent meta-learning approaches are extensively hand-designed, either using architectures specialized to a particular application, or hard-coding algorithmic components that constrain how the meta-learner solves the task. We propose a class of simple and generic meta-learner architectures that use a novel combination of temporal convolutions and soft attention; the former to aggregate information from past experience and the latter to pinpoint specific pieces of information. In the most extensive set of meta-learning experiments to date, we evaluate the resulting Simple Neural AttentIve Learner (or SNAIL) on several heavily-benchmarked tasks. On all tasks, in both supervised and reinforcement learning, SNAIL attains state-of-the-art performance by significant margins.

The paper proposes a general neural network structure that includes TC (temporal convolution) blocks and Attention blocks for meta-learning, specifically, for episodic task learning. Through intensive experiments on various settings including few-shot image classification on Omniglot and Mini-ImageNet, and four reinforcement learning applications, the authors show that the proposed structure can achieve highly comparable performance wrt the corresponding specially designed state-of-the-art methods. The experiment results seem solid and the proposed structure is with simple design and highly generalizable. The concern is that the contribution is quite incremental from the theoretical side though it involves large amount of experimental efforts, which could be impactful. Please see the major comment below.
One major comment:
- Despite that the work is more application oriented, the paper would have been stronger and more impactful if it includes more work on the theoretical side.
Specifically, for two folds:
(1) in general, some more work in investigating the task space would be nice. The paper assumes the tasks are “related” or “similar” and thus transferrable; also particularly in Section 2, the authors define that the tasks follow the same distribution. But what exactly should the distribution be like to be learnable and how to quantify such “related” or “similar” relationship across tasks?
(2) in particular, for each of the experiments that the authors conduct, it would be nice to investigate some more on when the proposed TC + Attention network would work better and thus should be used by the community; some questions to answer include: when should we prefer the proposed combination of TC + attention blocks over the other methods? The result from the paper seems to answer with “in all cases” but then that always brings the issue of “overfitting” or parameter tuning issue. I believe the paper would have been much stronger if either of the two above are further investigated.
More detailed comments:
- On Page 1, “the optimal strategy for an arbitrary range of tasks” lacks definition of “range”; also, in the setting in this paper, these tasks should share “similarity” or follow the same “distribution” and thus such “arbitrariness” is actually constrained.
- On Page 2, the notation and formulation for the meta-learning could be more mathematically rigid; the distribution over tasks is not defined. It is understandable that the authors try to make the paradigm very generalizable; but the ambiguity or the abstraction over the “task distribution” is too large to be meaningful. One suggestion would be to split into two sections, one for supervised learning and one for reinforcement learning; but both share the same design paradigm, which is generalizable.
- For results in Table 1 and Table 2, how are the confidence intervals computed? Is it over multiple runs or within the same run? It would be nice to make clear; in addition, I personally prefer either reporting raw standard deviations or conduct hypothesis testing with specified tests. The confidence intervals may not be clear without elaboration; such is also concerning in the caption for Table 3 about claiming “not statistically-significantly different” because no significance test is reported.
- At last, some more details in implementation would be nice (package availability, run time analysis); I suppose the package or the source code would be publicly available afterwards?


Distributed training of deep learning is widely conducted with large neural networks and large datasets. Besides asynchronous stochastic gradient descent (SGD), synchronous SGD is a reasonable alternative with better convergence guarantees. However, synchronous SGD suffers from stragglers. To make things worse, although there are some strategies dealing with slow workers, the issue of slow servers is commonly ignored. In this paper, we propose a new parameter server (PS) framework dealing with not only slow workers, but also slow servers by weakening the synchronization criterion. The empirical results show good performance when there are stragglers.

This paper introduces a parameter server architecture to improve distributed training of CNNs in the presence of stragglers. Specifically, the paper proposes partial pulling where a worker only waits for first b blocks rather than all the blocks of the parameters. This technique is combined with existing methods such as partial pushing (Pan et. al. 2017) for a partial synchronous SGD method. The method is evaluated with Resnet -50 using synthetic delays.
Comments for the author:
The paper is well-written and easy to follow. The problem of synchronization costs being addressed is important but it is unclear how much of this is arising due to large blocks.
1) The partial pushing method (Pan et. al. 2017, section 3.1) shows a clear evidence for the problem using a real workload with a large number of workers. Unfortunately, in your Figure 2, this is not as obvious and not real since it is using simulated delays. More specifically, it is not clear how the workers behave in a real environment and whether you get a clear benefit from using a partial number of blocks as opposed to sending all of them.
2) Did you modify your code to support block-wise sending of gradients (some description of how the framework was modified will be helpful)? The idea is to send partial parameter blocks and when 'b' blocks are received, compute the gradients. I feel that, with such a design, you may actually end up hurting the performance by sending a large number of small packets in the no failure case. For real, large data centers, this may cause a packet storm and subsequent throughput collapse (e.g. the incast problem). You need to show the evidence that you do not hurt the failure-free case for a large number of workers.
3) The evaluation is on fairly small workloads (CIFAR-10). Again, evaluating over Imagenet and demonstrating a clear speedup over existing sync methods will be helpful. Furthermore, a clear description of your “pull” configuration (such as in Figure 1) i.e. how many actual bytes or blocks are sent and what is the threshold will be helpful (beyond a vague 90%).
4) Another concern with partial synchronization methods that I have is that how do you pick these configurations (pull 0.75 etc). These appear to be dataset specific and finding the optimal configuration here requires significant experimentation that takes significantly more time than just running the baseline.
Overall, I feel there is not enough evidence for the problem specifically generating large blocks of gradients and this needs to be clearly shown. To propose a solution for stragglers, evaluation should be done in a datacenter environment with the presence of stragglers (and not small workloads with synthetic delays). Furthermore, the proposed technique despite the simplicity appears as a rather incremental contribution.


In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

*Summary*
The paper applies variational inference (VI) with the 'reparameterisation' trick for Bayesian recurrent neural networks (BRNNs). The paper first considers the "Bayes by Backprop" approach of Blundell et al. (2015) and then modifies the BRNN model with a hierarchical prior over the network parameters, which then requires a hierarchical variational approximation with a simple linear recognition model. Several experiments demonstrate the quality of the prediction and the uncertainty over dropout.
*Originality + significance*
To my knowledge, there is no other previous work on VI with the reparameterisation trick for BRNNs. However, one could say that this paper is, on careful examination, an application of reparameterisation gradient VI for a specific application.
Nevertheless, the parameterisation of the conditional variational distribution q(\theta | \phi, (x, y)) using recognition model is interesting and could be useful in other models. However, this has not been tested or concretely shown in this paper. The idea of modifying the model by introducing variables to obtain a looser bound which can accommodate a richer variational family is also not new, see: hierarchical variational model (Ranganath et al., 2016) for example.
*Clarity*
The paper is, in general, well-written. However, the presentation in 4 is hard to follow. I would prefer if appendix A3 was moved up front -- in this case, it would make it clear that the model is modified to contain \phi, a variational approximation over both \theta and \phi is needed, and a q that couples \theta, \phi and and the gradient of the log likelihood term wrt \phi is chosen.
Additional comments:
Why is the variational approximation called "sharpened"?
At test time, normal VI just uses the fixed q(\theta) after training. It's not clear to me how prediction is done when using 'posterior sharpening' -- how is q(\theta | \phi, x) in eqs. 19-20 parameterised? The first paragraph of page 5 uses q(\theta | \phi, (x, y)), but y is not known at test time.
What is C in eq. 9?
This comment "variational typically underestimate the uncertainty in the posterior...whereas expectation propagation methods are mode averaging and so tend to overestimate uncertainty..." is not precise. EP can do mode averaging as well as mode seeking, depending on the underlying and approximate factor graphs. In the Bayesian neural network setting when the likelihood is factorised point-wise and there is one factor for each likelihood, EP is just as mode-seeking as variational. On the other hand, variational methods can avoid modes too, see the mixture of Gaussians example in the "Two problems with variational EM... " paper by Turner and Sahani (2010).
There are also many hyperparameters that need to be chosen -- what would happen if these are optimised using the free-energy? Was there any KL reweighting scheduling as done in the original BBB paper?
What is the significance of the difference between BBB and BBB with sharpening in the language modelling task? Was sharpening used in the image caption generation task?
What is the computational complexity of BBB with posterior sharpening? Twice that BBB? If this is the case, would BBB get to the same performance if we optimise it for longer? Would be interesting to see the time/accuracy frontier.


We propose a method for semi-supervised semantic segmentation using the adversarial network. While most existing discriminators are trained to classify input images as real or fake on the image level, we design a discriminator in a fully convolutional manner to differentiate the predicted probability maps from the ground truth segmentation distribution with the consideration of the spatial resolution. We show that the proposed discriminator can be used to improve the performance on semantic segmentation by coupling the adversarial loss with the standard cross entropy loss on the segmentation network. In addition, the fully convolutional discriminator enables the semi-supervised learning through discovering the trustworthy regions in prediction results of unlabeled images, providing additional supervisory signals. In contrast to existing methods that utilize weakly-labeled images, our method leverages unlabeled images without any annotation to enhance the segmentation model. Experimental results on both the PASCAL VOC 2012 dataset and the Cityscapes dataset demonstrate the effectiveness of our algorithm.

This paper describes techniques for training semantic segmentation networks. There are two key ideas:
- Attach a pixel-level GAN loss to the output semantic segmentation map. That is, add a discriminator network that decides whether each pixel in the label map belongs to a real label map or not. Of course, this loss alone is unaware of the input image and would drive the network to produce plausible label maps that have no relation to the input image. An additional cross-entropy loss (the standard semantic segmentation loss) is used to tie the network to the input and the ground-truth label map, when available.
- Additional unlabeled data is utilized by using a trained semantic segmentation network to produce a label map with associated confidences; high-confidence pixels are used as ground-truth labels and are fed back to the network as training data.
The paper is fine and the work is competently done, but the experimental results never quite come together. The technical development isn’t surprising and doesn’t have much to teach researchers working in the area. Given that the technical novelty is rather light and the experimental benefits are not quite there, I cannot recommend the paper for publication in a first-tier conference.
Some more detailed comments:
1. The GAN and the semi-supervised training scheme appear to be largely independent. The GAN can be applied without any unlabeled data, for example. The paper generally appears to present two largely independent ideas. This is fine, except they don’t convincingly pan out in experiments.
2. The biggest issue is that the experimental results do not convincingly indicate that the presented ideas are useful.
2a. In the “Full” condition, the presented approach does not come close to the performance of the DeepLab baseline, even though the DeepLab network is used in the presented approach. Perhaps the authors have taken out some components of the DeepLab scheme for these experiments, such as multi-scale processing, but the question then is “Why?”. These components are not illegal, they are not cheating, they are not overly complex and are widely used. If the authors cannot demonstrate an improvement with these components, their ideas are unlikely to be adopted in state-of-the-art semantic systems, which do use these components and are doing fine.
2b. In the 1/8, 1/4, and 1/2 conditions, the performance of the baselines is not quoted. This is wrong. Since the authors are evaluating on the validation sets, there is no reason not to train the baselines on the same amount of labeled data (1/8, 1/4, 1/2) and report the results. The training scripts are widely available and such training of baselines for controlled experiments is commonly done in the literature. The reviewer is left to suspect, with no evidence given to the contrary, that the presented approach does not outperform the DeepLab baseline even in the reduced-data conditions.
A somewhat unflattering view of the work would be that this is another example of throwing a GAN at everything to see if it sticks. In this case, the experiments do not indicate that it did.


The resemblance between the methods used in studying quantum-many body physics and in machine learning has drawn considerable attention. In particular, tensor networks (TNs) and deep learning architectures bear striking similarities to the extent that TNs can be used for machine learning. Previous results used one-dimensional TNs in image recognition, showing limited scalability and a request of high bond dimension. In this work, we train two-dimensional hierarchical TNs to solve image recognition problems, using a training algorithm derived from the multipartite entanglement renormalization ansatz (MERA). This approach overcomes scalability issues and implies novel mathematical connections among quantum many-body physics, quantum information theory, and machine learning. While keeping the TN unitary in the training phase, TN states can be defined, which optimally encodes each class of the images into a quantum many-body state. We study the quantum features of the TN states, including quantum entanglement and fidelity. We suggest these quantities could be novel properties that characterize the image classes, as well as the machine learning tasks. Our work could be further applied to identifying possible quantum properties of certain artificial intelligence methods.

Full disclosure: the authors' submission is not anonymous. They included a github link at the bottom of page 6 and I am aware of the name of the author and coauthors (and have previously read their work and am a fan of it). Thus, this review is not double blind. I notified the area chair last week and we agreed that I submit this review.
---
This is an interesting application of tensor networks to machine learning. The work proposes using a tree tensor network for image classification. Each image is first mapped into a higher-dimensional space. Then the input features are contracted with the tensors of the tensor network. The maximum value of the final layer of the network gives the predicted class. The training algorithm is inspired by the multipartite entanglement renormalization ansatz: it corresponds to updating each tensor in the network by performing a singular value decomposition of the environment tensor (everything in the cost function after removing the current tensor to be updated).
Overall, I think this is an interesting, novel contribution, but it is not accessible to non-physicists right now. The paper could be rewritten to be accessible to non-physicists and would be a highly-valuable interdisciplinary contribution.
* Consider redoing the experiments with a different cost function: least squares is an unnatural cost function to use for classification. Cross entropy would be better.
* discuss the scalability: why did you downsample MNIST from 28x28 pixels to 16x16 pixels? Why is training accuracy not reported on the 10-class model in Table 1? If it is because of a slow implementation, that's fine. But if it is because of the scalability of the method, it would be good to report that. In either case it wouldn't hurt the paper, it is just important to know.
* In section 5, you say "input vectors are still initially arranged ... according to their spatial locations in the image". But don't you change the spatial locations of the image to follow equation (10)? It would be good to add a sentence clarifying this.
---
In its current form, reading the paper requires a physics background.
There are a few things that would make it easier to read for a general machine learning audience:
* connect your method to matrix factorization and tensor decomposition approaches
* include an algorithm box for Strategy-I and Strategy-II
* include an appendix, with a brief review of upward and downward indices which is crucial for understanding your method (few people in machine learning are familiar with Einstein notation)
* relate your interesting ideas about quantum states to existing work in information theory. I am skeptical of the label 'quantum': how do quantum mechanical tools apply to images? What is a 'quantum' many-body state here? There is no intrinsic uncertainty principle at play in image classification. I would guess that the ideas you propose are equivalent to existing work in information theory. That would make it less confusing.
* in general, maybe mention the inspiration of your work from MERA, but avoid using physics language when there are no clear physical systems. This will make your work more understandable and easier to follow. A high-level motivation for MERA from a physics perspective suffices; the rest can be phrased in terms of tensor decompositions.
---
Minor nits:
* replace \citet with \citep everywhere - all citations are malformed
* figure 1 could be clarified - say that see-through gray dots are dimensions, blue squares are tensors, edges are contractions
* all figure x and y labels and legends are too small
* some typos: "which classify an input image by choosing"; "we apply different feature map to each"; small grammar issues in many places
* Figure 4: "up-down" and "left-right" not defined anywhere


Published as a conference paper at ICLR 2018 TWIN NETWORKS: MATCHING THE FUTURE FOR SEQUENCE GENERATION
We propose a simple technique for encouraging generative RNNs to plan ahead. We train a "backward" recurrent network to generate a given sequence in reverse order, and we encourage states of the forward model to predict cotemporal states of the backward model. The backward network is used only during training, and plays no role during sampling or inference. We hypothesize that our approach eases modeling of long-term dependencies by implicitly forcing the forward states to hold information about the longer-term future (as contained in the backward states). We show empirically that our approach achieves 9% relative improvement for a speech recognition task, and achieves significant improvement on a COCO caption generation task.

1) Summary
This paper proposes a recurrent neural network (RNN) training formulation for encouraging RNN the hidden representations to contain information useful for predicting future timesteps reliably. The authors propose to train a forward and backward RNN in parallel. The forward RNN predicts forward in time and the backward RNN predicts backwards in time. While the forward RNN is trained to predict the next timestep, its hidden representation is forced to be similar to the representation of the backward RNN in the same optimization step. In experiments, it is shown that the proposed method improves training speed in terms of number of training iterations, achieves 0.8 CIDEr points improvement over baselines using the proposed training, and also achieves improved performance for the task of speech recognition.
2) Pros:
+ Novel idea that makes sense for learning a more robust representation for predicting the future and prevent only local temporal correlations learned.
+ Informative analysis for clearly identifying the strengths of the proposed method and where it is failing to perform as expected.
+ Improved performance in speech recognition task.
+ The idea is clearly explained and well motivated.
3) Cons:
Image captioning experiment:
In the experimental section, there is an image captioning result in which the proposed method is used on top of two baselines. This experiment shows improvement over such baselines, however, the performance is still worse compared against baselines such as Lu et al, 2017 and Yao et al, 2016. It would be optimal if the authors can use their training method on such baselines and show improved performance, or explain why this cannot be done.
Unconditioned generation experiments:
In these experiments, sequential pixel-by-pixel MNIST generation is performed in which the proposed method did not help. Because of this, two conditioned set ups are performed: 1) 25% of pixels are given before generation, and 2) 75% of pixels are given before generation. The proposed method performs similar to the baseline in the 25% case, and better than the baseline in the 75% case. For completeness, and to come to a stronger conclusion on how much uncertainty really affects the proposed method, this experiment needs a case in which 50% of the pixels are given. Observing 25% of the pixels gives almost no information about the identity of the digit and it makes sense that it’s hard to encode the future, however, 50% of the pixels give a good idea of what the digit identity is. If the authors believe that the 50% case is not necessary, please feel free to explain why.
Additional comments:
The method is shown to converge faster compared to the baselines, however, it is possible that the baseline may finish training faster (the authors do acknowledge the additional computation needed in the backward RNN).
It would be informative for the research community to see the relationship of training time (how long it takes in hours) versus how fast it learns (iterations taken to learn).
Experiments on RL planning tasks would be interesting to see (Maybe on a simple/predictable environment).
4) Conclusion
The paper proposes a method for training RNN architectures to better model the future in its internal state supervised by another RNN modeling the future in reverse. Correctly modeling the future is very important for tasks that require making decisions of what to do in the future based on what we predict from the past. The proposed method presents a possible way of better modeling the future, however, some the results do not clearly back up the claim yet. The given score will improve if the authors are able to address the stated issues.
POST REBUTTAL RESPONSE:
The authors have addressed the comments on the MNIST experiments and show better results, however, as far as I can see, they did not address my concern about the comparisons on the image captioning experiment. In the image captioning experiment the authors choose two networks (Show & Tell and Soft attention) that they improve using the proposed method that end up performing similar to the second best baseline (Yao et al. 2016) based on Table 3 and their response. I requested for the authors to use their method on the best performing baselines (i.e. Yao et al. 2016 or Liu et al. 2017) or explain why this cannot be done (maybe my request was not clearly stated). Applying the proposed method on the strong baselines would highlight the author's claims more strongly than just applying on the average performing chosen baselines. This request was not addressed and instead the authors just improved the average performing baselines in Table 3 to meet the best baselines. Given, that the authors were able to improve the results in the sequential MNIST and improve the average baselines, my rating improves one point. However, I still have concerns about this method not being shown to improve the best methods presented in Table 3 which would give a more solid result. My rating changes to marginally above threshold for acceptance.


Can we build models that automatically learn about object motion from raw, unlabeled videos? In this paper, we study the problem of multi-step video prediction, where the goal is to predict a sequence of future frames conditioned on a short context. We focus specifically on two aspects of video prediction: accurately modeling object motion, and producing naturalistic image predictions. Our model is based on a flow-based generator network with a discriminator used to improve prediction quality. The implicit flow in the generator can be examined to determine its accuracy, and the predicted images can be evaluated for image quality. We argue that these two metrics are critical for understanding whether the model has effectively learned object motion, and propose a novel evaluation benchmark based on ground truth object flow. Our network achieves state-of-the-art results in terms of both the realism of the predicted images, as determined by human judges, and the accuracy of the predicted flow. Videos and full results can be viewed on the supplementary website: https://sites.google.com/site/omvideoprediction.

This is a fine paper that generally reads as a new episode in a series on motion-based video prediction with an eye towards robotic manipulation [Finn et al. 2016, Finn and Levine 2017, Ebert et al. 2017]. The work is rather incremental but is competently executed. It is in line with current trends in the research community and is a good fit for ICLR. The paper is well-written, reasonably scholarly, and contains stimulating insights.
I recommend acceptance, despite some reservations. My chief criticism is a matter of research style: instead of this deluge of barely distinguishable least-publishable-unit papers on the same topic, in every single conference, I wish the authors didn’t slice so thinly, devoted more time to each paper, and served up a more substantial dish.
Some more detailed comments:
- The argument for evaluating visual realism never quite gels and is not convincing. The paper advocates two primary metrics: accuracy of the predicted motion and perceptual realism of the synthesized images. The argument for motion accuracy is clear and is clearly stated: it’s the measure that is actually tied to the intended application, which is using action-conditional motion prediction for control. A corresponding argument for perceptual realism is missing. Indeed, a skeptical reviewer may suspect that the authors needed to add perceptual realism to the evaluation because that’s the only thing that justifies the adversarial loss. The adversarial loss is presented as the central conceptual contribution of the paper, but doesn’t actually make a difference in terms of task-relevant metrics. A skeptical perspective on the paper is that the adversarial loss just makes the images look prettier but makes no difference in terms of task performance (control). This is an informative negative result. It's not how the paper is written, though.
- The “no adversary”/“no adv” condition in Table 1 and Figure 4 is misleading. It’s not properly controlled. It is not the case that the adversarial loss was simply removed. The regression loss was also changed from l_1 to l_2. This is not right. The motivation for this control is to evaluate the impact of the adversarial loss, which is presented as the key conceptual contribution of the paper. It should be a proper control. The other loss should remain what it is in the full “Ours” condition (i.e., l_1).
- The last sentence in the caption of Table 1 -- “Slight improvement in motion is observed by training with an adversary as well” -- should be removed. The improvement is in the noise.
- Generally, the quantitative impact of the adversarial loss never comes together. The only statistically significant improvement is on perceptual image realism. The relevance of perceptual image realism to the intended task (control) is not substantiated, as discussed earlier.
- In the perceptual evaluation procedure, the “1 second” restriction is artificial and makes the evaluated methods appear better than they are. If we are serious about evaluating image realism and working towards passing the visual Turing test, we should report results without an artificial time limit. They won’t look as flattering, but will properly report our progress on this journey. If desired, the results of timed comparisons can also be reported, but reporting just a timed comparison with an artificial limit of 1 second may mislead some readers into thinking that we are farther along than we actually are.
There are some broken sentences that mar an otherwise well-written paper:
- End of Section 1, “producing use a learned discriminator and show improvements in visual quality”
- Beginning of Section 3, “We first present the our overall network architecture”
- page 4, “to choose to copy pixels from the previous frame, used transformed versions of the previous frame”
- page 4, “convolving in the input image with”
- page 5, “is know to produce”
- page 5, “an additional indicating”
- page 5, “Adam Kingma & Ba (2015)” (use the other cite command)
- page 5, “we observes”
- page 5, “smaller batch sizes degrades”
- page 5, “larger batch sizes provides”


We introduce an unsupervised structure learning algorithm for deep, feed-forward, neural networks. We propose a new interpretation for depth and inter-layer connectivity where a hierarchy of independencies in the input distribution is encoded in the network structure. This results in structures allowing neurons to connect to neurons in any deeper layer skipping intermediate layers. Moreover, neurons in deeper layers encode low-order (small condition sets) independencies and have a wide scope of the input, whereas neurons in the first layers encode higher-order (larger condition sets) independencies and have a narrower scope. Thus, the depth of the network is automatically determined-equal to the maximal order of independence in the input distribution, which is the recursion-depth of the algorithm. The proposed algorithm constructs two main graphical models: 1) a generative latent graph (a deep belief network) learned from data and 2) a deep discriminative graph constructed from the generative latent graph. We prove that conditional dependencies between the nodes in the learned generative latent graph are preserved in the class-conditional discriminative graph. Finally, a deep neural network structure is constructed based on the discriminative graph. We demonstrate on image classification benchmarks that the algorithm replaces the deepest layers (convolutional and dense layers) of common convolutional networks, achieving high classification accuracy, while constructing significantly smaller structures. The proposed structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU.

The paper proposes an unsupervised structure learning method for deep neural networks. It first constructs a fully visible DAG by learning from data, and decomposes variables into autonomous sets. Then latent variables are introduced and stochastic inverse is generated. Later a deep neural network structure is constructed based on the discriminative graph. Both the problem considered in the paper and the proposed method look interesting. The resulting structure seems nice.
However, the reviewer indeed finds a major technical flaw in the paper. The foundation of the proposed method is on preserving the conditional dependencies in graph G. And each step mentioned in the paper, as it claims, can preserve all the conditional dependencies. However, in section 2.2, it seems that the stochastic inverse cannot. In Fig. 3(b), A and B are no longer dependent conditioned on {C,D,E} due to the v-structure induced in node H_A and H_B. Also in Fig. 3(c), if the reviewer understands correctly, the bidirectional edge between H_A and H_B is equivalent to H_A <- h -> H_B, which also induces a v-structure, blocking the dependency between A and B. Therefore, the very foundation of the proposed method is shattered. And the reviewer requests an explicit explanation of this issue.
Besides that, the reviewer also finds unfair comparisons in the experiments.
1. In section 5.1, although the authors show that the learned structure achieves 99.04%-99.07% compared with 98.4%-98.75% for fully connected layers, the comparisons are made by keeping the number of parameters similar in both cases. The comparisons are reasonable but not very convincing. Observing that the learned structures would be much sparser than the fully connected ones, it means that the number of neurons in the fully connected network is significantly smaller. Did the authors compare with fully connected network with similar number of neurons? In such case, which one is better? (Having fewer parameters is a plus, but in terms of accuracy the number of neurons really matters for fair comparison. In practice, we definitely would not use that small number of neurons in fully connected layers.)
2. In section 5.2, it is interesting to observe that using features from conv10 is better than that from last dense layer. But it is not a fair comparison with vanilla network. In vanilla VGG-16-D, there are 3 more conv layers and 3 more fully connected layers. If you find that taking features from conv10 is good for the learned structure, then maybe it will also be good by taking features from conv10 and then apply 2-3 fully-connected layers directly (The proposed structure learning is not comparable to convolutional layers, and what it should really compare to is fully-connected layers.) In such case, which one is better?
Secondly, VGG-16 is a large network designed for ImageNet data. For small dataset such as CIFAR10 and CIFAR100, it is really overkilled. That's maybe the reason why taking the output of shallow layers could achieve pretty good results.
3. In Fig. 6, again, comparing the learned structure with fully-connected network by keeping parameters to be similar and resulting in large difference of the number of neurons is unfair from my point of view.
Furthermore, all the comparisons are made with respect to fully-connected network or vanilla CNNs. No other structure learning methods are compared with. Reasonable baseline methods should be included.
In conclusion, due to the above issues both in method and experiments, the reviewer thinks that this paper is not ready for publication.