TWO-TIMESCALE NETWORKS FOR NONLINEAR VALUE FUNCTION APPROXIMATION
A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control. Many of the algorithms for learning values, however, are designed for linear function approximation-with a fixed basis or fixed representation. Though there have been a few sound extensions to nonlinear function approximation, such as nonlinear gradient temporal difference learning, these methods have largely not been adopted, eschewed in favour of simpler but not sound methods like temporal difference learning and Q-learning. In this work, we provide a two-timescale network (TTN) architecture that enables linear methods to be used to learn values, with a nonlinear representation learned at a slower timescale. The approach facilitates the use of algorithms developed for the linear setting, such as data-efficient least-squares methods, eligibility traces and the myriad of recently developed linear policy evaluation algorithms, to provide nonlinear value estimates. We prove convergence for TTNs, with particular care given to ensure convergence of the fast linear component under potentially dependent features provided by the learned representation. We empirically demonstrate the benefits of TTNs, compared to other nonlinear value function approximation algorithms, both for policy evaluation and control.

The paper introduces an algorithm (TTN) for non-linear online and on-policy value function approximation. The main novelty of the paper is to view non-linear value estimation as two separate components. One of representation learning from a non-linear mapping and one of linear value function estimation. The soundness of the approach stems from the rate at which each component is updated. The authors argue that if the non-linear component is updated at a slower rate than the linear component, the former can be viewed as fixed in the limit and what remains is a linear value function estimation problem for which several sound algorithms exist. TTN is evaluated on 4 domains and compared to several other value estimation methods as well as DQN on a control problem with two variations on the task's state space.
I'll start off the review by stating that I find the idea and theoretical justification of separating the non-linear and linear parts of value function estimation to be quite interesting, potentially impacting RL at large. Indeed, this view promises to reconcile latest developments in deep RL with the long-lasting work on RL with linear function approximators. However, there are a few unclear aspects that do not allow one to be fully convinced that this paper lives up to the aforementioned promise.
- For the theoretical contribution. The authors claim that the main challenge was to deal with the potentially dependent features outputted by the neural network. It is dealt with by using a projection that projects the linear parameters of the value function to a compact subset of the parameter space. Bar the appendix, there is no mention of this projection in the paper, on how this compact subset (that must include the optimal parameter) is defined and if this projection is merely a theoretical tool or if it was necessary to implement it in practice. There is a projection for the neural net weights too but I can see how for these it might not be necessary to use in practice. However, for the linear weights, as their computation potentially involves inverting ill-conditioned matrices, they can indeed blow-up relatively fast.
- I found the experimental validation to be quite rich but not done in a systematic enough manner. For instance, the experiment "utility of optimizing the MSPBE" demonstrates quite nicely the importance of each component but is only performed on a single task. As the theoretical analysis does not say anything about the improvements the representation learning can have on the linear value estimation nor if the loss used for learning the representation effectively yields better features for the MSPBE minimization, this experiment is rather important and should have been performed on more than a single domain.
Secondly, I do not find the chosen baselines to be sufficiently competitive. The authors state in Sec. 2 that nonlinear-GTD has not seen widespread use, but having this algorithm as the main competitor does not provide strong evidence that TTN will know a better fate. In the abstract, it is implied that outside of nonlinear-GTD, value function approximation methods are not sound. In approximate policy iteration algorithms such as DDPG or TRPO, there is a need in performing value estimation. It is done by essentially a fitted-Q iteration procedure which is sound. Why wasn't TTN compared to these methods? If it is because they are not online, why being online in the experiments of the paper is important? Showing that TTN is competitive with currently widespread methods for value estimated would have been more convincing than the comparison with nonlinear-GTD.
Thirdly, for the sake of reproducibility, as LSTD seems to be the method of choice for learning the linear part, it would have been adequate to provide an algorithm box for this version as is done for GTD2/TDC. LSTD is essentially a batch algorithm and there could be many ways to turn it into an online algorithm. With which algorithm were the results in the experimental section obtained?
Finally, on the control task, the authors add several modifications to their algorithm which results in an algorithm that is very close to that of Levine et al., 2017. Why was not the latter a baseline for this experiment? Especially since it was included in other experiments.


Unsupervised domain adaptation (uDA) models focus on pairwise adaptation settings where there is a single, labeled, source and a single target domain. However, in many real-world settings one seeks to adapt to multiple, but somewhat similar, target domains. Applying pairwise adaptation approaches to this setting may be suboptimal, as they fail to leverage shared information among multiple domains. In this work, we propose an information theoretic approach for domain adaptation in the novel context of multiple target domains with unlabeled instances and one source domain with labeled instances. Our model aims to find a shared latent space common to all domains, while simultaneously accounting for the remaining private, domain-specific factors. Disentanglement of shared and private information is accomplished using a unified information-theoretic approach, which also serves to establish a stronger link between the latent representations and the observed data. The resulting model, accompanied by an efficient optimization algorithm, allows simultaneous adaptation from a single source to multiple target domains. We test our approach on three challenging publicly-available datasets, showing that it outperforms several popular domain adaptation methods.

In this paper, the authors proposed a new domain adaptation setting for adaptation between single source but multiple target domains. To address the setting, the authors proposed a so-called information theoretic approach to disentangle shared and private features and meanwhile to take advantage of the relationship between multiple target domains.
Pros:
- This paper conducts comprehensive empirical studies.
Cons:
- The motivation for this new domain adaptation setting is not clear to me. In the real world, the domain adaptation between multiple source domains and single target domain is in desperate need, as like human beings an agent may gradually encounter many source domains which could altogether benefit a target domain. However, I do not think that the adaptation between single source and multiple targets is intuitively in need.
- The proposed framework is quite similar to DSN, which limits this work's novelty. Though the authors take a large paragraph to illustrate the connections and differences between this work and DSN, I cannot be convinced. Especially during empirical study, the comparison is not fair. The adapted mp-DSN models multiple encoders for multiple target domains, while it is correct to extend DSN with a shared encoder for multiple target domains just like MDTA-ITA.
- There are technical flaws. The authors claimed that this work is different from ELBO optimisation, but follows an information theoretical approach. Actually, the right way to optimise the proposed loss in Eqn. (1)(2) is exactly the ELBO. Instead, the authors replace the probability/distribution q(x|z) and q(d|z) with concrete terms, which is technically wrong. such concrete term ||x-F(z;\phi)|| cannot represent a probability/distribution.
- In the experiments for feature visualisation, I do not think such comparison with original features makes any sense. The features by DSN which also separates private from shared features  should be compared.
- The presentation is in a quite poor quality, including many typos/grammatical errors.
   - The most annoying is the inappropriate citations. Every citation should be included in a brace, e.g. "the same underlying distribution Sun et al. (2016)" -> "the same underlying distribution (Sun et al. (2016))". Please kindly refer to other submissions to ICLR 2019.
   -  Typos: in the beginning of Section 2, "without loss of generalizability" -> "without loss of generality"; in the end of Page 3,  the last equation is not right, where p(y|z_s) should be q(y|z_s).
   - The font in Table 2 is too small to read.


Graph convolutional networks (GCNs) have been widely used for classifying graph nodes in the semi-supervised setting. Previous work have shown that GCNs are vulnerable to the perturbation on adjacency and feature matrices of existing nodes. However, it is unrealistic to change existing nodes in many applications, such as existing users in social networks. In this paper, we design algorithms to attack GCNs by adding fake nodes. A greedy algorithm is proposed to generate adjacency and feature matrices of fake nodes, aiming to minimize the classification accuracy on the existing nodes. In additional, we introduce a discriminator to classify fake nodes from real nodes, and propose a Greedy-GAN attack to simultaneously update the discriminator and the attacker, to make fake nodes indistinguishable to the real ones. Our non-targeted attack decreases the accuracy of GCN down to 0.10, and our targeted attack reaches a success rate of 99% on the whole datasets, and 94% on average for attacking a single target node.

The authors propose a new adversarial technique to add “fake” nodes to fool a GCN-based classifier. The basic approach relies on a greedy heuristic to add edge/node features, and the authors also present a GAN-based approach, which allows the model to add “fake” nodes that are not easily distinguishable from regular nodes. The primary motivation behind the idea of adding “fake” is that it is unrealistic to change the features/edges of existing nodes. Experimental results show that adding a large number (20% in most cases) of fake nodes can significantly degrade accuracy of a GCN, and results show that the GAN-based approach is somewhat effective at making the “fake” nodes less distinguishable.  In terms of strengths, the GAN-based approach is well-motivated and it appears that the authors were thorough in their experiments on Cora/Citseer (e.g., with a number of ablation/sensitivity studies).
However, while interesting, this paper has a number of areas where it could be substantially improved:
1) With regards to the motivation: It is not clear what substantive technical novelty there is in the idea of “adding fake nodes”, compared to existing approaches that simply modify existing nodes in an adversarial way. Intuitively, the approach of Zugner et al can already handle this case of "adding new nodes". One just adds a set of nodes with random/null edges/features to the graph, treats this as their “attacker node” set and then runs Zugner et al's greedy algorithm. Some clarification on why this simple application of Zugner et al's approach does not work would be useful and/or empirical results using their method as a baseline would be useful. (Also, Zugner et al was published in KDD 2018, so the citation should be corrected).
2) In Zugner et al, they derive approximations and algorithms that allow them to compute the score of adding/removing an edge in constant time. The greedy approach in this work appears quite expensive as every greedy update requires an expensive gradient computation. Some discussion of computational complexity would improve the paper.
3) Results are only provided on two small datasets (presumably due to the large computational cost for the approach). These two very small datasets are not indicative of many real-world scenarios, and additional results on larger (and more diverse) datasets would greatly strengthen the paper.
4) Adding 20% fake nodes seems like a prohibitively large number. Even 5% fake nodes is extremely large. It is unclear what real-world applications could admit such drastic numbers of fake nodes, and some comments on this would greatly strengthen the paper.
5) The GAN method is interesting and well-motivated, but it is not clear if this method offers any utility beyond the “distribution matching” approach of Zugner et al (Section 4.1 of their paper). A comparison between these methods is necessary to justify the utility of the proposed GAN-greedy approach.


Neural approaches to sequence labeling often use a Conditional Random Field (CRF) to model their output dependencies, while Recurrent Neural Networks (RNN) are used for the same purpose in other tasks. We set out to establish RNNs as an attractive alternative to CRFs for sequence labeling. To do so, we address one of the RNN's most prominent shortcomings, the fact that it is not exposed to its own errors with the maximum-likelihood training. We frame the prediction of the output sequence as a sequential decision-making process, where we train the network with an adjusted actor-critic algorithm (AC-RNN). We comprehensively compare this strategy with maximum-likelihood training for both RNNs and CRFs on three structured-output tasks. The proposed AC-RNN efficiently matches the performance of the CRF on NER and CCG tagging, and outperforms it on Machine Transliteration. We also show that our training strategy is significantly better than other techniques for addressing RNN's exposure bias, such as Scheduled Sampling, and Self-Critical policy training.

I found the paper difficult to follow. The method proposed is not well motivated, and  the literature review explains well the novelty. Here are some questions/points for discussion:
- the token-level MLE training is not what causes the exposure bias: one can train with MLE and still avoid it by generating appropriate sequences using the RNN, as in scheduled sampling. The problem with MLE (or cross entropy) is that the labels to be predicted might not be the correct ones. See the paper by Ranzato et al. (ICLR2016) for a good discussion of the issue: https://arxiv.org/pdf/1511.06732.pdf
- The criticism against previous works for not comparing agains CRFs seems odd: CRFs are given the number of labels, words, etc. to predict, typically the same as the number of words to be tagged. If one  has this, as well as binary rewards for each decision, then there is little benefit for RL/IL based approaches to be used. The point for them is the use of non-decomposable loss functions such as BLEU, which are not common in tagging, but in tasks like MT, where CRFs can't be used. In fact, for the transliteration experiments in the paper, the CRF approach is padded to perform the task, which highlights that it is not the right comparison.
- the approach proposed seems very similar to MIXER, which also learns a regressor to predict the reward for each action. A direct comparison both in terms of how the approaches operate and empirically is needed.
- why is it a problem that previous works by Ranzato, Bahdanau and Paulus combine MLE and RL? You are using the same supervision, ie. the labeled corpus.
- the adjusted training seems to essentially not reward correct predictions (top branch in the equation). Why is this a good idea?
- In figure 1 it is not clear at all that the proposed approach works; depending on the epoch the ranking among the three variants differs
- what does it mean for one method to surpass the other in flexibilty? If anything the requirement for immediate rewards after every action restricts flexibility, as one can't use non-decomposable loss functions such as BLEU which are prety common in NLP.
- How is the training efficiency measured in the paper?
- Why not compare against MIXER, as well as more recent work by Leblonde et al. (2018): https://arxiv.org/abs/1706.04499 ? I don't see why the Rennie et al. 2017 method is picked for comparison.
- It is not true that in IL one needs a gold standard policy, one can learn with sub-optimal policies, see Sun et al. (2018): https://arxiv.org/pdf/1703.01030.pdf
- It is odd to say that an approach proposed earlier (Dagger) reduces to a variant of a later proposed one (Scheduled sampling), the reduction should be the other way around
- are the randomly initialized character embeddings for transliteration tuned during training?
- How were the alignments for training the CRF obtained?


In this work, we developed a new representation of the chemical information for the machine learning models, with benefits from both the real space (R-space) and energy space (K-space). Different from the previous symmetric matrix presentations, the charge transfer channel based on Pauling's electronegativity is derived from the dependence on real space distance and orbitals for the heteroatomic structures. This representation can work for the bulk materials as well as the low dimensional nanomaterials, and can map the R-space and K-space into the pixel space (P-space) by training and testing 130k structures. P-space can well reproduce the R-space quantities within error 0.53. This new asymmetric matrix representation doubles the information storage than the previous symmetric representations. This work provides a new dimension for the computational chemistry towards the machine learning architecture.

This paper proposes an architecture for learning representations on molecules. The paper contains a number of typos, and presents results and an entire paragraph that is a near duplicate copy from prior work (which is not cited). In particular Table 1 is a near copy of Table 2 appearing in [1].
I find it particularly suspicious that the authors have a near copy of this prior table, which reports the ratio of MAE to chemical accuracy, and not MAE directly. The authors have the exact same numbers as this prior Table 2 but claim they are reporting MAE directly. Despite this table being a direct copy, it conveniently omits the columns from [1] which outperform the results presented here.
Also the paragraph describing this table is nearly identical to the paragraph in [1], that is the authors write
"These baselines include 5 different hand engineered molecular representations, which then get fed through a standard, off-the-shelf classifier. These input representations include the Coulomb Matrix (CM, Neese (2003)), BoB, Bonds Angles, Machine Learning (BAML, Huang & von Lilienfeld (2016)), Extended Connectivity Fingerprints (ECPF4, Rogers & Hahn (2010)), and Projected Histograms (HDAD, Faber et al. (2017a)) representations. In addition to these hand engineered features we include the Molecular Graph Convolutions model (GC, Kearnes et al. (2016)), the original GG-NN model(Li et al., 2015) trained with distance bins and DTNN."
In [1] it was written
"These baselines include 5 different hand engineered molecular representations, which then get fed through a standard, off-the-shelf classifier. These input representations include the Coulomb Matrix (CM, Rupp et al. (2012)), Bag of Bonds (BoB, Hansen et al. (2015)), Bonds Angles, Machine Learning (BAML, Huang & von Lilienfeld (2016)), Extended Connectivity Fingerprints (ECPF4, Rogers & Hahn (2010)), and “Projected Histograms” (HDAD, Faber et al. (2017)) representations. In addition to these hand engineered features we include two existing baseline MPNNs, the Molecular Graph Convolutions model (GC) from Kearnes et al. (2016), and the original GG-NN model Li et al. (2016) trained with distance bins."
The footnote in the table is also a duplicate copy of the footnote in [1].
Theirs: "As reported in DTNN. The model was trained on a different train/test split with 100k training samples vs
about 110k used in our experiments."
[1]: "As reported in Schutt et al. ¨ (2017). The model was trained on a different train/test split with 100k training samples vs 110k used in our experiments."
The proposed method itself is a variant of the MPNN framework introduced in [1], yet underperforms the original results in [1], as well as improved MPNNs (or GNNs) shown in [2,3].
1. https://arxiv.org/pdf/1704.01212.pdf
2. http://papers.nips.cc/paper/6700-schnet-a-continuous-filter-convolutional-neural-network-for-modeling-quantum-interactions
3. https://arxiv.org/pdf/1806.03146.pdf


Neural Architecture Search has recently shown potential to automate the design of Neural Networks. The use of Neural Network agents trained with Reinforcement Learning can offer the possibility to learn complex patterns, as well as the ability to explore a vast and compositional search space. On the other hand, evolutionary algorithms offer the greediness and sample efficiency needed for such an application, as each sample requires a considerable amount of resources. We propose a class of Evolutionary-Neural hybrid agents (Evo-NAS), that retain the best qualities of the two approaches. We show that the Evo-NAS agent can outperform both Neural and Evolutionary agents, both on a synthetic task, and on architecture search for a suite of text classification datasets.

Summary:
The paper proposes a hybrid approach which combines evolution and RL. The key idea is to conduct tournament selection over a population of architectures with learned mutations. The mutations are defined as the output of an RNN controller which either reuses or alters the sequence descriptor of the parent at each step. The proposed hybrid architect is evaluated on both synthetic and text classification tasks and then compared against pure evolutionary and RL-based agents.
Pros:
* The method can be viewed as a generalization of conventional evolution by replacing the handcrafted (uniform) distribution of mutations with a learned one. On the one hand, this should hopefully improve the sample efficiency of pure genetic methods since the population can evolve towards more meaningful directions, assuming useful patterns can be learned by the mutation controller. On the other hand, mutating existing architectures seem a easier task than sampling the entire architecture from scratch.
* The synthetic experiment is interesting, though it's hard to draw any conclusions based a single task.
Cons:
* To my knowledge, all text classification tasks used in 5.2 are quite small. There is no evidence that the method can scale to and work well on large-scale tasks, where improving the sample efficiency becomes truly crucial and challenging.
* It is good to see comparisons against pure evo and RL within the authors' own search space. However, the advantage of the proposed evo-NAS, especially when evaluated on real-world text classification tasks, does not seem significant enough. In particular, there is a clear overlap between the performance of architectures found by NAS, evo and evo-NAS (Figure 4). The advantage of evo-NAS is even smaller if we compare the very best model (as can be read from Figure 4) instead of the average among the top 10 (as reported in Table 2). In my option, performance of the strongest model is arguably more interesting than the averaged one in practice.
* Since no results on CIFAR or ImageNet are provided as in most prior works in the literature, it is impossible to empirically compare the method with the state-of-the-art. The experiments would be more convincing if a comparison can be provided on those benchmarks. Otherwise, it is possible that the current search space & hyperparameters are tailored towards evo-NAS and it remains unclear whether the method can generalize well to other domains and/or search spaces.


BIAS-REDUCED UNCERTAINTY ESTIMATION FOR DEEP NEURAL CLASSIFIERS
We consider the problem of uncertainty estimation in the context of (non-Bayesian) deep neural classification. In this context, all known methods are based on extracting uncertainty signals from a trained network optimized to solve the classification problem at hand. We demonstrate that such techniques tend to introduce biased estimates for instances whose predictions are supposed to be highly confident. We argue that this deficiency is an artifact of the dynamics of training with SGD-like optimizers, and it has some properties similar to overfitting. Based on this observation, we develop an uncertainty estimation algorithm that selectively estimates the uncertainty of highly confident points, using earlier snapshots of the trained model, before their estimates are jittered (and way before they are ready for actual classification). We present extensive experiments indicating that the proposed algorithm provides uncertainty estimates that are consistently better than all known methods.

-- Paper Summary --
The proposed methodology draws on the connection between boosting in ensemble learning and SGD for training DNNs, whereby misclassified instances are implicitly targeted in later training iterations once easier examples have been classified correctly. The authors observe that this incurs a trade-off in which easily-classified examples become susceptible to overfitting at later stages in the training procedure when the network parameters adapt to fit more complex examples. Two early stopping algorithms are proposed in order to mitigate this issue. The first approach, PES, is more robust, but too computationally expensive to be applied in practice; on the other hand, AES approximates the former procedure by directly assuming that easier training examples will be learnt earlier on in the training procedure. The proposed technique is shown to calibrate the confidence scores obtained from state-of-the-art approaches for training deep nets, resulting in substantial performance improvements with respect to the proposed E-AURC metric.
-- General Commentary --
- The paper isolates itself from other post-calibration methods by stating that ‘our focus here is only on the core task of ranking uncertainties’. In doing so, there is no comparison to other calibration methods, which makes it difficult to properly assess the impact of this work in comparison to other papers addressing the poor calibration of uncertainty typically associated with deep nets. The authors immediately dismiss PES as being too computationally expensive, so I’d be interested in at least seeing AES be compared to more lightweight calibration methods.
 - This paper champions the use of an alternative metric (E-AURC) for assessing model quality, which is the sole quantity of interest in the experimental evaluation. While the E-AURC metric is indeed well-motivated in Section 3, I could see there being some scepticism as to why more traditional metrics such as log likelihood aren’t used here. This would also facilitate comparison to other post-calibration methods. In this regard, the authors should consider supplementing their experiments with more widely-used metrics not limited to uncertainty ranking.
- I would be interested in seeing the analysis shown in Figure 2 extended to each of the baseline models discussed in the paper. Such examples would give a clearer perspective of which methods are particularly susceptible to the overfitting problem targeted by the methodology proposed in this work.
- Some of the notation in the problem statement is a bit confusing, with i being simultaneously  used as the training iteration number as well as an index for Y. This needs to be updated.

- There’s a lot of whitespace in Figure 1 which could be avoided by giving additional examples of how the metric works.
- ‘Early Stopping without a Validation Set (Mahsereci et al, 2017)’ warrants a citation here.
- The paper is otherwise generally well-written and a pleasure to read. Some spotted typos:
P1: for highly confident instance(s)
P3: which borrows element(s)
P3: ‘unit-less’ : this is unhyphenated in another part of the text
P5: Final reference to Figure 2(b) should refer to Figure 2(c) instead
P7: which (is) initialized
-- Recommendation --
I admit to feeling fairly ambivalent about this paper - on one hand, the paper is well-written and its contributions are effectively communicated. While myopic, the experiments also convincingly showcase the performance improvements obtained by applying AES over the baseline methods. On the downside, this paper limits itself to comparing the proposed approaches to baseline methods where no other calibration is carried out. Lack of direct comparison against other post-calibration methods results in the paper adding little to the overall literature on DNNs other than asserting that calibration through early stopping is better than not doing anything else.
Pros/Cons:
+ Properly-motivated contributions and well-written paper.
+ The two early stopping algorithms are explained well, even if the appealing connection to boosting gets lost somewhere along the way.
+ Results show that AES improves the results of several DNN training approaches.
- Use of E-AURC as the sole metric for assessing quality in the Experiments section exposes this paper to instant criticism.
- The notion of preserving model snapshots can be problematic when training requires thousands of epochs.
- No comparison to other post-calibration techniques.
** Post-rebuttal
Score increased to a 7 following rebuttal and paper revision.


One form of characterizing the expressiveness of a piecewise linear neural network is by the number of linear regions, or pieces, of the function modeled. We have observed substantial progress in this topic through lower and upper bounds on the maximum number of linear regions and a counting procedure. However, these bounds only account for the dimensions of the network and the exact counting may take a prohibitive amount of time, therefore making it infeasible to benchmark the expressiveness of networks. In this work, we approximate the number of linear regions of specific rectifier networks with an algorithm for probabilistic lower bounds of mixed-integer linear sets. In addition, we present a tighter upper bound that leverages network coefficients. We test both on trained networks. The algorithm for probabilistic lower bounds is several orders of magnitude faster than exact counting and the values reach similar orders of magnitude, hence making our approach a viable method to compare the expressiveness of such networks. The refined upper bound is particularly stronger on networks with narrow layers.

Summary:
This paper builds off of previous work that has studied the counting of linear regions in deep neural networks. The function learned by a deep neural network with piecewise linear activations (such as Relus) is itself piecewise linear on the input, and a measure of expressiveness of the network has been to count the number of linear regions.
However counting linear regions in a typical neural network is usually intractable, and there have been a sequence of upper and lower bounds proposed. Upper bounds are based on counting hyperplane arrangements (Zaslavsky, 1975; Raghu 2017; Montufar 2017; Serra 2018), and lower bounds based on counting regions in specific networks.
This paper improves the upper bound proposed in Serra (2018) by improving on a dimensionality constraint: the upper bound can be tightened if the dimensionality of the ambient space is shown to be smaller than the maximum possible value (number of neurons.) The paper defines A_l(k) -- the number of active neurons in layer l given k active neurons in layer l-1, and I_l(k) similar for inactive neurons, and proves an improved upper bound.
For the lower bound, the paper extends the existing MBound algorithm to probabilistically count the number of linear regions, with experiments (Figure 1) demonstrating the speed of this lower bound algorithm compared to counting.
Clarity: The presentation for this paper is relatively clear, but it is quite technical, so some parts are hard to follow, without knowing the prior work in detail.
Originality: Defining A_l(k) and I_l(k) for a refined upper bound, as well as the idea of using a probabilistic lower bound is new compared to prior work.
Comments on Quality and Significance:
The theoretical results presented in this paper are interesting and novel, both the bounds and the adaptation of existing methods (Nemhauser 1978; Gomes 2006) for purposes of estimating bounds. However, I'm uncertain as to the practical applications. One thing that was unclear to me was what Proposition 3, 4 mean for the quantities A_l(k) and I_l(k) in practice (in trained networks). The text makes a comment on the weights and biases having the same number of positive/negative elements but that is likely to only be true for random networks.  It would be interesting to see Figure 1 left for random and trained networks.
Given the long line of work in this area however, I think this paper will be interesting to the community.


We present a formal measure-theoretical theory of neural networks (NN) built on probability coupling theory. Particularly, we present an algorithm framework, Hierarchical Measure Group and Approximate System (HMGAS), nicknamed SSystem, of which NNs are special cases. In addition to many other results, the framework enables us to prove that 1) NNs implement renormalization group (RG) using information geometry, which points out that the large scale property to renormalize is dual Bregman divergence and completes the analog between NNs and RG; 2) and under a set of realistic boundedness and diversity conditions, for large size nonlinear deep NNs with a class of losses, including the hinge loss, all local minima are global minima with zero loss errors, using random matrix theory.

The paper provides a new framework "S-System" as a generalization of hierarchal models including neural networks. The paper shows an alternative way to derive the activation functions commonly used in practice in a principled way. It further shows that the landscape of the optimization problem of neural networks has nice properties in the setting where the number of input/hidden units tending to infinity and the neurons satisfy certain diversity conditions.
Overall, the paper presents super interesting ideas that can potentially lead to a deeper understanding of the fundamentals of deep learning. However, for a general reader it is a hard-to-follow paper. Without a full understanding of the various domains this paper presents ideas from, it is hard to verify and fully understand the claims. I believe the paper would be better appreciated by an audience of a mathematical journal. As an alternative, I would encourage the readers to split the paper and possibly simplify the content by using a running example (more concrete than the one of MLP used) to explain the implications as well as assumptions.
A clearer, more accessible presentation is necessary so that a non-expert can understand the paper's results. Thus, I vote to reject.


IMAGENET-TRAINED CNNS ARE BIASED TOWARDS TEXTURE; INCREASING SHAPE BIAS IMPROVES ACCURACY AND ROBUSTNESS
Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNettrained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on 'StylizedImageNet', a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.
Figure 1: Classification of a standard ResNet-50 of (a) a texture image (elephant skin: only texture cues); (b) a normal image of a cat (with both shape and texture cues), and (c) an image with a texture-shape cue conflict, generated by style transfer between the first two images.

Review of ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.
In this submission, the authors provide evidence through clever image manipulations and psychophysical experiments that CNNs image recognition is strongly influenced by texture identification as opposed the global object shape (as opposed to humans). The authors attempt to address this problem by using image stylization to augment the training data. The resulting networks appear much more aligned with human judgements and less biased towards image textures.
If the authors address my major concerns, I would increasing my rating 1-2 points.
Major Comments:
The results of this paper are quite compelling and address some underlying challenges in the literature on how CNN's function. I particularly appreciated Figure 5 demonstrating how the resulting stylized-augmented networks more closely align with human judgements. Additionally, it is surprising to me how poor BagNet performs on Stylized-ImageNet (SIN) implying that ResNet-50 trained on Stylized ImageNet may be better perceptually aligned with global object structure. Very cool.
1. Please make sure to tone down the claims in your manuscript. Although I share enthusiasm for your results, please recognize that stating that your results are 'conclusive' is premature and not appropriate. (Conclusive requires more papers and much work by the larger scientific community for a hypothesis to become readily accepted). Some sentences of concern include:
  --> "These experiments provide conclusive behavioural evidence in favour of the texture hypothesis"
  --> "we conclude the following: Textures, not object shapes, are the most important cues for CNN object recognition."
I would prefer to see language such as "We provide evidence that textures provide a more powerful statistical signal then global object shape for CNNs." or "We provide evidence that CNNs are overly sensitive to textures in comparison to humans perceptual judgements". This would be more measured and better reflect what has been accomplished in this study. Please do a thorough read of the rest of your manuscript and identify other text accordingly.
2. Domain shifts and data augmentation. I agree with your comment that domain shifts present the largest confound to Figure 2. The results of Geirhos et al, 2018 (Figure 4) indicate that individual image augmentations/distortions do not generalize well. Given these results, I would like to understand what image distortions were used in training each and all of your networks. Did you try a baseline with no image distortions (and/or just Stylized-ImageNet)?
Although the robustness in Figure 6 are great, how much of this can be attributed solely to Stylized-ImageNet versus the other types of image distortions/augmentations in each network. For instance, would contrast-insensitivity in Stylized-ImageNet diminish substantially if no contrast image distortion were used during training?
3. Semantics of 'object shape'. I suspect that others in the field of computer vision may take issue with your definition of 'object shape'. Please provide a crisp definition of what you test for as 'object shape' in each of your experiments (i.e. "the convex outline of object segmentation", etc.).
Minor Comments:
- Writing style in introduction. Rather then quoting phrases from individual papers, I would rather see you summarize their ideas in your own language and cite accordingly. This would demonstrate how you regard for their ideas and how these ideas fit together.
- Figure 2. Are people forced to select a choice or could they select 'I don't know'? Did you monitor response times to see if the manipulated images required longer times for individuals to pass decisions? I would expect that for some of the image manipulations that humans would have less confidence about their choices and that to be reflected in this study above and beyond an accuracy score.
- In your human studies, please provide some discussion about how you monitored performance to guard against human fatigue or lack of interest.
- Why did you use AdaIN instead of the original Gatys et al optimization method for image stylization? Was there some requirement/need for fast image stylization?
- Do you have any comment on the large variations in the results across class labels in Figure 4? Are there any easy explanations for this variation across class labels?
- Please use names of Shape-ResNet, etc. in Table 2.
- Are Pascal-VOC mAP results with fixed image features or did you fine-tune (back-propagate the errors to update the image features) during training? The latter would be particularly interesting as this would indicate that the resulting network features are better generic features as opposed to having used better data augmentation techniques.
- A.2. "not not used in the experiment" --> "not used in the experiment"


Domain transfer is an exciting and challenging branch of machine learning because models must learn to smoothly transfer between domains, preserving local variations and capturing many aspects of variation without labels. However, most successful applications to date require the two domains to be closely related (e.g., image-to-image, video-video), utilizing similar or shared networks to transform domain-specific properties like texture, coloring, and line shapes. Here, we demonstrate that it is possible to transfer across modalities (e.g., image-to-audio) by first abstracting the data with latent generative models and then learning transformations between latent spaces. We find that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion, and even between different types of models (e.g., variational autoencoder and a generative adversarial network). We can further impose desired semantic alignment of attributes with a linear classifier in the shared latent space. Through qualitative and quantitative evaluations, we demonstrate that the proposed model preserves both locality and semantic alignment through the transfer process. Finally, the hierarchical structure decouples the cost of training the base generative models and semantic alignments, enabling computationally efficient and data efficient retraining of personalized mapping functions.

In this paper, the authors have proposed a cross domain transferring methods, supervised by three category of losses. The experiments somewhat demonstrate the effective of this method. However, this paper still suffers from some drawbacks as below:
The paper is not well-organized, the structure of the paper need improving. For example, the related work is put almost at the end of the paper and the tables and figures are hard to follow sometimes.
The technical implementation of the proposition is somewhat trivial. Why the generative model should be pre-trained. Why not try in the end-to-end way.
The experiments are not convincing. The authors argue that CycleGAN suffers from some drawback. Why do not the authors compare with CycleGAN in this paper? By the way, the authors also need to compare with more state-of-the-art methods, such as StarGAN.
Some implementation details are not clearly stated. For example, the authors say “Our goal can thus be stated as learning transformations that preserve locality and semantic alignment, while requiring as few labels from a user as possible.” So, how many labeled samples are used in Table 2?


PRACTICAL LOSSLESS COMPRESSION WITH LATENT VARIABLES USING BITS BACK CODING
Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present 'Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational autoencoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly amenable to parallelization, we conclude that with a sufficiently high quality generative model this scheme could be used to achieve substantial improvements in compression rate with acceptable running time. We make our implementation available open source at https://github.com/bits-back/bits-back.

This paper is built on a simple but profound observation: Frey's bits-back coding algorithm can be implemented much more elegantly when replacing arithmetic coding (AC) with asymmetric numerical systems (ANS), a much more recent development not known at the time, simply due to the fact that it encodes symbols in a stack-like fashion rather than queue-like.
This simple observation makes for an elegantly written paper, with promising results on MNIST. I truly enjoyed reading it, and I'm convinced that it will spark some very interesting further work in the field of compression with latent-variable models.
Having said that, I would like to point out some possible limitations of the proposed approach, which I hope the authors will be able to address/clarify:
1. At the beginning of section 2.1, the authors define the symbols as chained conditionals prod_n p(s_n | s_1 ... s_n-1), which is generally permissible in AC as well as ANS, as long as the decoding order is taken into account. That is, in AC, the symbols need to be encoded starting with the first symbol in the chain (s_1), while in ANS, the symbols must be encoded starting with the last symbol in the chain, because the decoding order is inverted.
In their description of BB-ANS, the authors omit the discussion of conditional chains. It is unclear to me if a conditioning of the symbols is feasible in BB-ANS due to the necessity to maintain a strict decoding order. It would be very helpful if the authors could clarify this, and update the paper accordingly, because this could present a serious limitation. For instance, the authors simply extrapolate the performance of their method to PixelVAE; however, this model is autoregressive, so a conditioning of symbols seems necessary. Similarly, in appendix A, the authors mention the work of Minnen et al. (2018), where the same situation would apply, albeit one probabilistic level higher (on encoding/decoding the latents with an autoregressive prior).
2. Furthermore, in both cases (PixelVAE and Minnen et al.), the symbols (s) and latents (y) are defined as jointly conditioned on each other (i.e., computing the posterior on one element of y requires knowledge of all elements of s, and computing the likelihood on one element of s requires knowledge of all elements of y). This seems to imply that all operations pertaining to one data vector (i.e. to one image) would have to be done in a monolithic fashion, i.e.: first sample all elements of y from the stack, then encode all elements of s, and then encode all elements of y. Hence, if the goal is to compress only one image, the algorithm would never get to the point of reusing the "bits back", and the overhead of BB-ANS would be prohibitive. It seems that in the MNIST experiments, the authors avoid this problem by always encoding a large number of images at a time, such that the overhead is amortized.
3. Similarly, although the compression of continuous-valued variables up to arbitrary precision is an exciting development and I do not wish to undermine the importance of this finding, it should be noted that the finer the quantization gets, the larger the potential overhead of the coding scheme will grow. In practice, this would make it necessary to encode more and more images together, in order to still benefit from the method. This would be a good point to make in the discussion.
4. The authors state in the appendix that learned compression methods like Ballé et al. (2018) and Minnen et al. (2018) could be improved by using BB-ANS. The potential gain of BB-ANS for these models seems rather small, though, as the entropy of y must be larger or equal to the entropy of y conditioned on s: H[y] >= H[y|s], the latter of which should represent the potential coding gain. Ballé et al. (2018), however, found that the bits used to encode the hierarchical prior (i.e. H[y]) is only a small fraction of the total bitrate, thus upper bounding the potential gains for this type of model.
Overall, I think this is a well-written, important and elegant paper, and I would like to see it accepted at this conference. If the authors can satisfactorily address some of the above potential limitations, it might turn out to be even better.


THREE MECHANISMS OF WEIGHT DECAY REGULARIZATION
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of L 2 regularization. Literal weight decay has been shown to outperform L 2 regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the inputoutput Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

This paper discusses the effect of weight decay on the training of deep network models with and without batch normalization and when using first/second order optimization methods.
First, it is discussed how weight decay affects the learning dynamics in networks with batch normalization when trained with SGD. The dominant generalization benefit due to weight decay comes from increasing the effective learning rate of parameters on which batch normalization is applied. The authors therefore hypothesize that a larger learning rate has a regularization effect.
Second, the role of weight decay is discussed when training with second order methods without batch normalization. Under the approximation of not differentiating the curvature matrix used in second order method, it is shown that using weight decay is equivalent to adding to the loss an L2 regularization in the metric space of the curvature matrix considered. It is then shown that if the curvature matrix is the Gauss-Newton matrix, this L2 regularization (and hence the weight decay) is equivalent to the Frobenius norm of the input-output Jacobian when the input has a spherical Gaussian distribution. Similar arguments are made about KFAC with Gauss-Newton norm. The generalization benefit due to weight decay in this case is claimed based on the recent paper by Novak et al 2018 which empirically shows a strong correlation between input-output Jacobian norm and generalization error.
Finally, the role of weight decay is discussed for second order methods when using batch normalization. In this case it is discussed for Gauss-Newton KFAC that the benefit mostly comes from the application of weight decay on the softmax layer and the effect of weight decay on other weights cancel out due to batch normalization. A comparison between Gauss-Newton KFAC and Fischer KFAC is also made. Thus the generalization benefit is presumably attributed to the second order properties of KFAC and a smaller norm of softmax layer weights.
Comments:
The paper is technically correct and proofs look good.
I have mixed comments about this paper. I find the analysis in section 4.2 and 4.3 which discuss about the role of weight decay for second order methods (with and without batch-norm) to be novel and insightful (described above).
But on the other hand, I feel section 4.1 is more of a discussion on existing work rather than novel contribution. Most of what is said, both analytically and experimentally, is a repetition of van Laarhoven 2017, except for a few details. It would have been interesting to carefully study the effect of weight decay on the gamma parameter of batch-norm which controls the complexity of the network along with the softmax layer weights as it was left for future work in van Laarhoven 2017. But instead the authors brush it under the carpet by saying they did not find the gamma and beta parameters to have significant impact on performance, and fixed them during training.  I also find the claim of section 4.1 to be a bit mis-leading because it is claimed that weight decay applied with SGD and batch normalization only has benefits due to batch-norm dynamics, and not due to complexity control even though in Fig 2 and 4, there is a noticeable difference between training without weight decay, and training with weight decay only on last layer. Furthermore, when hypothesizing the regularization effect of large learning rate in section 4.1, a large body of literature that has studied this effect has not been cited. Examples are [1], [2], [3].
I have other concerns which mainly stem from lack of clarity in writing:
1. In the line right above remark 1, it is not clear what “assumption” refer to. I am guessing the distribution of the input being spherical Gaussian?
2. In remark 1, regarding the claim about the equivalence of L2 norm of theta under Gauss-Newton metric and the Frobenius norm of input-output Jacobian, why does f_theta need to be a linear function without any non-linearity? I think the linearity part is only needed for the KFAC result.
3. In remark 1, what does it mean by “Furthermore, if G is approximated by KFAC”? For linear f_theta, given lemma 1 and theorem 1, the claimed equivalence always holds true, no?
4. In the 1st line of last paragraph of page 6, what are the general conditions under which the connection between Gauss-Newton norm and Jacobian norm does not hold true?
5. In figure 5, how are the different points in the plots achieved? By varying hyper-parameters?
A minor suggestion: in theorem 1 (and lemma 1), instead of assuming network has no bias, it can be said that the L2 regularization term does not have bias terms. This is more reasonable because bias terms have no effect on complexity and so it is reasonable to not apply weight decay on bias.
Overall I think the paper is good *if* section 4.1 is sorted out and writing (especially in section 4.2) is improved. For these reasons, I am currently giving a score of 6, but I will increase it if my concerns are addressed.
[1] a bayesian perspective on generalization and stochastic gradient descent
[2] Train longer, generalize better: closing the generalization gap in large batch training of neural networks
[3] Three Factors Influencing Minima in SGD


WHAT DO YOU LEARN FROM CONTEXT? PROBING FOR SENTENCE STRUCTURE IN CONTEXTUALIZED WORD REPRESENTATIONS
Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

This is a nice paper that attempts to tease apart some questions about the effectiveness of contextual word embeddings (ELMo, CoVe, and the Transformer LM). The main question is about the value of context in these representations, and in particular how their ability to encode context allows them to also (implicitly) represent linguistic properties of words. What I really like about the paper is the “Edge probing” method it introduces. The idea is to probe the representations using diagnostic classifiers—something that’s already widespread practice—but to focus on the relationship between spans rather than individual words. This is really nice because it enables them to look at more than just tagging problems: the paper looks at syntactic constituency, dependencies, entity labels, and semantic role labeling. I think the combination of an interesting research question and a new method (which will probably be picked up by others working in this area) make this a strong candidate for ICLR. The paper is well-written and experimentally thorough.
Nitpick: It would be nice to see some examples of cases where the edge probe is correct, and where it isn’t.


Reinforcement learning agents learn by exploring the environment and then exploiting what they have learned. This frees the human trainers from having to know the preferred action or intrinsic value of each encountered state. The cost of this freedom is reinforcement learning is slower and more unstable than supervised learning. We explore the possibility that ensemble methods can remedy these shortcomings and do so by investigating a novel technique which harnesses the wisdom of the crowds by bagging Q-function approximator estimates. Our results show that this proposed approach improves all three tasks and reinforcement learning approaches attempted. We are able to demonstrate that this is a direct result of the increased stability of the action portion of the state-actionvalue function used by Q-learning to select actions and by policy gradient methods to train the policy.

This paper proposes the deep reinforcement learning with ensembles of Q-functions. Its main idea is updating multiple Q-functions, instead of one, with independently sampled experience replay memory, then take the action selected by the ensemble. Experimental results demonstrate that the proposed method can achieve better performance than non-ensemble one under the same training steps, and the decision space can also be stabilized.
This paper is well-written. The main ideas and claims are clearly expressed. Using ensembles of Q-function can naturally reduce the variance of decisions, so it can speed up the training procedure for certain tasks. This idea is simple and works well. The main contribution is it provides a way to reduce the number of interactions with the environment. My main concern about the paper is the time cost. Since the method requires updating multiple Q-functions, it may cost much more time for each RL time step, so I’m not sure whether the ensemble method can outperform the non-ensemble one within the same time period. This problem is important for practical usage. However, the authors didn’t show these results in the paper.
Minor things:
+The main idea is described too sketchily. I think more examples, such as in section 8.1, should be put in the main text.
+Page6 Line2, duplicated ‘the’.


TOWARDS ROBUST, LOCALLY LINEAR DEEP NET- WORKS
Deep networks realize complex mappings that are often understood by their locally linear behavior at or around points of interest. For example, we use the derivative of the mapping with respect to its inputs for sensitivity analysis, or to explain (obtain coordinate relevance for) a prediction. One key challenge is that such derivatives are themselves inherently unstable. In this paper, we propose a new learning problem to encourage deep networks to have stable derivatives over larger regions. While the problem is challenging in general, we focus on networks with piecewise linear activation functions. Our algorithm consists of an inference step that identifies a region around a point where linear approximation is provably stable, and an optimization step to expand such regions. We propose a novel relaxation to scale the algorithm to realistic models. We illustrate our method with residual and recurrent networks on image and sequence datasets.

########## Updated Review ##########
The author(s) have presented a very good rebuttal, and I am impressed. My concerns have been addressed and my confusions have been clarified. To reflect this, I am raising my points to 8. It is a good paper, job well done. I enthusiastically recommend acceptance.
################################
A key challenge that presents the deep learning community is that state-of-the-art solutions are oftentimes associated with unstable derivatives, compromising the robustness of the network. In this paper, the author(s) explore the problem of how to train a neural network with stable derivatives by expanding the linear region associated with training samples.
The author(s) studied deep networks with piecewise linear activations, which allow them to derive lower bounds on the $l_p$ margin with provably stable derivatives. In the special case of $l_2$ metric, this bound is analytic, albeit rigid and non-smooth. To avoid associated computational issues, the author(s) borrowed an idea from transductive/semi-supervised SVM (TSVM) to derive a relaxed formulation.
In general, I find this paper rather interesting and well written. However, I do have a few concerns and confusions as listed below:
Major ones:
- I would prefer some elaborations on why the relaxation proposed in Eqn (8) serves to encourage the margin of L2 ball? What's the working mechanism or heuristic behind this relaxation? This is supposedly one of the key techniques used in optimization, yet remains obscure.
- On empirical gains, the author(s) claimed that "about 30 times larger margins can be achieved by trading off 1% accuracy."  It seems that consistently yields inferior prediction accuracy. In my opinion, the author(s) failed to showcase the practical utility of their solution. A better job should be done to validate the claim `` The problem we tackle has implications to interpretability and transparency of complex models. ''
- As always, gradient-based penalties suffer from heavy computational overhead. The final objectives derived in this paper (Eqn (7) & Eqn (9)) seem no exception to this, and perhaps even worse since the gradient is taken wrt each neuron. Could the author(s) provide statistics on empirical wallclock performance? How much drain does this extra gradient penalty impose on the training efficiency?
Minor ones:
- Just to clarify, does the | - | used in Eqn (9) for |I(x,\gamma)|  denote counting measure?
- I do not see the necessity of introducing Lemma 7 in the text. Please explain.
- Lemma 8, ``... then any optimal solutions for the problem is also optimal for Eqn (4). '' Do you mean ``the following problem'' (Eqn (5))?


EIDETIC 3D LSTM: A MODEL FOR VIDEO PREDICTION AND BEYOND
Spatiotemporal predictive learning, though long considered to be a promising selfsupervised feature learning method, seldom shows its effectiveness beyond future video prediction. The reason is that it is difficult to learn good representations for both short-term frame dependency and long-term high-level relations. We present a new model, Eidetic 3D LSTM (E3D-LSTM), that integrates 3D convolutions into RNNs. The encapsulated 3D-Conv makes local perceptrons of RNNs motion-aware and enables the memory cell to store better short-term features. For long-term relations, we make the present memory state interact with its historical records via a gate-controlled self-attention module. We describe this memory transition mechanism eidetic as it is able to effectively recall the stored memories across multiple time stamps even after long periods of disturbance. We first evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance. Then we show that the E3D-LSTM network also performs well on the early activity recognition to infer what is happening or what will happen after observing only limited frames of video. This task aligns well with video prediction in modeling action intentions and tendency.

# 1. Summary
This paper presents a model for future video prediction, which integrates 3D convolutions into RNNs. The internal operations of the RNN are modified by adding historical records controlled via a gate-controlled self-attention module. The authors show that the model is effective also for other tasks such as early activity recognition.
Strengths:
* Nice extensive experimentation on video prediction and early activity recognition tasks and comparison with recent papers
* Each choice in the model definition are motivated, although some clarity is still missing (see below)
Weaknesses:
* Novelty: the proposed model is a small extension of a previous work (Wang et al., 2017)
# 2. Clarity and Motivation
In general, the paper is clear and general motivation makes sense, however some points need to be improved with further discussion and motivation:
A) Page 2 “Unlike the conventional memory transition function, it learns the size of temporal interactions. For longer sequences, this allows attending to distant states containing salient information”: This is not obvious. Can the authors add more details and motivate these two sentences? How is long-term relations are learned given Eq. 1?
B) Page 5 “These two terms are respectively designed for short-term and long-term video modeling”: How do you make sure that Recall(.) does not focus on the short-term modeling instead? Not clear why this should model long-term relations.
C) Page 5 and Eq 1: motivation why layer norm is required when defining C_t^k is not clear
D) What if the Recall is instead modeled as attention? The idea is to consider only C_{1:t-1}^k (not consider R_t) and have an attentional model that learn what to recall based only on C. Also, why does Recall need to depend on R_t?
E) Page 5 “to minimize the l1 + l2 loss over every pixel in the frame”: this sentence is not clear. How does it relate to Eq. 2?
# 3. Novelty
Novelty is the major concern of this paper. Although the introduced new concepts and ideas are interesting, the work seems to be an extension of ST-LSTM and PredRNN where Eq 1 is slightly modified by introducing Recall.
In addition the existing relation between the proposed model and ST-LSTM is not clearly state. Page 2, first paragraph: here the authors should state that model is and extension of ST-LSTM and highlight what are the difference and advantage of the new model.
# 4. Significance of the work
This paper deals with an interesting and challenging topic (video prediction) as well as it shows some results on the early activity recognition task. These are definitively nice problem which are far to be solved. From the application perspective this work is significant, however from the methodological perspective it lacks a bit of significance because of the novelty issues highlighted above.
# 5. Experimentation
The experiments are robust with nice comparisons with recent methods and ablation study motivating the different components of the model (Table 1 and 2). Some suggested improvements:
A) Page 7 “Seq 1 and Seq 2 are completely irrelevant, and ahead of them, another sub-sequence called prior context is given as the input, which is exactly the same as Seq 2”: The COPY task is a bit unclear and need to be better explained. Why are Seq. 1 and 2 irrelevant? I would suggest to rephrase this part.
B) Sec. 4.2, “Dataset and setup”: which architecture has been used here?
C) Sec. 4.3, “Hyper-parameters and Baselines“: the something-something dataset is more realising that the other two “toy” dataset. Why did the authors choose to train a 2 layers 3D-CNN encoders, instead of using existing pretrained 3D CNNs? I would suspect that the results can improve quite a bit.
# 6. Others
* The term “self-supervised auxiliary learning” is introduced in the abstract, but at this point it’s meaning is not clear. I’d suggest to either remove it or explain its meaning.
* Figure 1(a): inconsistent notation with 2b. Also add citation (Wang et al., 2017) since it ie the same model of that paper
-------
# Post-discussion
I increased my rating: even if novelty is not high, the results support the incremental ideas proposed by the authors.


This paper proposes a versatile and powerful training algorithm named Featurelevel Ensemble Effect for knowledge Distillation (FEED), which is inspired by the work of factor transfer. The factor transfer is one of the knowledge transfer methods that improves the performance of a student network with a strong teacher network. It transfers the knowledge of a teacher in the feature map level using high-capacity teacher network, and our training algorithm FEED is an extension of it. FEED aims to transfer ensemble knowledge, using either multiple teacher in parallel or multiple training sequences. Adapting peer-teaching framework, we introduce a couple of training algorithms that transfer ensemble knowledge to the student at the feature map level, both of which help the student network find more generalized solutions in the parameter space. Experimental results on CIFAR-100 and ImageNet show that our method, FEED, has clear performance enhancements, without introducing any additional parameters or computations at test time.

In summary, I think this paper contains some reasonable results based on a reasonable, moderately novel, idea, but unfortunately, it is not yet ready for publication. Reading it made me rather confused.
Good things:
- The main idea is sensible, though distilling into the same architecture (sFEED) is not that novel. I think the pFEED is probably the more novel part.
- The numerical results are quite good.
- It's a fairly simple method. If others reproduced these results, I think it would be useful.
Problems:
- Some parts of the paper are written in a way that makes the reader confused about what this paper is about. For example the first paragraph. Some motivations I just did not understand.
- Some parts of the paper are repeating itself. For example "introduction" and "related works". The section on related work also includes some quite unrelated papers.
- The references in the paper are often pointing to work that came much later than the original idea or some pretty random recent papers. For example the idea of model compression (or knowledge distillation) is much older than Hinton et al. I believe it was first proposed by Bucila et al. [1] (which the authors mention later as if knowledge distillation and model compression were very different ideas), it definitely doesn't come from Kim et al. (2018). Learning from intermediate representations of the network is at least as old as Romero et al. [2]. Compression into a network of the same architecture is definitely older than Furnarello et al. (2018). It was done, for example, by Geras et al. [3]. The paper also cites Goodfellow et al. (2016) in some pretty random contexts. I don't want to be too petty about references, but unfortunately, this paper is just below a threshold that I would still find acceptable in this respect.
- The comparison in Table 6 would make more sense if the same architectures would be clearly compared. As it is, it is difficult to be certain where the improvement is coming from and how it actually compares to different methods.
Typos: Titap X, ResNext, prarphraser.
References:
[1] Bucila et al. Model Compression. 2006.
[2] Romero et al. FitNets: Hints for Thin Deep Nets. 2014.
[3] Geras et al. Blending LSTMs into CNNs. 2016.


MEASURING AND REGULARIZING NETWORKS IN FUNCTION SPACE
To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a L 2 Hilbert space. We examine how typical networks behave in this space, and compare how parameter 2 distances compare to function L 2 distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the L 2 / 2 ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the L 2 distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through L 2 -space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature.

This paper proposes a method for functional regularization for training neural nets, such that the sequence of neural nets during training is stable in function space. Specifically, the authors define a L2 norm (i.e., a Hilbert norm), which can be used to measure distances in this space between two functions. The authors argue that this can aid in preventing catastrophic forgetting, which is demonstrated in a synthetic multi-task variant of MNIST.   The authors also show how to regularize the gradient updates to be conservative in function space in standard stochastic gradient style learning, but with rather inconclusive empirical results.  The authors also draw upon a connection to the natural gradient.
***Clarity***
The paper is reasonably well written.  I think the logical flow could be improved at places.   I think the major issue with clarity is the title.  The authors use the term "regularizing" in a fairly narrow sense, in particular regularizing the training trajectory to be stable in function space.  However, the more dominant usage for regularizing is to regularize the final learned function to some prior, which is not studied or even really discussed in the paper.
Detailed comments:
-- The notation in Section 2 could be cleaned up.  The use of \mu is a bit disconnected from the rest of the notation.
-- Computing the empirical L2 distance accurately can also be NP hard.  There's no stated guarantee of how large N needs to be to have a good empirical estimate.  Figure 3 is nice, but I think a more thorough discussion on this point could be useful.
-- L2-Space was never formally defined.
-- Section 2.1 isn't explained clearly.  For instance, in the last paragraph, the first sentence states "the networks are initialized at very different point", and halfway into the paragraph a sentence states "all three initializations begin at approximately the same point in function space.".  The upshot is that Figure 1 doesn't crisply capture the intuition the authors aim to convey.
***Originality***
Strictly speaking, the proposed formulation is novel as far as I am aware.  However, the basic idea has been the air for a while.  For instance, there are some related work in RL/IL on functional regularization:
-- https://arxiv.org/abs/1606.00968
The proposed formulation is, in some sense, the obvious thing to try (which is a good thing).  The detailed connection to the natural gradient is nice.  I do wish that the authors made stronger use of properties of a Hilbert space, as the usage of Hilbert spaces is fairly superficial.  For instance, one can apply operators in a Hilbert space, or utilize an inner product.  It just feels like there was a lost opportunity to really explore the implications.
***Significance***
This is the place where the contributions of this paper are most questionable.  While the multi-task MNIST experiments are nice in demonstrating resilience against catastrophic forgetting, the experiments are pretty synthetic.  What about a more "real" multi-task learning problem?
More broadly, it feels like this paper is suffering from a bit of an identity crisis.  It uses regularizing in a narrow sense to generate conservative updates.  It argues that this can help in catastrophic forgetting.  It also shows how to employ this to construct the standard bounded-update gradient descent rules, although without much rigorous discussion for the implications.  There are some nice empirical results on a synthetic multi-task learning task, and inconclusive results otherwise.  There's a nice little discussion on the connection to the natural gradient.  It argues that that this form of regularization lives in a Hilbert space, but the usage of a Hilbert space is fairly superficial.  All in all, there are some nice pieces of work here and there, but it's all together neither here or there in terms of an overall contribution.
***Overall Quality***
I think if the authors really pushed one of the angles to a more meaningful contribution, this paper would've been much stronger.  As it stands, the paper just feels too scattered in its focus, without a truly compelling result, either theoretically or empirically.


This paper aims at rapid deployment of the state-of-the-art deep neural networks (DNNs) to energy efficient accelerators without time-consuming fine tuning or the availability of the full datasets. Converting DNNs in full precision to limited precision is essential in taking advantage of the accelerators with reduced memory footprint and computation power. However, such a task is not trivial since it often requires the full training and validation datasets for profiling the network statistics and fine tuning the networks to recover the accuracy lost after quantization. To address these issues, we propose a simple method recognizing channel-level distribution to reduce the quantization-induced accuracy loss and minimize the required image samples for profiling. We evaluated our method on eleven networks trained on the ImageNet classification benchmark and a network trained on the Pascal VOC object detection benchmark. The results prove that the networks can be quantized into 8-bit integer precision without fine tuning.

This paper proposes a technique for channel-wise quantization of CNNs
to 8-bit, fixed point precision. The authors propose several
techniques for analyzing the statistical properties of output channel
activations in order to select the best fractional bit length for each
channel. Experimental results on eleven different CNN architectures
demonstrate that the approaches proposed result in significantly less
accuracy loss when compared to a layer-wise baseline.
The paper has the following strengths:
 1. The experimental results on eleven different architectures (of
    varying depth and breadth) are convincing, and are consistently
    better than layer-wise MAX for choosing fractional bit length.
The paper has the following weak points:
 1. There is not much coherence between the description of the
    approach in section 2.1, Figure 1, Algorithm 1, and
    Figure 2. Notation is used in Algorithm 1 which is never defined.
 2. Related to the previous point, the proposed technique has a lot of
    moving parts and I don't feel that it would be easy to reproduce
    the results of the paper. There are some vague statements, like
    "We resolve this complication by pre-coordinating the fractional
    lengths of the weights", which require significantly more
    precision. This issue -- one of the main issues with channel-wise
    versus layer-wise quantization -- is never returned to in the
    definition of the method.
 3. The experimental comparison with layer-wise quantization is
    somewhat lacking. Is layer-wise MAX the state-of-the-art in CNN
    quantization? The results comparing channel-wise and layer-wise
    MAX are already convincing, but are the moment-analysis approaches
    not equally applicable to layer-wise quantization?
    State-of-the-art results that are less sensitive to outliers
    should be included in Table 1. A comparison with layer-wise
    approaches would be nice to have also in Figure 4 to show
    sensitivity to profiling set size.
The experimental results in the paper are impressive, and the analysis
motivating the approach is convincing. However, there are presentation
and clarity issues in the technical development, and the comparative
analysis is lacking broader comparisons with the state-of-the-art (to
be fair, the authors recognize that layer-wise MAX as a baseline is
particularly susceptible to outliers). These two aspects combined,
however, lead me to the opinion that this work is just not quite ready
for publication at ICLR.


ABSTRACT An important question in task transfer learning is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems. Inspired by a principled information theoretic approach, H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature. This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efficient transfer learning policies. Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice, but also it can generalize to inference problems beyond classification, such as recognition tasks for 3D indoor-scene understanding.

The authors propose an information theoretic metric to determine a priori if a representation learned from a source task can be useful when learning another task. This is a timely topic and being able to compute such as metric between tasks upfront would be of great interest.
While the motivation and contributions of the paper are clearly stated, a number of questions remain. Central concepts like error exponent or maximal HGR correlation are mentioned in passing. The former is important to understand whether (1) makes sense and the latter is a fundamental quantity for this work. The surprising fact is that HGR is not discussed at all, nor is there any intuition provided of what this quantity represents.
While the proposed H-score is attractive when a source task has to be selected, as the denominator of the task transferability itself does not need to be computed, it is not straightforward to compute this quantity in general. The authors suggest to use the alternative conditional expectation (ACE) algorithm in order to optimize (2). ACE is not discussed in any detail (e.g., how accurate is this procedure; what are the choices and/or trade-offs if any?) and (2) is not justified. Overall, I felt section 4.2 was relatively inaccessible, but important as it indicates whether the proposed metric is of any use in practice.
Finally, I did not understand the argument that says the H-score is to be prepared over the mutual information. To me the mutual information is still the golden standard. It was also not clear why the proposed approach would not straightforwardly apply to non classification problems as suggested in the future work.


Learning rich and compact representations is an open topic in many fields such as word embedding, visual question-answering, object recognition or image retrieval. Although deep neural networks (convolutional or not) have made a major breakthrough during the last few years by providing hierarchical, semantic and abstract representations for all of these tasks, these representations are not necessary as rich as needed nor as compact as expected. Models using higher order statistics, such as bilinear pooling, provide richer representations at the cost of higher dimensional features. Factorization schemes have been proposed but without being able to reach the original compactness of first order models, or at a heavy loss in performances. This paper addresses these two points by extending factorization schemes to codebook strategies, allowing compact representations with the same dimensionality as first order representations, but with second order performances. Moreover, we extend this framework with a joint codebook and factorization scheme, granting a reduction both in terms of parameters and computation cost. This formulation leads to state-of-the-art results and compact second-order models with few additional parameters and intermediate representations with a dimension similar to that of first-order statistics.

Summary:
This paper proposes a novel bilinear representation based on a codebook model.
The proposed method is build upon the form of Hadamard Product for efficient representation of a bilinear model.
Through introducing the framework of codebook, it is naturally extended into a multiple-rank representation while the efficient pooling scheme is also derived from the (sparse) codeword assignment.
In addition, the authors also present an efficient formulation in which the codebook-based projections are factorized via a shared projection to further reduce the parameter size.
The experimental results on image retrieval tasks show that the proposed method produces better classification accuracy with a limited amount of parameters.
Comments:
Pros:
+ It is interesting that one-rank bilinear pooling is naturally extended to multiple-rank one via introducing codebooks.
+ Good performance in the image retrieval tasks.
Cons:
- This paper lacks important comparison for fairly evaluating the effectiveness of the proposed formulation.
- It also lacks detailed description and discussion for the methods.
Due to the above-mentioned weak points, the reviewer cannot fully understand whether the performance improvement really comes from the proposed formulation or not. Thus, this manuscript is currently judged as border. The detailed comments are shown in the followings.
- Comparison
Eventually, the proposed method is closely related to the multiple-rank representation of a bilinear model;
z_i = x^T W_i x (Eq.5) ~ x^T u_i v_i^T x (one-rank, Eq.6) ~ x^T U_i V_i^T x (multiple-rank), ... Eq.(A)
which is a straightforward extension from the one-rank model. From this viewpoint, the proposed form in Eq.10 is regarded as an extension of (A) by introducing non-linearity as
z_i = x^T U_i {h(x)h(x)^T} V_i^T x.  ... Eq.(10)
Thus, the main technical contribution is found in the weighting by {h(x)h(x)^T}, but its impact on the performance is not evaluated in the experiments. Furthermore, it is also possible to simply introduce such a non-linearity into the model (A) according to [Kim et al.,2017];
z_i = \sigma(x^T U_i) \sigma(V_i^T x) = 1^T {\sigma(U_i^T x) .* \sigma(V_i^T x)}, ... Eq.(B)
where ".*" indicates Hadamard Product, and we can more directly apply the method of [Kim et al., 2017] to the multiple-rank model by
z_i = p^T {\sigma(U_i^T x) .* \sigma(V_i^T x)}, ... Eq.(C)
where p is a R-dimensional vector. On the other hand, it is also necessary to compare the proposed method with [Kim et al.,2017] which is formulated by
z = P^T {\sigma(U^T x) .* \sigma(V^T x)}, ... Eq.(D)
where U and V are matrices of d x K and P is K x D. The parameter K (shared rank) should be determined so that the total parameter size of (2dK + KD) is compatible to that of the proposed method, 2NdD.
In summary, for demonstrating the effectiveness of the proposed method in Eq.(10), it is inevitable to compare it with the models (A, B, D) and hopefully (C).
- Presentation
In Section 4.2, the performance results of the factorization model in Eq.(13) are merely shown without deep discussion nor analysis on them. In particular, it is unclear why the JCF of N=32 and R=32 outperforms the CHPBP of N=32. Those two methods are different only in the form of U and V:
(CHPBP) U_i -> U'_i A (JCF),
where U_i and U'_i have the same dimensionality of d x 32, and thus we can say that JCF overly parameterizes the projection by redundantly introducing A of 32 x 32. Thus, the projection capacity of JCF is completely the same as that of CHPBP. Therefore, it needs detailed discussion for the performance improvement shown in Figure 1.
Minor comments are:
* There are lots of notations, and thus it would be better to show a summary of the notations.
* In Section 4.1, there is no clear description about the methods of BP and HPBP. Actually, the method of HPBP is different from the one presented in [Kim et al., 2017].


Generating novel molecules with optimal properties is a crucial step in many industries such as drug discovery. Recently, deep generative models have shown a promising way of performing de-novo molecular design. Although graph generative models are currently available they either have a graph size dependency in their number of parameters, limiting their use to only small graphs or are formulated as a sequence of discrete actions needed to construct a graph, making the output graph non-differentiable w.r.t the model parameters, therefore preventing them to be used in scenarios such as conditional graph generation. In this work we propose a model for conditional graph generation that directly optimises properties of the graph, and generates a probabilistic graph, making the decoding process differentiable. We demonstrate favourable performance of our model on prototype-based molecular graph conditional generation task.

In this paper, authors propose a deep generative model and a variant for graph generation and conditional graph generation respectively. It exploits an encoder which is built based on GCN and GraphSAGE, a autoregressive LSTM decoder which generates the graph embedding, and a factorized edge based probabilistic model for generating edge and node type. For conditional generation, authors also propose a discriminating training scheme based on maximizing the mutual information. Experiments on ZINC dataset show that the proposed method is promising.
Strength:
1, The problem this paper tries to tackle is very challenging and of great significance. Especially, the conditional graph generation direction under the deep learning context is novel.
2, The overall model is interesting although it is a bit complicated as it combines quite a few modules.
Weakness:
1, In the reconstruction experiment, comparisons with several recent competitive methods are missing. For example, the methods which have been already discussed in the related work, Li et al. (2018a), You et al. (2018a) and You et al. (2018b). Moreover, it is not explained whether the comparison setting is the same as Jin et al. (2018) and what the size of the latent code of their method is. It seems less convincing by just taking results from their paper and do the comparison.
2, Authors motive their work by saying in the abstract that “other graph generative models are either computationally expensive, limiting their use to only small graphs or are formulated as a sequence of discrete actions needed to construct a graph, making the output graph non-differentiable w.r.t the model parameters”. However, if I understood correctly, in Eq. (7), authors compute the soft adjacency tensor which is a dense tensor and of size #node by #node by #edge types. Therefore, I did not see why this method can scale to large graphs.
3, The overall model exploits a lot of design choices without doing any ablation study to justify. For example, how does the pre-trained discriminator affect the performance of the conditional graph generation? Why not fine-tune it along with the generator? The overall model has quite a few loss functions and associated weights of which the values are not explained at all.
4, Conditional generation part is not written clearly. Especially, the description of variational mutual information phase is so brief that I do not understand the motivation of designing such an objective function. What is the architecture of the discriminator?
5, How do authors get real attributes from the conditionally generated molecules? It is not explained in the paper.
Typos:
1, There are a few references missing (question mark) in the first and second paragraphs of section 2.
2, Methods in the experiment section are given without explicit reference, like GCPN.
3, Since edge type is introduced, I suggest authors explicitly mention the generated graphs are multi-graph in the beginning of model section.
Overall, I do not think this paper is ready for publishing and it could be improved significantly.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Update:
Thanks for the detailed explanation. The new figure 1 is indeed helpful for demonstrating the overall idea.
However, I still found some claims made by authors problematic.
For example, it reads in the abstract that "...or are formulated as a sequence of discrete actions needed to construct a graph, making the output graph non-differentiable w.r.t the model parameters...".
Clearly, Li et al. 2018b has a differentiable formulation which falls under your description.
Besides, I suggest authors adjust the experiment such that it focuses more on comparing conditional generation.
Also, please set up some reasonable baselines based on previous work rather than saying it is not directly comparable.
Directly taking numbers from other papers for a comparison is not a good idea given the fact that these experiments usually involve quite a few details which could potentially vary significantly.
Therefore, I would like to keep my original rating.