Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy. In this paper we propose a cheap pruning algorithm based on difference of convex functions (DC) optimisation. We also provide theoretical analysis for the growth in the Generalization Error (GE) of the new pruned network. Our method can be used with any convex regulariser and allows for a controlled degradation in classification accuracy, while being orders of magnitude faster than competing approaches. Experiments on common feedforward neural networks show that for sparsity levels above 90% our method achieves 10% higher classification accuracy compared to Hard Thresholding.

This paper casts the pruning optimization problem of NetTrim as a difference of convex problems, and uses DCA to obtain the smaller weight matrix; this algorithm is also analyzed theoretically to provide a bound on the generalization error of the pruned network.
However, there are many questions that aren't answered in the paper that make it difficult to evaluate: in particular, some experimental results leave open more questions for performance analysis.
Quality: of good quality, but incomplete.
Clarity: clear with some typos
Originality: a new approach to the NetTrim algorithm, which is somewhat original, and a new generalization bound for the algorithm.
Significance: somewhat significant.
PROS
- A very efficient algorithm for pruning, which can run orders of magnitude faster than the approaches that were compared to on certain architectures.
- An interesting generalization bound for the pruned network which is in line experimentally with decreasing robustness to pruning on layers close to the input.
CONS
- Non-trivial loss of accuracy on the pruned network, which cannot be estimated for larger-scale pruning as the experiments only prune one layer.
- No in-depth analysis of the generalization bound.
Main questions:
- You mention you use a variant of DCA: could you detail what differences Alg. 2 has with classical DCA?
- Where do you use the 0-1 loss in Thm. 3.2?
- I think your result in Theorem 3.2 would be significantly stronger if you could provide an analysis of the bound you obtain: in which cases can we expect certain terms to be larger or smaller, etc.
- Your experiments in section 4.2 show a non-trivial degradation of the accuracy with FeTa. Although the time savings seem worth the tradeoff to prune *one* layer, have you run the same experiments when pruning multiple layers? Could you comment on how the accuracy evolves with multiple pruned layers?
- It would be nice to see the curves for NetTrim and/or LOBS in Fig. 2.
- Have you tried retraining the network after pruning? Did you observe the same behavior as mentioned in (Dong et al., 2017) and (Wolfe et al., 2017)?
- It would be interseting to plot the theoretical (pessimistic) GE bound as well as the experimental accuracy degradation.
Nitpicks:
-Ubiquitous (first paragraph)
-difference of convex problemS
- The references should be placed before the appendix.
- The amount of white space should be reduced (e.g. around Eq. (1)).


In vanilla backpropagation (VBP), activation function matters considerably in terms of non-linearity and differentiability. Vanishing gradient has been an important problem related to the bad choice of activation function in deep learning (DL). This work shows that a differentiable activation function is not necessary any more for error backpropagation. The derivative of the activation function can be replaced by an iterative temporal differencing (ITD) using fixed random feedback weight alignment (FBA). Using FBA with ITD, we can transform the VBP into a more biologically plausible approach for learning deep neural network architectures. We don't claim that ITD works completely the same as the spike-time dependent plasticity (STDP) in our brain but this work can be a step toward the integration of STDP-based error backpropagation in deep learning.

- This paper is not well written and incomplete. There is no clear explanation of what exactly the authors want to achieve in the paper, what exactly is their approach/contribution, experimental setup, and analysis of their results.
- The paper is hard to read due to many abbreviations, e.g., the last paragraph in page 2.
- The format is inconsistent. Section 1 is numbered, but not the other sections.
- in page 2, what do the numbers mean at the end of each sentence? Probably the figures?
- in page 2, "in this figure": which figure is this referring to?
Comments on prior work:
p 1: authors write: "vanilla backpropagation (VBP)" "was proposed around 1987 Rumelhart et al. (1985)."
Not true. A main problem with the 1985 paper is that it does not cite the inventors of backpropagation. The VBP that everybody is using now is the one published by  Linnainmaa in 1970, extending Kelley's work of 1960. The first to publish the application of VBP to NNs was Werbos in 1982. Please correct.
p 1: authors write: "Almost at the same time, biologically inspired convolutional networks was also introduced as well using VBP LeCun et al. (1989)."
Here one must cite the person who really invented this biologically inspired convolutional architecture (but did not apply backprop to it): Fukushima (1979). He is cited later, but in a misleading way. Please correct.
p 1: authors write: "Deep learning (DL) was introduced as an approach to learn deep neural network architecture using VBP LeCun et al. (1989; 2015); Krizhevsky et al. (2012)."
Not true. Deep Learning was introduced by Ivakhnenko and Lapa in 1965: the first working method for learning in multilayer perceptrons of arbitrary depth. Please correct. (The term "deep learning" was introduced to ML in 1986 by Dechter for something else.)
p1: authors write: "Extremely deep networks learning reached 152 layers of representation with residual and highway networks He et al. (2016); Srivastava et al. (2015)."
Highway networks were published half a year earlier than resnets, and reached many hundreds of layers before resnets. Please correct.
General recommendation: Clear rejection for now. But perhaps the author want to resubmit this to another conference, taking into account the reviewer comments.


Published as a conference paper at ICLR 2018 TEMPORAL DIFFERENCE MODELS: MODEL-FREE DEEP RL FOR MODEL-BASED CONTROL
Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

This paper proposes a "temporal difference model learning", a method that aims to combine the benefits of model-based and model-free RL.  The proposed method essentially learns a time-varying goal-conditional value function for a specific reward formulation, which acts as a surrogate for a model in an MPC-like setting.  The authors show that the method outperforms some alternatives on three continuous control domains and real robot system.
I believe this paper to be borderline, but ultimately below the threshold for acceptance.  On the positive side, there are certainly some interesting ideas here: the notion of goal-conditioned value functions as proxies for a model, and as a means of merging model-free and model-based approaches is very really interesting, and hints at a deeper structure to goal-conditioned value functions in general.  Ultimately, though, I feel that there are two main issues that make this research feel as though it is still ultimately in the earlier stages: 1) the very large focus on the perspective that this approach is unifying model-based and model-free RL, when it fact this connection seems a bit tenuous; and 2) the rather lackluster experimental results, which show only marginal improvement over purely model-based methods (at the cost of much additional complexity), and which make me wonder if there's an issue with their implementation of prior work (namely the Highsight Experience Replay algorithm).
To address the first point, although the paper stresses it to a very high degree, I can't help but feel that the connection that the claimed advance of "unifying model-based and model-free RL" is overstated.  As far as I can tell, the connection is as follows: the learned quantity here is a time-varying goal-conditioned value function, and under some specific definition of reward, we can interpret the constraint that this value function equal zero as a proxy for the dynamics constraint in MPC.  But the exact correspondence between this and the MPC formulation only occurs for a horizon of size zero: longer horizons require a multi-step MPC for the definition of the model-free and model-based correspondence.  The fact that the action selection of a model-based method and this approach have some function which looks similar (but only under certain conditions), just seems like a fairly odd connection to highlight so heavily.
Rather, it seems to me that what's happening here is really quite simple: the authors are extending goal-conditioned value functions to the case of non-stationary finite horizon value functions (the claimed "key insight" in eq (5) is a completely standard finite-horizon MDP formulation).  This seems to describe perfectly well what is happening here, and it does also seem intuitive that this provides an advantage over stationary goal-conditioned value functions: just as goal conditioned value functions offer the advantage of considering "every state as a goal", this method can consider "every state as a goal for every time horizon".  This seems interesting enough on its own, and I admit I don't see the need for the method to be yet another claimed unification of model-free and model-based RL.
I would also suggest that the authors look into the literature on how TD methods implicitly learn models (see e.g. Boyan 1997 "Least-squares temporal difference learning", and Parr et al., 2007 "An analysis of linear models...").  In these works it has been shown that least squares TD methods (at least in the linear feature setting), implicitly learn a dynamics model in feature space, but only the "projection" of the reward function is actually needed to learn the TD weights.  In building the proposed value functions, it seems like the authors are effectively solving for multiple rewards simultaneously, which would effectively preserve the learned dynamics model.  I feel like this may be an interesting line of analysis for the paper if the authors _do_ want to stick with the notion of the method as unifying model-free and model-based RL.
All these points may ultimately just be a matter of interpretation, though, if not for the second issue with the paper, which is that the results seem quite lackluster, and the claimed performance of HER seems rather suspicious.  But instead, the authors evaluate the algorithm on just three continuous control tasks (and a real robot, which is more impressive, but the task here is still so extremely simple for a real robot system that it really just qualifies as a real-world demonstration rather than an actual application).  And in these three settings, a model-based approach seems to work just as well on two of the tasks, and may soon perform just as well after a few more episodes on the last task (it doesn't appear to have converged yet).  And despite the HER paper showing improvement over traditional policy approaches, in these experiments plain DDPG consistently performs as well or better than HER.


BI-DIRECTIONAL BLOCK SELF-ATTENTION FOR FAST AND MEMORY-EFFICIENT SEQUENCE MODELING
Recurrent neural networks (RNN), convolutional neural networks (CNN) and selfattention networks (SAN) are commonly used to produce context-aware representations. RNN can capture long-range dependency but is hard to parallelize and not time-efficient. CNN focuses on local dependency but does not perform well on some tasks. SAN can model both such dependencies via highly parallelizable computation, but memory requirement grows rapidly in line with sequence length. In this paper, we propose a model, called "bi-directional block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding. It requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN splits the entire sequence into blocks, and applies an intra-block SAN to each block for modeling local context, then applies an inter-block SAN to the outputs for all blocks to capture long-range dependency. Thus, each SAN only needs to process a short sequence, and only a small amount of memory is required. Additionally, we use feature-level attention to handle the variation of contexts around the same word, and use forward/backward masks to encode temporal order information. On nine benchmark datasets for different NLP tasks, Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better efficiency-memory trade-off than existing RNN/CNN/SAN.

Pros:
The paper proposes a “bi-directional block self-attention network (Bi-BloSAN)” for sequence encoding, which inherits the advantages of multi-head (Vaswani et al., 2017) and DiSAN (Shen et al., 2017) network but is claimed to be more memory-efficient. The paper is written clearly and is easy to follow. The source code is released for duplicability. The main originality is using block (or hierarchical) structures; i.e., the proposed models split the an entire sequence into blocks, apply an intra-block SAN to each block for modeling local context, and then apply an inter-block SAN to the output for all blocks to capture long-range dependency. The proposed model was tested on nine benchmarks  and achieve good efficiency-memory trade-off.
Cons:
- Methodology of the paper is very incremental compared with previous models.
- Many of the baselines listed in the paper are not competitive; e.g.,  for SNLI, state-of-the-art results are not included in the paper.
- The paper argues advantages of the proposed models over CNN by assuming the latter only captures local dependency, which, however, is not supported by discussion on or comparison with hierarchical CNN.
- The block splitting (as detailed in appendix) is rather arbitrary in terms of that it potentially divides coherent language segments apart. This is unnatural, e.g., compared  with alternatives such as using linguistic segments as blocks.
- The main originality of paper is the block style. However, the paper doesn’t analyze how and why the block brings improvement.
-If we remove intra-block self-attention (but only keep token-level self-attention), whether the performance will be significantly worse?


Published as a conference paper at ICLR 2018 DEEP REWIRING: TRAINING VERY SPARSE DEEP NET- WORKS
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

This paper presents an iterative approach to sparsify a network already during training. During the training process, the amount of connections in the network is guaranteed to stay under a specific threshold. This is a big advantage when training is performed on hardware with computational limitations, in comparison to "post-hoc" sparsification methods, that compress the network after training.
The method is derived by considering the "rewiring" of an (artificial) neural network as a stochastic process. This perspective is based on a recent model in computational biology but also can be interpreted as a (sequential) monte carlo sampling based stochastic gradient descent approach. References to previous work in this area are missing, e.g.
[1] de Freitas et al., Sequential Monte Carlo Methods to Train Neural Network
Models, Neural Computation 2000
[2] Welling et al., Bayesian Learning via Stochastic Gradient Langevin Dynamics, ICML 2011
Especially the stochastic gradient method in [2] is strongly related to the existing approach.
Positive aspects
- The presented approach is well grounded in the theory of stochastic processes. The authors provide proofs of convergence by showing that the iterative updates converge to a fixpoint of the stochastic process
- By keeping the temperature parameter of the stochastic process high, it can be directly applied to online transfer learning.
- The method is specifically designed for online learning with limited hardware ressources.
Negative aspects
- The presented approach is outperformed for moderate compression levels (by Han's pruning method for >5% connectivity on MNIST, Fig. 3 A, and by l1-shrinkage for >40% connectivity on CIFAR-10 and TIMIT, Fig. 3 B&C). Especially the results on MNIST suggest that this method is most advantageous for very high compression levels. However in these cases the overall classification accuracy has already dropped significantly which could limit the practical applicability.
- A detailled discussion of the relation to previously existing very similar work is missing (see above)
Technical Remarks
Fig. 1, 2 and 3 are referenced on the pages following the page containing the figure. Readibility could be slightly increased by putting the figures on the respective pages.


Graph kernels have been successfully applied to many graph classification problems. Typically, a kernel is first designed, and then an SVM classifier is trained based on the features defined implicitly by this kernel. This two-stage approach decouples data representation from learning, which is suboptimal. On the other hand, Convolutional Neural Networks (CNNs) have the capability to learn their own features directly from the raw data during training. Unfortunately, they cannot handle irregular data such as graphs. We address this challenge by using graph kernels to embed meaningful local neighborhoods of the graphs in a continuous vector space. A set of filters is then convolved with these patches, pooled, and the output is then passed to a feedforward network. With limited parameter tuning, our approach outperforms strong baselines on 7 out of 10 benchmark datasets, and reaches comparable performance elsewhere. Code and data are publicly available 1 .

This paper proposes a graph classification method by integrating three techniques, community detection, graph kernels, and CNNs.
* This paper is clearly written and easy to follow. Thus the clarity is high.
* The originality is not high as the application of neural networks for graph classification has already been studied elsewhere and the proposed method is a direct combination of three existing methods, community detection, graph kernels, and CNNs.
* The quality and the significance of this paper it not high due to the following reasons:
- The motivation is misleading in two folds.
  First, the authors say that the graph kernel + SVM approach has a drawback due to two independent processes of graph representation and learning.
  However, the parameters included in respective graph kernel is usually optimized via the SVM classification, hence they are not independent with each other.
  Second, the authors say that the proposed method addresses the above issue of independence between graph representation and learning.
  However, it also uses the two-step procedure as it first obtain the kernel matrix K via graph kernels and then apply CNN for classification, which is fundamentally the same as the existing approach.
  Although community detection is used before graph kernels, such subgraph extraction process is already implicitly employed in various graph kernels.
  I recommend to revise and clarify this point.
- In experimental evaluation, why several kernels including SP, RW, and WL are not used in the latter five datasets?
  This missing experiment significantly deteriorate the quality of empirical evaluation and I strongly recommend to add results for such kernels.
- It is mentioned that the parameter h is fixed to 5 in the WL kernel. However, it is known that the performance of the WL kernel depends on the parameter and it should be tuned by cross-validation.
  In contrast, parameters (number of epochs and the learning rate) are tuned in the proposed method. Thus the current comparison is not fair.
- In addition to the above point, how are parameters for GR and RW?
- Runtime is shown in Table 4 but there is no comparison with other methods. Although it is mentioned in the main text that the proposed method is faster than Graph CNN and Depp Graph Kernels, there is no concrete values and this statement is questionable (Runtime will easily vary due to the hardware configuration).
* Additional comment:
- Why is the community detection step needed? What will happen if K is directly constructed from given N graphs and what is the advantage of using not the original graphs but extracted subgraphs?
- In the first step of finding characteristic subgraphs, frequent subgraph mining can be used instead community detection.
  Frequent subgraph mining is extensively used in various methods for classification of graph-structured data, for example:
  * Tsuda, K., Entire regularization paths for graph data, ICML 2007.
  * Thoma, M. et al., Discriminative frequent subgraph mining with optimality guarantees, Statistical Analysis and Data Mining, 2010
  * Takigawa, I., Mamitsuka, H., Generalized Sparse Learning of Linear Models Over the Complete Subgraph Feature Set, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
  What is the advantage of using the community detection compared to frequent subgraph mining or other subgraph enumeration methods?


The rise of graph-structured data such as social networks, regulatory networks, citation graphs, and functional brain networks, in combination with resounding success of deep learning in various applications, has brought the interest in generalizing deep learning models to non-Euclidean domains. In this paper, we introduce a new spectral domain convolutional architecture for deep learning on graphs. The core ingredient of our model is a new class of parametric rational complex functions (Cayley polynomials) allowing to efficiently compute spectral filters on graphs that specialize on frequency bands of interest. Our model generates rich spectral filters that are localized in space, scales linearly with the size of the input data for sparsely-connected graphs, and can handle different constructions of Laplacian operators. Extensive experimental results show the superior performance of our approach on spectral image classification, community detection, vertex classification and matrix completion tasks.

Summary: This paper proposes a new graph-convolution architecture, based on Cayley transform of the matrix. Succinctly, if L denotes the Laplacian of a graph, this filter corresponds to an operator that is a low degree polynomial of C(L) = (hL - i)/(hL+i), where h is a scalar and i denotes sqrt(-1). The authors contend that such filters are interesting because they can 'zoom' into a part of the spectrum, depending on the choice of h, and that C(L) is always a rotation matrix with eigenvalues with magnitude 1. The authors propose to compute them using Jacobi iteration (using the diagonal as a preconditioner), and present experimental results.
Opinion: Though the Cayley filters seem to have interesting properties,  I find the authors theoretical and experimental justification insufficient to conclude that they offer sufficient advantage over existing methods. I list my major criticisms below:
1. The comparison to Chebyshev filters  (small degree polynomials in the Chebyshev basis) at several places is unconvincing. The results on CORA (Fig 5a) compare filters with the same order, though Cayley filters have twice the number of variables for the same order as Chebyshev filters. Similarly for Fig 1, order 3 Cayley should be compared to Order 6 Chebyshev (roughly).
2. Since Chebyshev polynomials blow up exponentially when applied to values larger than 1, applying Chebyshev filters to unnormalized Laplacians (Fig 5b) is an unfair comparison.
3. The authors basically apply Jacobi iteration (gradient descent using a diagonal preconditioner) to estimate the Cayley filters, and contend that a constant number of iterations of Jacobi are sufficient. This ignores the fact that their convergence rate scales quadratically in h and the max-degree of the graph. Moreover, this means that the Filter is effectively a low degree polynomial in (D^(-1)A)^K, where A is the adjacency matrix of the graph, and K is the number of Jacobi iterations. It's unclear how (or why) a choice of K might be good, or why does it make sense to throw away all powers of D^(-1)Af, even though we're computing all of them.
Also, note that this means a K-fold increase in the runtime for each evaluation of the network, compared to the Chebyshev filter.
Among the other experimental results, the synthetic results do clearly convey a significant advantage at least over Chebyshev filters with the same number of parameters. The CORA results (table 2) do convey a small but clear advantage. The MNIST result seems a tie, and the comparison for MovieLens doesn't make it obvious that the number of parameters is the same.
Overall, this leads me to conclude that the paper presents insufficient justification to conclude that Cayley filters offer a significant advantage over existing work.


Workshop track -ICLR 2018 STABLE DISTRIBUTION ALIGNMENT USING THE DUAL OF THE ADVERSARIAL DISTANCE
Methods that align distributions by minimizing an adversarial distance between them have recently achieved impressive results. However, these approaches are difficult to optimize with gradient descent and they often do not converge well without careful hyperparameter tuning and proper initialization. We investigate whether turning the adversarial min-max problem into an optimization problem by replacing the maximization part with its dual improves the quality of the resulting alignment and explore its connections to Maximum Mean Discrepancy. Our empirical results suggest that using the dual formulation for the restricted family of linear discriminators results in a more stable convergence to a desirable solution when compared with the performance of a primal min-max GAN-like objective and an MMD objective under the same restrictions. We test our hypothesis on the problem of aligning two synthetic point clouds on a plane and on a real-image domain adaptation problem on digits. In both cases, the dual formulation yields an iterative procedure that gives more stable and monotonic improvement over time.

This paper proposes to re-formulate the GAN saddle point objective (for a logistic regression discriminator) as a minimization problem by dualizing the maximum likelihood objective for regularized logistic regression (where the dual function can be obtained in closed form when the discriminator is linear). They motivate their approach by repeating the previously made claim that the naive gradient approach is non-convergent for generic saddle point problems (Figure 1); while a gradient approach often works well for a minimization formulation.
The dual problem of regularized logistic regression is an entropy-regularized concave quadratic objective problem where the Hessian is y_i y_j <x_i, x_j>, thus highlighting the pairwise similarities between the points x_i & x_j; here the labels represent whether the point x comes from the samples A from the target distribution or B from the proposal distribution. This paper then compare this objective with the MMD distance between the samples A & B. It points out that the adversarial logistic distance can be viewed as an iteratively reweighted empirical estimator of the MMD distance, an interesting analogy (but also showing the limited power of the adversarial logistic distance for getting good generating distributions, given e.g. that the MMD has been observed in the past to perform poorly for face generation [Dziugaite et al. UAI 2015]). From this analogy, one could expect the method to improves over MMD, but not necessarily significantly in comparison to an approach which would use more powerful discriminators.
This paper then investigates the behavior of this adversarial logistic distance in the context of aligning distributions for domain adaptation, with experiments on a visual adaptation task. They observe better performance for their approach in comparison to ADDA, improved WGAN and MMD, when restricting the discriminators to be a linear classifier.
== Evaluation
I found this paper quite clear to read and enjoyed reading it. The observations are interesting, despite being on the toyish side. I am not an expert on GANs for domain adaptation, and thus I can not judge of the quality of the experimental comparison for this application, but it seemed reasonable, apart for the restriction to the linear discriminators (which is required by the framework of this paper).
One concern about the paper (but this is an unfortunate common feature of most GAN papers) is that it ignores the vast knowledge on saddle point optimization coming from the optimization community. The instability of a gradient method on non-strongly convex-concave saddle point problems (like the bilinear form of Figure 1) is a well-known property, and many alternative *stable* gradient based algorithms have been proposed to solve saddle point problems which do not require transforming them to a minimization problem as suggested in this paper. Moreover, the transformation to the minimization form crucially required the closed form computation of the dual function (with w* just defined above equation (2)), and this is limited to linear discriminators,  thus ruling out the use of the proposed approach to more powerful discriminators like deep neural nets. Thus the significance appears a bit limited to me.
== Other comments
1) Note that d(A, B'_theta) is *equal* to min_alpha max_w  (...)  above equation (2) (it is not just an upper bound). This is a standard result coming from the fact that the Fenchel dual problem to regularized maximum likelihood is the maximum entropy problem with a quadratic objective as (2).  See e.g. Section 2.2 of [Collins et al. JMLR 2008] (this is for the more general multiclass logistic regression problem, but (2) is just the binary special case of equation (4) in the [Collins ... ] reference). And note that the "w(u)" defined in this reference is the lambda*w*(alpha) optimal relationship defined in this paper (but without the 1/lambda factor because of just slightly different writing; the point though is that strong duality holds there and thus one really has equality).
[Collins et al. JMLR 2008] Michael Collins, Amir Globerson, Terry Koo , Xavier Carreras, Peter L. Bartlett, Exponentiated Gradient Algorithms for Conditional Random Fields and Max-Margin Markov Networks, , JMLR 2008.
 [Dziugaite et al. UAI 2015] Gintare Karolina Dziugaite, Daniel M. Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In UAI, 2015


Published as a conference paper at ICLR 2018 VARIATIONAL IMAGE COMPRESSION WITH A SCALE HYPERPRIOR
We describe an end-to-end trainable model for image compression based on variational autoencoders. The model incorporates a hyperprior to effectively capture spatial dependencies in the latent representation. This hyperprior relates to side information, a concept universal to virtually all modern image codecs, but largely unexplored in image compression using artificial neural networks (ANNs). Unlike existing autoencoder compression methods, our model trains a complex prior jointly with the underlying autoencoder. We demonstrate that this model leads to state-of-the-art image compression when measuring visual quality using the popular MS-SSIM index, and yields rate-distortion performance surpassing published ANN-based methods when evaluated using a more traditional metric based on squared error (PSNR). Furthermore, we provide a qualitative comparison of models trained for different distortion metrics.

Authors propose a transform coding solution by extending the work in Balle 2016. They define an hyperprior for the entropy coder to model the spatial relation between the transformed coefficients.
The paper is well written, although I had trouble following some parts. The results of the proposal are state-of-the-art, and there is an extremely exhaustive comparison with many methods.
In my opinion the work has a good quality to be presented at the ICLR. However, I think it could be excellent if some parts are improved. Below I detail some parts that I think could be improved.
*** MAIN ISSUES
I have two main concerns about motivation that are related. The first refers to hyperprior motivation. It is not clear why, if GDN was proposed to eliminate statistical dependencies between pixels in the image, the main motivation is that GDN coefficients are not independent. Perhaps this confusion could be resolved by broadening the explanation in Figure 2. My second concern is that it is not clear why it is better to modify the probability distribution for the entropy encoder than to improve the GDN model. I think this is a very interesting issue, although it may be outside the scope of this work. As far as I know, there is no theoretical solution to find the right balance between the complexity of transformation and the entropy encoder. However, it would be interesting to discuss this as it is the main novelty of the work compared to other methods of image compression based on deep learning.
*** OTHER ISSUES
INTRODUCTION
-"...because our models are optimized end-to-end, they minimize the total expected code length by learning to balance the amount of side information with the expected improvement of the entropy model."
I think this point is very interesting, it would be good to see some numbers of how this happens for the results presented, and also during the training procedure. For example, a simple comparison of the number of bits in the signal and side information depending on the compression rate or the number of iterations during model training.
COMPRESSION WITH VARIATIONAL MODELS
- There is something missing in the sentence: "...such as arithmetic coding () and transmitted..."
- Fig1. To me it is not clear how to read the left hand schemes. Could it be possible to include the distributions specifically? Also it is strange that there is a \tiled{y} in both schemes but with different conditional dependencies. Another thing is that the symbol ψ appears in this figure and is not used in section 2.
- It would be easier to follow if change the symbols of the functions parameters by something like \theta_a and \theta_s.
- "Distortion is the expected difference between..." Why is the "expected" word used here?
- "...and substituting additive uniform noise..." is this phrase correct? Are authors is Balle 2016 substituting additive uniform noise?
- In equation (1), is the first term zero or constant? when talking about equation (7) authors say "Again, the first term is constant,...".
- The sentence "Most previous work assumes..." sounds strange.
- The example in Fig. 2 is extremely important to understand the motivation behind the hyperprior but I think it needs a little more explanation. This example is so important that it may need to be explained at the beginning of the work. Is this a real example, of a model trained without and with normalization? If so specify it please. Why is GDN not able to eliminate these spatial dependencies? Would these dependencies be eliminated if normalization were applied between spatial coefficients? Could you remove dependencies with more layers or different parameters in the GDN?
INTRODUCTION OF A SCALE HYPERPRIOR
- TYPO "...from the center pane of..."
- "...and propose the following extension of the model (figure 3):" there is nothing after the colon. Maybe there is something missing, or maybe it should be a dot instead of a colon. However to me there is a lack of explanation about the model.

RESULTS
- "...,the probability mass functions P_ŷi need to be constructed “on the fly”..."
How computationally costly is this?
- "...batch normalization or learning rate decay were found to have no beneficial effect (this may be due to the local normalization properties of GDN, which contain global normalization as a special case)."
This is extremely interesting. I see the connection for batch normalization, but not for decay of the learning rate. Please, clarify it. Does this mean that when using GDN instead of regular nonlinearity we no longer need to use batch normalization? Or in other words, do you think that batch normalization is useful only because it is special case of GSN? It would be useful for the community to assess what are the benefits of local normalization versus global normalization.
- "...each of these combinations with 8 different values of λ in order to cover a range of rate–distortion tradeoffs."
Would it be possible with your methods including \lambda as an input and the model parameters as side information?
- I guess you included the side information when computing the total entropy (or number of bits), was there a different way of compressing the image and the side information?
- Using the same metrics to train and to evaluate is a little bit misleading. Evaluation plots using a different perceptual metric would be helpful.
-"Since MS-SSIM yields values between 0 (worst) and 1 (best), and most of the compared methods achieve values well above 0.9, we converted the quantity to decibels in order to improve legibility."
Are differences of MS-SSIM with this conversion significant? Is this transformation necessary, I lose the intuition. Besides, probably is my fault but I have not being able to "unconvert" the dB to MS-SSIM units, for instance 20*log10(1)= 20 but most curves surpass this value.
- "..., results differ substantially depending on which distortion metric is used in the loss function during training."
It would be informative to understand how the parameters change depending on the metric employed, or at least get an intuition about which set of parameters adapt more g_a, g_s, h_a and h_s.
- Figs 5, 8 and 9. How are the curves aggregated for different images? Is it the mean for each rate value? Note that depending on how this is done it could be totally misleading.

- It would be nice to include results from other methods (like the BPG and Rippel 2017) to compare with visually.
RELATED WORK
Balle et al. already published a work including a perceptual metric in the end-to-end training procedure, which I think is one of the main contributions of this work. Please include it in related work:
"End-to-end optimization of nonlinear transform codes for perceptual quality." J. Ballé, V. Laparra, and E.P. Simoncelli. PCS: Picture Coding Symposium, (2016)
DISCUSSION
First paragraphs of discussion section look more like a second section of "related work".
I think it is more interesting if the authors discuss the relevance of putting effort into modelling hyperprior or the distribution of images (or transformation). Are these things equivalent? Or is there some reason why we can't include hyperprior modeling in the g_a transformation? For me it is not clear why we should model the distribution of outputs as, in principle, the g_a transformation has to enforce (using the training procedure) that the transformed data follow the imposed distribution. Is it because the GDN is not powerful enough to make the outputs independent? or is it because it is beneficial in compression to divide the problem into two parts?
REFERENCES
- Balle 2016 and Theis 2017 seem to be published in the same conference the same year. Using different years for the references is confusing.
- There is something strange with these references
Ballé, J, V Laparra, and E P Simoncelli (2016). “Density Modeling of Images using a Generalized
Normalization Transformation”. In: Int’l. Conf. on Learning Representations (ICLR2016). URL :
https://arxiv.org/abs/1511.06281.
Ballé, Valero Laparra, and Eero P. Simoncelli (2015). “Density Modeling of Images Using a Gen-
eralized Normalization Transformation”. In: arXiv e-prints. Published as a conference paper at
the 4th International Conference for Learning Representations, San Juan, 2016. arXiv: 1511.
06281.
– (2016). “End-to-end Optimized Image Compression”. In: arXiv e-prints. 5th Int. Conf. for Learn-
ing Representations.


Existing sequence prediction methods are mostly concerned with time-independent sequences, in which the actual time span between events is irrelevant and the distance between events is simply the difference between their order positions in the sequence. While this time-independent view of sequences is applicable for data such as natural languages, e.g., dealing with words in a sentence, it is inappropriate and inefficient for many real world events that are observed and collected at unequally spaced points of time as they naturally arise, e.g., when a person goes to a grocery store or makes a phone call. The time span between events can carry important information about the sequence dependence of human behaviors. In this work, we propose a set of methods for using time in sequence prediction. Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization. We also introduce two methods for using next event duration as regularization for training a sequence prediction model. We discuss these methods based on recurrent neural nets. We evaluate these methods as well as baseline models on five datasets that resemble a variety of sequence prediction tasks. The experiments revealed that the proposed methods offer accuracy gain over baseline models in a range of settings.

The authors present a model base on an RNN to predict marks and duration of events in a temporal point process. The main innovation of the paper is a new representation of a point process with duration (which could also be understood as marks), which allows them to use a "time mask", following the idea of word mask introduced by Choi et al, 2016. In Addition to the mask, the authors also propose a discretization of the duration using one hot encoding and using the event duration as a regularizer. They compare their method to several variations of their own method, two trivial baselines, and one state of the art method (RMTPP) using several real-world datasets and report small gains with respect to that state of the art method.
Overall, the technical contribution of the paper is minor, the gains in performance with respect to a single state of the art are minimal, and the authors oversell their contribution specially in comparison with the related literature. More specifically, my concerns, which prevent me from recommending acceptance, are as follows:
- The authors assume the point process contains duration and intervals, however, point processes generally do not have duration per event but they are discrete events localized in particular time points. Moreover, the duration in their representation (Figure 1) is sometimes an interevent time and sometimes a duration, which makes the whole construction inconsistent. Moreover, what happens then to the representation depicted in Figure 1 when duration is nonexistent or zero?
- The use of "time mask" is not properly justified and the authors are just extending the idea of word mask to their setting -- it is unclear why the duration of an event is going to provide context and in any case this seems like a minor technical contribution.
- The use of a time mask does not appear "more principled" than previous work (Due et al., Mei & Esiner, Xiao et al.). Previous work use the framework of temporal point processes in a principled way, the current work does not. I would encourage to authors to tone down their language.
- The regularization proposed by the authors uses a Gaussian on the "prediction error" of the duration or just cross entropy on a discretization of the duration. Given the inconsistency in the definition of the duration (sometimes it is duration, sometimes is interevent time), the resulting regularization may lead to unexpected/undesirable results. Moreover, it is unclear why the authors do not model the duration time with an appropriate distribution (e.g., Weibull) and add the log-likelihood of the durations under that distribution as regularization.
- The difference in performance  with respect to a single nontrivial baseline (the remaining baselines are trivial or versions of their own model) is minimal. Moreover, the authors fail to compare with other methods, e.g., the method by Mei & Eisner, which beats RMTPP. This is specially surprising since the authors mention such work in the related work and there is available source code at https://github.com/HMEIatJHU/neurawkes.


Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem into the network. But understanding how weight-sharing can be used effectively in general is a topic that has not been studied extensively. Chen et al. (2015) proposed HashedNets, which augments a multi-layer perceptron with a hash table, as a method for neural network compression. We generalize this method into a framework (ArbNets) that allows for efficient arbitrary weightsharing, and use it to study the role of weight-sharing in neural networks. We show that common neural networks can be expressed as ArbNets with different hash functions. We also present two novel hash functions, the Dirichlet hash and the Neighborhood hash, and use them to demonstrate experimentally that balanced and deterministic weight-sharing helps with the performance of a neural network.

The manuscript advocates to study the weight sharing in a more systematic way by proposing ArbNets which defines the weight sharing function as a hash function. In this framework, any existing neural network architectures, including CNN and RNN, could be incorporated into ArbNets.
The manuscript is not well written. There are multiple grammar errors and typos. Content-wise, it is already well known that CNN and RNN can be expressed as general MLP with weight sharing. The introduction of ArbNets does not bring much value or insight to this area. So it seems that most content before experimental section is common sense.
In the experimental section, it is interesting to see how different hash function with different level of entropy can affect the performance of neural nets. However, this single observation cannot enrich the whole manuscript. Two questions:
(1) What is the definition of sparsity here, and how is it controlled?
(2) There seems to be a step change in Figure 3. All the results are either between 10 to 20, or near 50. And the blue line goes up and down. Is this expected?


Published as a conference paper at ICLR 2018 UNBIASED ONLINE RECURRENT OPTIMIZATION
The novel Unbiased Online Recurrent Optimization (UORO) algorithm allows for online learning of general recurrent computational graphs such as recurrent network models. It works in a streaming fashion and avoids backtracking through past activations and inputs. UORO is computationally as costly as Truncated Backpropagation Through Time (truncated BPTT), a widespread algorithm for online learning of recurrent networks Jaeger (2002). UORO is a modification of NoBackTrack Ollivier et al. (2015) that bypasses the need for model sparsity and makes implementation easy in current deep learning frameworks, even for complex models. Like NoBackTrack, UORO provides unbiased gradient estimates; unbiasedness is the core hypothesis in stochastic gradient descent theory, without which convergence to a local optimum is not guaranteed. On the contrary, truncated BPTT does not provide this property, leading to possible divergence. On synthetic tasks where truncated BPTT is shown to diverge, UORO converges. For instance, when a parameter has a positive short-term but negative long-term influence, truncated BPTT diverges unless the truncation span is very significantly longer than the intrinsic temporal range of the interactions, while UORO performs well thanks to the unbiasedness of its gradients.
Current recurrent network learning algorithms are ill-suited to online learning via a single pass through long sequences of temporal data. Backpropagation Through Time (BPTT Jaeger (2002)), the current standard for training recurrent architectures, is well suited to many short training sequences. Treating long sequences with BPTT requires either storing all past inputs in memory and waiting for a long time between each learning step, or arbitrarily splitting the input sequence into smaller sequences, and applying BPTT to each of those short sequences, at the cost of losing long term dependencies. This paper introduces Unbiased Online Recurrent Optimization (UORO), an online and memoryless learning algorithm for recurrent architectures: UORO processes and learns from data samples sequentially, one sample at a time. Contrary to BPTT, UORO does not maintain a history of previous inputs and activations. Moreover, UORO is scalable: processing data samples with UORO comes at a similar computational and memory cost as just running the recurrent model on those data.
Like most neural network training algorithms, UORO relies on stochastic gradient optimization. The theory of stochastic gradient crucially relies on the unbiasedness of gradient estimates to provide convergence to a local optimum. To this end, in the footsteps of NoBackTrack (NBT) Ollivier et al. (2015), UORO provides provably unbiased gradient estimates, in a scalable, streaming fashion.
Unlike NBT, though, UORO can be easily implemented in a black-box fashion on top of an existing recurrent model in current machine learning software, without delving into the structure and code of the model.
The framework for recurrent optimization and UORO is introduced in Section 2. The final algorithm is reasonably simple (Alg. 1), but its derivation (Section 3) is more complex. In Section 6, UORO is shown to provide convergence on a set of synthetic experiments where truncated BPTT fails to display reliable convergence. An implementation of UORO is provided as supplementary material.
Current recurrent network learning algorithms are ill-suited to online learning via a single pass through long sequences of temporal data. Backpropagation Through Time (BPTT Jaeger (2002)), the current standard for training recurrent architectures, is well suited to many short training sequences. Treating long sequences with BPTT requires either storing all past inputs in memory and waiting for a long time between each learning step, or arbitrarily splitting the input sequence into smaller sequences, and applying BPTT to each of those short sequences, at the cost of losing long term dependencies. This paper introduces Unbiased Online Recurrent Optimization (UORO), an online and memoryless learning algorithm for recurrent architectures: UORO processes and learns from data samples sequentially, one sample at a time. Contrary to BPTT, UORO does not maintain a history of previous inputs and activations. Moreover, UORO is scalable: processing data samples with UORO comes at a similar computational and memory cost as just running the recurrent model on those data.
Like most neural network training algorithms, UORO relies on stochastic gradient optimization. The theory of stochastic gradient crucially relies on the unbiasedness of gradient estimates to provide convergence to a local optimum. To this end, in the footsteps of NoBackTrack (NBT) Ollivier et al. (2015), UORO provides provably unbiased gradient estimates, in a scalable, streaming fashion.
Unlike NBT, though, UORO can be easily implemented in a black-box fashion on top of an existing recurrent model in current machine learning software, without delving into the structure and code of the model.
The framework for recurrent optimization and UORO is introduced in Section 2. The final algorithm is reasonably simple (Alg. 1), but its derivation (Section 3) is more complex. In Section 6, UORO is shown to provide convergence on a set of synthetic experiments where truncated BPTT fails to display reliable convergence. An implementation of UORO is provided as supplementary material.

Post-rebuttal update:
I am happy with the rebuttal and therefore I will keep the score of 7.
This is a very interesting paper. Training RNN's in an online fashion (with no backpropagation through time) is one of those problems which are not well explored in the research community. And I think, this paper approaches this problem in a very principled manner. The authors proposes to use forward approach for the calculation of the gradients. The author proposes to modify RTRL by maintaining a rank one approximation of jacobian matrix (derivative of state w.r.t parameters)  which was done in NoBackTrack Paper. The way I think this paper is different from NoBackTrack Paper is that this version can be implemented in a black box fashion and hence easy to implement using current DL libraries like Pytorch.
Pros.
- Its an interesting paper, very easy to follow, and with proper literature survey.
Cons:
- The results are quite preliminary. I'll note that this is a very difficult problem.
- "The proof of UORO’s convergence to a local optimum is soon to be published Masse & Ollivier (To appear)."  I think, paper violates the anonymity.  So, I'd encourage the authors to remove this.
Some Points:
- I find the argument of stochastic gradient descent wrong (I could be wrong though). RNN's follow the markov property (wrt hidden states from previous time step and the current input) so from time step t to t+1, if you change the parameters, the hidden state at time t (and all the time steps before) would carry stale information unless until you're using something like eligibility traces from RL literature. I also don't know how to overcome this issue.
- I'd be worried about the variance in the estimate of rank one approximation. All the experiments carried out by the authors are small scale (hidden size = 64). I'm curious if authors tried experimenting with larger networks, I'd guess it wont perform well due to the high variance in the approximation. I'd like to see an experiment with hidden size  = 128/256/512/1024. My intuition is that because of high variance it would be difficult to train this network, but I could be wrong. I'm curious what the authors had to say about this.
- If the variance of the approximation is indeed high, can we use something to control the dynamics of the network which can result in less variance. Have authors thought about this ?
- I'd also like to see experiments on copying task/adding task (as these are standard experiments which are done for analysis of long term dependencies)
- I'd also like to see what effect the length of sequence has on the approximation. As small errors in approximation on each step can compound giving rise to chaotic dynamics. (small change in input => large change in output)
- I'd also like to know how using UORO changes the optimization as compared to Back-propagation through time in the sense, does the two approaches would reach same local minimum ? or is there a possibility that the former can reach "less" number of potential local minimas as compared to BPTT.
I'm tempted to give high score for this paper( Score - 7) , as it is unexplored direction in our research community, and I think this paper makes a very useful contribution to tackle this problem in a very principled way.  But I'd like some more experiments to be done (which I have mentioned above), failing to do those experiments, I'd be forced to reduce the score (to score - 5)


THE HIGH-DIMENSIONAL GEOMETRY OF BINARY NEURAL NETWORKS
Recent research has shown that one can train a neural network with binary weights and activations at train time by augmenting the weights with a high-precision continuous latent variable that accumulates small changes from stochastic gradient descent. However, there is a dearth of work to explain why one can effectively capture the features in data with binary weights and activations. Our main result is that the neural networks with binary weights and activations trained using the method of Courbariaux, Hubara et al. (2016) work because of the high-dimensional geometry of binary vectors. In particular, the ideal continuous vectors that extract out features in the intermediate representations of these BNNs are well-approximated by binary vectors in the sense that dot products are approximately preserved. Furthermore, the results and analysis used on BNNs are shown to generalize to neural networks with ternary weights and activations. Compared to previous research that demonstrated good classification performance with BNNs, our work explains why these BNNs work in terms of HD geometry. Our theory serves as a foundation for understanding not only BNNs but a variety of methods that seek to compress traditional neural networks. Furthermore, a better understanding of multilayer binary neural networks serves as a starting point for generalizing BNNs to other neural network architectures such as recurrent neural networks.

This paper investigates numerically and theoretically the reasons behind the empirical success of binarized neural networks. Specifically, they observe that:
(1) The angle between continuous vectors sampled from a spherical symmetric distribution and their binarized version is relatively small in high dimensions (proven to be about 37 degrees when the dimension goes to infinity), and this demonstrated empirically to be true for the binarized weight matrices of a convenet.
(2) Except the first layer, the dot product of weights*activations in each layer is highly correlated with the dot product of (binarized weights)*activations in each layer. There is also a strong correlation between (binarized weights)*activations and (binarized weights)*(binarized activations). This is claimed to entail that the continuous weights of the binarized neural net approximate the continuous weights of a non-binarized neural net trained in the same manner.
(3) To correct the issue with the first layer in (2) it is suggested to use a random rotation, or simply use continues weights in that layer.
The first observation is interesting, is explained clearly and convincingly, and is novel to the best of my knowledge.
The second observation is much less clear to me. Specifically,
a.	The author claim that “A sufficient condition for \delta u to be the same in both cases is L’(x = f(u)) ~ L’(x = g(u))”. However, I’m not sure if I see why this is true: in a binarized neural net, u also changes, since the previous layers are also binarized.
b.	Related to the previous issue, it is not clear to me if in figure 3 and 5, did the authors binarize the activations of that specific layer or all the layers? If it is the first case, I would be interested to know the latter: It is possible that if all layers are binarized, then the differences between the binarized and non-binarized version become more amplified.
c.	For BNNs, where both the weights and activations are binarized, shouldn’t we compare weights*activations to (binarized weights)*(binarized activations)?
d.	To make sure, in figure 4, the permutation of the activations was randomized (independently) for each data sample? If not, then C is not proportional the identity matrix, as claimed in section 5.3.
e.	It is not completely clear to me that batch-normalization takes care of the scale constant (if so, then why did XNOR-NET needed an additional scale constant?), perhaps this should be further clarified.
The third observation seems less useful to me. Though a random rotation may improve angle preservation in certain cases (as demonstrated in Figure 4), it may hurt classification performance (e.g., distinguishing between 6 and 9 in MNIST). Furthermore, since it uses non-binary operations, it is not clear if this rotation may have some benefits (in terms of resource efficiency) over simply keeping the input layer non-binarized.
To summarize, the first part is interesting and nice, the second part was not clear to me, and the last part does not seem very useful.
%%% After Author's response %%%
a. My mistake. Perhaps it should be clarified in the text that u are the weights. I thought that g(u) is a forward propagation function, and therefore u is the neural input (i.e., pre-activation).
Following the author's response and revisions, I have raised my grade.


Deep multitask networks, in which one neural network produces multiple predictive outputs, are more scalable and often better regularized than their single-task counterparts. Such advantages can potentially lead to gains in both speed and performance, but multitask networks are also difficult to train without finding the right balance between tasks. We present a novel gradient normalization (GradNorm) technique which automatically balances the multitask loss function by directly tuning the gradients to equalize task training rates. We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting over single networks, static baselines, and other adaptive multitask loss balancing techniques. GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter α. Thus, what was once a tedious search process which incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks. Ultimately, we hope to demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

Paper summary:
Existing works on multi-task neural networks typically use hand-tuned weights for weighing losses across different tasks. This work proposes a dynamic weight update scheme that updates weights for different task losses during training time by making use of the loss ratios of different tasks. Experiments on two different network indicate that the proposed scheme is better than using hand-tuned weights for multi-task neural networks.
Paper Strengths:
- The proposed technique seems simple yet effective for multi-task learning.
- Experiments on two different network architectures showcasing the generality of the proposed method.
Major Weaknesses:
- The main weakness of this work is the unclear exposition of the proposed technique. Entire technique is explained in a short section-3.1 with many important details missing. There is no clear basis for the main equations 1 and 2. How does equation-2 follow from equation-1? Where is the expectation coming from? What exactly does ‘F’ refer to? There is dependency of ‘F’ on only one of sides in equations 1 and 2? More importantly, how does the gradient normalization relate to loss weight update? It is very difficult to decipher these details from the short descriptions given in the paper.
- Also, several details are missing in toy experiments. What is the task here? What are input and output distributions and what is the relation between input and output? Are they just random noises? If so, is the network learning to overfit to the data as there is no relationship between input and output?
Minor Weaknesses:
- There are no training time comparisons between the proposed technique and the standard fixed loss learning.
- Authors claim that they operate directly on the gradients inside the network. But, as far as I understood, the authors only update loss weights in this paper. Did authors also experiment with gradient normalization in the intermediate CNN layers?
- No comparison with state-of-the-art techniques on the experimented tasks and datasets.
Clarifications:
- See the above mentioned issues with the exposition of the technique.
- In the experiments, why are the input images downsampled to 320x320?
- What does it mean by ‘unofficial dataset’ (page-4). Any references here?
- Why is 'task normalized' test-time loss as good measure for comparison between models in the toy example (Section 4)? The loss ratios depend on initial loss, which is not important for the final performance of the system.
Suggestions:
- I strongly suggest the authors to clearly explain the proposed technique to get this into a publishable state.
- The term ’GradNorm’ seem to be not defined anywhere in the paper.
Review Summary:
Despite promising results, the proposed technique is quite unclear from the paper. With its poor exposition of the technique, it is difficult to recommend this paper for publication.


We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.

The authors provide an improved implementation of the idea of dynamic evaluation, where the update of the parameters used in the last time step proposed in (Mikolov et al. 2010) is replaced with a back-propagation through the last few time steps, and uses  RMSprop rather than vanilla SGD. The method is applied to word level and character level language modeling where it yields some gains in perplexity. The algorithm also appears able to perform domain adaptation, in a setting where a character-level language model trained mostly on English manages to quickly adapt to a Spanish test set.
While the general idea is not novel, the implementation choices matter, and the authors provide one which appears to work well with recently proposed models. The character level experiments on the multiplicative LSTM make the most convincing point, providing a significant improvement over already good results on medium size data sets. Figure 2 also makes a strong case for the method's suitability for applications where domain adaptation is important.
The paper's weakest part is the word level language modeling section. Given the small size of the data sets considered, the results provided are of limited use, especially since the development set is used to fit the RMSprop hyper-parameters. How sensitive are the final results to this choice? Comparing dynamic evaluation to neural cache models is a good idea, given how both depend en medium-term history: (Grave et al. 2017) provide results on the larger text8 and wiki103, it would be useful to see results for dynamic evaluation at least on the former.
An indication of the actual additional evaluation time for word-level, char-level and sparse char-level dynamic evaluation would also be welcome.
Pros:
- Good new implementation of an existing idea
- Significant perplexity gains on character level language modeling
- Good at domain adaptation
Cons:
- Memory requirements of the method
- Word-level language modeling experiments need to be run on larger data sets
(Edit: the authors did respond satisfactorily to the original concern about the size of the word-level data set)


Published as a conference paper at ICLR 2018 COMMUNICATION ALGORITHMS VIA DEEP LEARNING
Coding theory is a central discipline underpinning wireline and wireless modems that are the workhorses of the information age. Progress in coding theory is largely driven by individual human ingenuity with sporadic breakthroughs over the past century. In this paper we study whether it is possible to automate the discovery of decoding algorithms via deep learning. We study a family of sequential codes parametrized by recurrent neural network (RNN) architectures. We show that creatively designed and trained RNN architectures can decode well known sequential codes such as the convolutional and turbo codes with close to optimal performance on the additive white Gaussian noise (AWGN) channel, which itself is achieved by breakthrough algorithms of our times (Viterbi and BCJR decoders, representing dynamic programing and forward-backward algorithms). We show strong generalizations, i.e., we train at a specific signal to noise ratio and block length but test at a wide range of these quantities, as well as robustness and adaptivity to deviations from the AWGN setting.

In this paper the authors propose to use RNNs and LSTMs for channel coding. But I have the impression the authors completely miss the state of the art in channel coding and the results are completely useless for any current communication system. I believe that machine learning, in general, and deep learning, in particular, might be of useful for physical layer communications. I just do not see why it would be useful for channel coding over the AWGN channel. Let me explain.
If the decoder knows that the encoder is using a convolutional code, why does it need to learn the decoder instead of using the Viterbi or BCJR algorithms that are known to be optimal for sequences and symbols, respectively. I cannot imagine an scenario in which the decoder does not know the convolutional code that it is being used and the encoder sends 120,000 bits of training sequence (useless bits from information standpoint) for the decoder to learn it. More important question, do the authors envision that this learning is done every time there is a new connection or it is learnt once and for all. If it is learnt every time that would be ideal if we were discovering new channel codes everyday, clearly not the case. If we learnt it one and for all and then we incorporated in the standard that would only make sense if the GRU structure was computationally better than the BCJR or Viterbi. I would be surprise if it is. If instead of using 2 or 3 memories, we used 6-8 does 120,000 bits be good enough or we need to exponentially increase the training sequence? So the first result in the paper shows that a tailored structure for convolutional encoding can learn to decode it. Basically, the authors are solving a problem that does not need solving.
For the Turbocodes the same principle as before applies. In this case the comments of the authors really show that they do not know anything about coding. In Page 6, we can read: “Unlike the convolutional codes, the state of the art (message-passing) decoders for turbo codes are not the corresponding MAP decoders, so there is no contradiction in that our neural decoder would beat the message-passing ones”. This is so true, so I expected the DNN structure to be significantly better than turbodecoding. But actually, they do not. These results are in Figure 15 page 6 and the solution for the turbo decoders and the DNN architecture are equivalent. I am sure that the differences in the plots can be explained by the variability in the received sequence and not because the DNN is superior to the turbodecoder. Also in this case the training sequence is measured in the megabits for extremely simple components. If the convolutional encoders were larger 6-8 bits, we would be talking about significantly longer training sequences and more complicated NNs.
In the third set the NNs seems to be superior to the standard methods when burst-y noise is used, but the authors seems to indicate that that NN is trained with more information about these bursts that the other methods do not have. My impression is that the authors would be better of focusing on this example and explain it in a way that it is reproducible. This experiment is clearly not well explained and it is hard to know if there is any merit for the proposed NN structure.
Finally, the last result would be the more interesting one, because it would show that we can learn a better channel coding and decoding mechanism that the ones humans have been able to come up with. In this sense, if NNs can solve this problem that would be impressive and would turn around how channel coding is done nowadays. If this result were good enough, the authors should only focus in it and forget about the other 3 cases. The issue with this result is that it actually does not make sense. The main problem with the procedure is that the feedback proposal is unrealistic, this is easy to see in Figure 16 in which the neural encoder is proposed. It basically assumes that the received real-valued y_k can be sent (almost) noiselessly to the encoder with minimal delay and almost instantaneously. So the encoder knows the received error and is able to cancel it out. Even if this procedure could be implemented, which it cannot be. The code only uses 50 bits and it needed 10^7 iterations (500Mbs) to converge. The authors do not show how far they are from the Shannon limit, but I can imagine that with 50 bit code, it should be pretty far.
We know that with long enough LDPC codes we can (almost) reach the Shannon limit, so new structure are not needed. If we are focusing on shorter codes (e.g. latency?) then it will be good to understand why do we need to learn the channel codes. A comparison to the state of the art would be needed. Because clearly the used codes are not close to state of the art. For me the authors either do not know about coding or are assuming that we do not, which explains part of the tone of this review.


Two main families of reinforcement learning algorithms, Q-learning and policy gradients, have recently been proven to be equivalent when using a softmax relaxation on one part, and an entropic regularization on the other. We relate this result to the well-known convex duality of Shannon entropy and the softmax function. Such a result is also known as the Donsker-Varadhan formula. This provides a short proof of the equivalence. We then interpret this duality further, and use ideas of convex analysis to prove a new policy inequality relative to soft Q-learning.
• Policy gradients (V. ), looks to maximize the expected reward by improving policies to favor high-reward actions. In general, the target loss function is regularized by the addition of an entropic functional for the policy. This makes policies more diffuse and less likely to yield degenerate results.
A critical step in the theoretical understanding of the field has been a smooth relaxation of the greedy max operation involved in selecting actions, turned into a Boltzmann softmax O. Nachum & Schuurmans. (2017b.). This new context has lead to a breakthrough this year J. Schulman & Abbeel. (2017) with the proof of the equivalence of both methods of Q-learning and policy gradients. While that result is extremely impressive in its unification, we argue that it is critical to look additionally at the fundamental reasons as to why it occurs. We believe that the convexity of the entropy functional used for policy regularization is at the root of the phenomenon, and that (Lagrangian) duality can be exploited as well, either yielding faster proofs, or further understanding. The contributions of our paper are as follows:
1. We show how convex duality expedites the proof of the equivalence between soft Qlearning and softmax entropic policy gradients -heuristically in the general case, rigorously in the bandit case.
2. We introduce a transportation inequality that relates the expected optimality gap of any policy with its Kullback-Leibler divergence to the optimal policy.
We describe our notations here. Abusing notation heavily by identifying measures with their densities as in dπ(a|s) = π(a|s)da, if we note as either r(s, a) or r(a, s) the reward obtained by taking action a in state s, the expected reward expands as:
K r is a linear functional of π. Adding Shannon entropic regularization 1 improves numerical stability of the algorithm, and prevents early convergence to degenerate solutions. Noting regularization strength β, the objective becomes a free energy functional, named by analogy with a similar quantity in statistical mechanics:
Crucially, viewed as a functional of π, J is convex and is the sum of two parts
2 THE GIBBS VARIATIONAL PRINCIPLE FOR POLICY EVALUATION

Summary
*******
The paper provides a collection of existing results in statistics.
Comments
********
Page 1: references to Q-learning and Policy-gradients look awkwardly recent, given that these have been around for several decades.
I dont get what is the novelty in this paper. There is no doubt that all the tools that are detailed here are extremely useful and powerful results in mathematical statistics. But they are all known.
The Gibbs variational principle is folklore, Proposition 1,2 are available in all good text books on the topic,
and Proposition 4 is nothing but a transportation Lemma.
Now, Proposition 3 is about soft-Bellman operators. This perhaps is less standard because contraction property of soft-Bellman operator in infinite norm is more recent than for Bellman operators.
But as mentioned by the authors, this is not new either.
Also I don't really see the point of providing the proofs of these results in the main material, and not for instance in appendix, as there is no novelty either in the proof techniques.
I don't get the sentence "we have restricted so far the proof in the bandit setting": bandits are not even mentioned earlier.
Decision
********
I am sorry but unless I missed something (that then should be clarified) this seems to be an empty paper: Strong reject.


Human-computer conversation systems have attracted much attention in Natural Language Processing. Conversation systems can be roughly divided into two categories: retrieval-based and generation-based systems. Retrieval systems search a user-issued utterance (namely a query) in a large conversational repository and return a reply that best matches the query. Generative approaches synthesize new replies. Both ways have certain advantages but suffer from their own disadvantages. We propose a novel ensemble of retrieval-based and generation-based conversation system. The retrieved candidates, in addition to the original query, are fed to a reply generator via a neural network, so that the model is aware of more information. The generated reply together with the retrieved ones then participates in a re-ranking process to find the final reply to output. Experimental results show that such an ensemble system outperforms each single module by a large margin.

Summary:
The paper proposes a new dialog model combining both retrieval-based and generation-based modules. Answers are produced in three phases: a retrieval-based model extracts candidate answers; a generator model, conditioned on retrieved answers, produces an additional candidate; a reranker outputs the best among all candidates.
The approach is interesting: the proposed ensemble can improve on both the retrieval module and the generation module, since it does not restrict modeling power (e.g. the generator is not forced to be consistent with the candidates). I am not aware of similar approaches for this task. One work that comes to mind regarding the blend of retrieval and generation is Memory Networks (e.g. https://arxiv.org/pdf/1606.03126.pdf and references): given a query, a set of relevant memories is extracted from a KB using an inverted index and the memories are fed into the generator. However, the extracted items in the current work are candidate answers which are used both to feed the generator and to participate in reranking.
The experimental section focuses on the task of building conversational systems. The performance measures used are 1) a human evaluation score with three volunteers and 2) BLUE scores. While these methods are not very satisfying, effective evaluation of such systems is a known difficulty.
The results show that the ensemble outperforms the individual modules, indicating that: the multi-seq2seq models have learned to use the new inputs as needed and that the ranker is correlated with the evaluation metrics.
However, the results themselves do not look impressive to me: the subjective evaluation is close to the "borderline" score; in the examples provided, one is good, the other is borderline/bad, and the baseline always provides something very short. Does the LSTM work particularly poor on this dataset? Given that this is a novel dataset, I don't know what the state-of-the-art should be. Could you provide more insight? Have you considered adding a benchmark dataset (e.g. a QA dataset)?
Specific questions:
1. The paper motivates conditioning on the candidates in two ways. First, that the candidates bring additional information which the decoder can use (e.g. read from the candidates locations, actions, etc.). Second, that the probability of universal replies must decrease due to the additional condition. I think the second argument depends on how the conditioning is performed: if the candidates are simply appended to the input, the model can learn to ignore them.
2. The copy mechanism is a nice touch, encouraging the decoder to use the provided queries. Why not copy from the query too, e.g. with some answers reusing part of the query <"Where are you going?", "I'm going to the park">?
3. How often does the model select the generated answer vs. the extracted answers? In both examples provided the selected answer is the one merging the candidate answers.
Minor issues:
- Section 3.2: using and the state
- Section 3.2: more than one replies
- last sentence on page 3: what are the "following principles"?


In this research, we present a novel learning scheme called network iterative learning for deep neural networks. Different from traditional optimization algorithms that usually optimize directly on a static objective function, we propose in this work to optimize a dynamic objective function in an iterative fashion capable of adapting its function form when being optimized. The optimization is implemented as a series of intermediate neural net functions that is able to dynamically grow into the targeted neural net objective function. This is done via network morphism so that the network knowledge is fully preserved with each network growth. Experimental results demonstrate that the proposed network iterative learning scheme is able to significantly alleviate the degradation problem. Its effectiveness is verified on diverse benchmark datasets.

This submission develops a learning scheme for training deep neural networks with adoption of network morphism (Wei et al., 2016), which optimizes a dynamic objective function in an iterative fashion capable of adapting its function form when being optimized, instead of directly optimizing a static objective function. Overall, the idea looks interesting and the manuscript is well-written. The shown experimental results should be able to validate the effectiveness of the learning scheme to some extent.
It would be more convincing to include the performance evaluation of the learning scheme in some representative applications, since the optimality of the training objective function is not necessarily the same as that of the trained network in the application of interest.
Below are two minor issues:
- In page 2, it is stated that Fig. 2(e) illustrates the idea of the proposed network iterative learning scheme for deep neural networks based on network morphism. However, the idea seems not clear from Fig. 2(e).
- In page 4, “such network iterative learning process” should be “such a network iterative learning process”.


Published as a conference paper at ICLR 2018 NOISY NETWORKS FOR EXPLORATION
We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet is straightforward to implement and adds little computational overhead. We find that replacing the conventional exploration heuristics for A3C, DQN and Dueling agents (entropy reward and -greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.

This paper introdues NoisyNets, that are neural networks whose parameters are perturbed by a parametric noise function, and they apply them to 3 state-of-the-art deep reinforcement learning algorithms: DQN, Dueling networks and A3C. They obtain a substantial performance improvement over the baseline algorithms, without explaining clearly why.
The general concept is nice, the paper is well written and the experiments are convincing, so to me this paper should be accepted, despite a weak analysis.
Below are my comments for the authors.
---------------------------------
General, conceptual comments:
The second paragraph of the intro is rather nice, but it might be updated with recent work about exploration in RL.
Note that more than 30 papers are submitted to ICLR 2018 mentionning this topic, and many things have happened since this paper was
posted on arxiv (see the "official comments" too).
p2: "our NoisyNet approach requires only one extra parameter per weight" Parameters in a NN are mostly weights and biases, so from this sentence
one may understand that you close-to-double the number of parameters, which is not so few! If this is not what you mean, you should reformulate...
p2: "Though these methods often rely on a non-trainable noise of vanishing size as opposed to NoisyNet which tunes the parameter of noise by gradient descent."
Two ideas seem to be collapsed here: the idea of diminishing noise over an experiment, exploring first and exploiting later, and the idea of
adapting the amount of noise to a specific problem. It should be made clearer whether NoisyNet can address both issues and whether other
algorithms do so too...
In particular, an algorithm may adapt noise along an experiment or from an experiment to the next.
From Fig.3, one can see that having the same initial noise in all environments is not a good idea, so the second mechanism may help much.
BTW, the short section in Appendix B about initialization of noisy networks should be moved into the main text.
p4: the presentation of NoisyNets is not so easy to follow and could be clarified in several respects:
- a picture could be given to better explain the structure of parameters, particularly in the case of factorised (factorized, factored?) Gaussian noise.
- I would start with the paragraph "Considering a linear layer [...] below)" and only after this I would introduce \theta and \xi as a more synthetic notation.
Later in the paper, you then have to state "...are now noted \xi" several times, which I found rather clumsy.
p5: Why do you use option (b) for DQN and Dueling and option (a) for A3C? The reason why (if any) should be made clear from the clearer presentation required above.
By the way, a wild question: if you wanted to use NoisyNets in an actor-critic architecture like DDPG, would you put noise both in the actor and the critic?
The paragraph above Fig3 raises important questions which do not get a satisfactory answer.
Why is it that, in deterministic environments, the network does not converge to a deterministic policy, which should be able to perform better?
Why is it that the adequate level of noise changes depending on the environment? By the way, are we sure that the curves of Fig3 correspond to some progress
in noise tuning (that is, is the level of noise really "better" through time with these curves, or they they show something poorly correlated with the true reasons of success?)?
Finally, I would be glad to see the effect of your technique on algorithms like TRPO and PPO which require a stochastic policy for exploration, and where I believe that the role of the KL divergence bound is mostly to prevent the level of stochasticity from collasping too quickly.
-----------------------------------
Local comments:
The first sentence may make the reader think you only know about 4-5 old works about exploration.
Pp. 1-2 : "the approach differs ... from variational inference. [...] It also differs variational inference..."
If you mean it differs from variational inference in two ways, the paragraph should be reorganized.
p2: "At a high level our algorithm induces a randomised network for exploration, with care exploration
via randomised value functions can be provably-efficient with suitable linear basis (Osband et al., 2014)"
=> I don't understand this sentence at all.
At the top of p3, you may update your list with PPO and ACKTR, which are now "classical" baselines too.
Appendices A1 and A2 are a lot redundant with the main text (some sentences and equations are just copy-pasted), this should be improved.
The best would be to need to reject nothing to the Appendix.
---------------------------------------
Typos, language issues:
p2
the idea ... the optimization process have been => has
p2
Though these methods often rely on a non-trainable noise of vanishing size as opposed to NoisyNet which tunes the parameter of noise by gradient descent.
=> you should make a sentence...
p3
the the double-DQN
several times, an equation is cut over two lines, a line finishing with "=", which is inelegant
You should deal better with appendices: Every "Sec. Ax/By/Cz" should be replaced by "Appendix Ax/By/Cz".
Besides, the big table and the list of performances figures should themselves be put in two additional appendices
and you should refer to them as Appendix D or E rather than "the Appendix".


Published as a conference paper at ICLR 2018 TOWARDS NEURAL PHRASE-BASED MACHINE TRANSLATION
In this paper, we present Neural Phrase-based Machine Translation (NPMT).
1 Our method explicitly models the phrase structures in output sequences using SleepWAke Networks (SWAN), a recently proposed segmentation-based sequence modeling method. To mitigate the monotonic alignment requirement of SWAN, we introduce a new layer to perform (soft) local reordering of input sequences. Different from existing neural machine translation (NMT) approaches, NPMT does not use attention-based decoding mechanisms. Instead, it directly outputs phrases in a sequential order and can decode in linear time. Our experiments show that NPMT achieves superior performances on IWSLT 2014 German-English/EnglishGerman and IWSLT 2015 English-Vietnamese machine translation tasks compared with strong NMT baselines. We also observe that our method produces meaningful phrases in output languages.

Authors proposed a new neural-network based machine translation method that generates the target sentence by generating multiple partial segments in the target sentence from different positions in the source information. The model is based on the SWAN architecture which is previously proposed, and an additional "local reordering" layer to reshuffle source information to adjust those positions to the target sentence.
Using the SWAN architecture looks more reasonable than the conventional attention mechanism when the ground-truth word alignment is monotone. Also, the concept of local reordering mechanism looks well to improve the basic SWAN model to reconfigure it to the situation of machine translation tasks.
The "window size" of the local reordering layer looks like the "distortion limit" used in traditional phrase-based statistical machine translation methods, and this hyperparameter may impose a similar issue with that of the distortion limit into the proposed model; small window sizes may drop information about long dependency. For example, verbs in German sentences sometimes move to the tail of the sentence and they introduce a dependency between some distant words in the sentence. Since reordering windows restrict the context of each position to a limited number of neighbors, it may not capture distant information enough. I expected that some observations about this point will be unveiled in the paper, but unfortunately, the paper described only a few BLEU scores with different window sizes which have not enough information about it. It is useful for all followers of this paper to provide some observations about this point.
In addition, it could be very meaningful to provide some experimental results on linguistically distant language pairs, such as Japanese and English, or simply reversing word orders in either source or target sentences (this might work to simulate the case of distant reordering).
Authors argued some differences between conventional attention mechanism and the local reordering mechanism, but it is somewhat unclear that which ones are the definite difference between those approaches.
A super interesting and mysterious point of the proposed method is that it achieves better BLEU than conventional methods despite no any global language models (Table 1 row 8), and the language model options (Table 1 row 9 and footnote 4) may reduce the model accuracy as well as it works not so effectively. This phenomenon definitely goes against the intuitions about developing most of the conventional machine translation models. Specifically, it is unclear how the model correctly treats word connections between segments without any global language model. Authors should pay attention to explain more detailed analysis about this point in the paper.
Eq. (1) is incorrect. According to Fig. 2, the conditional probability in the product operator should be revised to p(a_t | x_{1:t}, a_{1:t-1}), and the independence approximation to remove a_{1:t-1} from the conditions should also be noted in the paper.
Nevertheless, the condition x_{1:t} could not be reduced because the source position is always conditioned by all previous positions through an RNN.