ON THE INSUFFICIENCY OF EXISTING MOMENTUM SCHEMES FOR STOCHASTIC OPTIMIZATION
Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD). Rigorously speaking, "fast gradient" methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact. In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case, they partially mimic their exact gradient counterparts, resulting in some practical gain. This work provides a counterpoint to this belief by proving that there exist simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters. These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances. These results suggest (along with empirical evidence) that HB or NAG's practical performance gains are a by-product of mini-batching. Furthermore, this work provides a viable (and provable) alternative, which, on the same set of problem instances, significantly improves over HB, NAG, and SGD's performance. This algorithm, referred to as Accelerated Stochastic Gradient Descent (ASGD), is a simple to implement stochastic algorithm, based on a relatively less popular variant of Nesterov's Acceleration. Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD. The code implementing the ASGD Algorithm can be found here 1 .

I like the idea of the paper. Momentum and accelerations are proved to be very useful both in deterministic and stochastic optimization. It is natural that it is understood better in the deterministic case. However, this comes quite naturally, as deterministic case is a bit easier ;) Indeed, just recently people start looking an accelerating in stochastic formulations. There is already accelerated SVRG, Jain et al 2017, or even Richtarik et al (arXiv: 1706.01108, arXiv:1710.10737).
I would somehow split the contributions into two parts:
1) Theoretical contribution: Proposition 3 (+ proofs in appendix)
2) Experimental comparison.
I like the experimental part (it is written clearly, and all experiments are described in a lot of detail).
I really like the Proposition 3 as this is the most important contribution of the paper. (Indeed, Algorithms 1 and 2 are for reference and Algorithm 3 was basically described in Jain, right?).
Significance: I think that this paper is important because it shows that the classical HB method cannot achieve acceleration in a stochastic regime.
Clarity: I was easy to read the paper and understand it.
Few minor comments:
1. Page 1, Paragraph 1: It is not known only for smooth problems, it is also true for simple non-smooth (see e.g. https://link.springer.com/article/10.1007/s10107-012-0629-5)
2. In abstract : Line 6 - not completely true, there is accelerated SVRG method, i.e. the gradient is not exact there, also see Recht (https://arxiv.org/pdf/1701.03863.pdf) or Richtarik et al (arXiv: 1706.01108, arXiv:1710.10737) for some examples where acceleration can be proved when you do not have an exact gradient.
3. Page 2, block "4" missing "." in "SGD We validate"....
4. Section 2. I think you are missing 1/2 in the definition of the function. Otherwise, you would have a constant "2" in the Hessian, i.e. H= 2 E[xx^T]. So please define the function as  f_i(w) = 1/2 (y - <w,x_i>)^2. The same applies to Section 3.
5. Page 6, last line, .... was downloaded from "pre". I know it is a link, but when printed, it looks weird.


A FRAMEWORK FOR THE QUANTITATIVE EVALUATION OF DISENTANGLED REPRESENTATIONS
Recent AI research has emphasised the importance of learning disentangled representations of the explanatory factors behind data. Despite the growing interest in models which can learn such representations, visual inspection remains the standard evaluation metric. While various desiderata have been implied in recent definitions, it is currently unclear what exactly makes one disentangled representation better than another. In this work we propose a framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available. Three criteria are explicitly defined and quantified to elucidate the quality of learnt representations and thus compare models on an equal basis. To illustrate the appropriateness of the framework, we employ it to compare quantitatively the representations learned by recent state-of-the-art models.

The paper addresses the problem of devising a quantitative benchmark to evaluate the capability of algorithms to disentangle factors of variation in the data.
*Quality*
The problem addressed is surely relevant in general terms. However, the contributed framework did not account for previously proposed metrics (such as equivariance, invariance and equivalence). Within the experimental results, only two methods are considered: although Info-GAN is a reliable competitor, PCA seems a little too basic to compete against. The choice of using noise-free data only is a limiting constraint (in [Chen et al. 2016], Info-GAN is applied to real-world data).
Finally, in order to corroborate the quantitative results, authors should have reported some visual experiments in order to assess whether a change in c_j really correspond to a change in the corresponding factor of variation z_i according to the learnt monomial matrix.
*Clarity*
The explanation of the theoretical framework is not clear. In fact, Figure 1 is straight in identifying disentanglement and completeness as a deviation from an ideal bijective mapping. But, then, the authors missed to clarify how the definitions of D_i and C_j translate this requirement into math.
Also, the criterion of informativeness of Section 2 is split into two sub-criteria in Section 3.3, namely test set NRMSE and Zero-Shot NRMSE: such shift needs to be smoothed and better explained, possibly introducing it in Section 2.
*Originality*
The paper does not allow to judge whether the three proposed criteria are original or not with respect to the previously proposed ones of [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015].
*Significance*
The significance of the proposed evaluation framework is not fully clear. The initial assumption of considering factors of variations related to graphics-generated data undermines the relevance of the work. Actually, authors only consider synthetic (noise-free) data belonging to one class only, thus not including the factors of variations related to noise and/or different classes.
PROS:
The problem faced by the authors is interesting
CONS:
The criteria of disentanglement, informativeness & completeness are not fully clear as they are presented.
The proposed criteria are not compared with previously proposed ones - equivariance, invariance and equivalence [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015]. Thus, it is not possible to elicit from the paper to which extent they are novel or how they are related..
The dataset considered is noise-free and considers one class only. Thus, several factors of variation are excluded a priori and this undermines the significance of the analysis.
The experimental evaluation only considers two methods, comparing Info-GAN, a state-of-the-art method, with a very basic PCA.
**FINAL EVALUATION**
The reviewer rates this paper with a weak reject due to the following points.
1) The novel criteria are not compared with existing ones [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015].
2) There are two flaws in the experimental validation:
	2.1) The number of methods in comparison (InfoGAN and PCA) is limited.
	2.2) A synthetic dataset is only considered.
The reviewer is favorable in rising the rating towards acceptance if points 1 and 2 will be fixed.
**EVALUATION AFTER AUTHORS' REBUTTAL**
The reviewer has read the responses provided by the authors during the rebuttal period. In particular, with respect to the highlighted points 1 and 2, point 1 has been thoroughly answered and the novelty with respect previous work is now clearly stated in the paper. Despite the same level of clarification has not been reached for what concerns point 2, the proposed framework (although still limited in relevance due to the lack of more realistic settings) can be useful for the community as a benchmark to verify the level of disentanglement than newly proposed deep architectures can achieve. Finally, by also taking into account the positive evaluation provided by the fellow reviewers, the rating of the paper has been risen towards acceptance.


It has been argued that the brain is a prediction machine that continuously learns how to make better predictions about the stimuli received from the external environment. It builds a model of the world around us and uses this model to infer the external stimulus. Predictive coding has been proposed as a mechanism through which the brain might be able to build such a model of the external environment. However, it is not clear how predictive coding can be used to build deep neural network models of the brain while complying with the architectural constraints imposed by the brain. In this paper, we describe an algorithm to build a deep generative model using predictive coding that can be used to infer latent representations about the stimuli received from external environment. Specifically, we used predictive coding to train a deep neural network on real-world images in a unsupervised learning paradigm. To understand the capacity of the network with regards to modeling the external environment, we studied the latent representations generated by the model on images of objects that are never presented to the model during training. Despite the novel features of these objects the model is able to infer the latent representations for them. Furthermore, the reconstructions of the original images obtained from these latent representations preserve the important details of these objects.

The paper "A Deep Predictive Coding Network for Learning Latent Representations" considers learning of a generative neural network. The network learns unsupervised using a predictive coding setup. A subset of the CIFAR-10 image database (1000 images horses and ships) are used for training. Then images generated using the latent representations inferred on these images, on translated images, and on images of other objects are shown. It is then claimed that the generated images show that the network has learned good latent representations.
I have some concerns about the paper, maybe most notably about the experimental result and the conclusions drawn from them. The numerical experiments are motivated as a way to "understand the capacity of the network with regards to modeling the external environment" (abstract). And it is concluded in the final three sentences of the paper that the presented network "can infer effective latent representations for images of other objects" (i.e., of objects that have not been used for training); and further, that "in this regards, the network is better than most existing algorithms [...]".
I expected the numerical experiments to show results instructive about what representations or what abstractions are learned in the different layers of the network using the learning algorithm and objectives suggested. Also some at least quantifiable (if not benchmarked) outcomes should have been presented given the rather strong claims/conclusions in abstract and discussion/conclusion sections. As a matter of fact, all images shown (including those in the appendix) are blurred versions of the original images, except of one single image: Fig. 4 last row, 2nd image (and that is not commented on). If these are the generated images, then some reconstruction is done by the network, fine, but also not unsurprising as the network was told to do so by the used objective function. What precisely do we learn here? I would have expected the presentation of experimental results to facilitate the development of an understanding of the computations going on in the trained network. How can the reader conclude any functioning from these images? Using the right objective function, reconstructions can also be obtained using random (not learned) generative fields and relatively basic models. The fact that image reconstruction for shifted images or new images is evidence for a sophisticated latent representations is, to my mind, not at all shown here. What would be a good measure for an "effective latent representation" that substantiates the claims made? The reconstruction of unseen images is claimed central but as far as I could see, Figures 2, 3, and 4 are not even referred to in the text, nor is there any objective measure discussed. Studying the relation between predictive coding and deep learning makes sense, but I do not come to the same (strong) conclusions as the author(s) by considering the experimental results - and I do not see evidence for a sophisticated latent representation learned by the network. I am not saying that there is none, but I do not see how the presented experimental results show evidence for this.
Furthermore, the authors stress that a main distinguishing feature of their approach (top of page 3) is that in their network information flows from latent space to observed space (e.g. in contrast to CNNs). That is a true statement but also one which is true for basically all generative models, e.g., of standard directed graphical models such as wake-sleep approaches (Hinton et al., 1995), deep SBNs and more recent generative models used in GANs (Goodfellow et al, 2014). Any of these references would have made a lot of sense.
With my evaluation I do not want to be discouraging about the general approach. But I can not at all give a good evaluation given the current experimental results (unless substantial new evidence which make me evaluate these results differently is provided in a discussion).
Minor:
- no legend for Fig. 1
-notes -> noted
have focused


Workshop track -ICLR 2018 COVARIANT COMPOSITIONAL NETWORKS FOR LEARNING GRAPHS
Most existing neural networks for learning graphs address permutation invariance by conceiving of the network as a message passing scheme, where each node sums the feature vectors coming from its neighbors. We argue that this imposes a limitation on their representation power, and instead propose a new general architecture for representing objects consisting of a hierarchy of parts, which we call covariant compositional networks (CCNs). Here, covariance means that the activation of each neuron must transform in a specific way under permutations, similarly to steerability in CNNs. We achieve covariance by making each activation transform according to a tensor representation of the permutation group, and derive the corresponding tensor aggregation rules that each neuron must implement. Experiments show that CCNs can outperform competing methods on standard graph learning benchmarks.

Thank you for your contribution to ICLR. The paper covers a very interesting topic and presents some though-provoking ideas.
The paper introduces "covariant compositional networks" with the purpose of learning graph representations. An example application also covered in the experimental section is graph classification.
Given a finite set S, a compositional network is simply a partially ordered set P where each element of P is a subset of S and where P contains all sets of cardinality 1 and the set S itself. Unfortunately, the presentation of the approach is extremely verbose and introduces old concepts (e.g., partially ordered set) under new names.  The basic idea (which is not new) of this work is that we need to impose some sort of hierarchical order on the nodes of the graph so as to learn hierarchical feature representations. Moreover, the hierarchical order of the nodes should be invariant to valid permutations of the nodes, that is, two isomorphic graphs should have the same hierarchical order on their nodes and the same feature representations. Since this is the case for graph embedding methods that collect feature representations from their neighbors in the graph (and where the feature aggregation functions are symmetric) it makes sense that "compositional networks" generalize graph convolutional networks (and the more general message passing neural networks framework).
The most challenging problem, however, namely the problem of finding a concrete and suitable permutation invariant hierarchical decomposition of the nodes plus some aggregation/pooling functions to compute the feature representations is not addressed in sufficient detail. The paper spends a lot of time on some theoretical definitions and (trivial) proofs but then fails to make the connection to an approach that works in practice. The description of the experiments and which compositional network is chosen and how it is chosen seems to be missing. The only part hinting at the model that was actually used in the experiments is the second paragraph of the section 'Experimental Setup', consisting of one long sentence that is incomprehensible to me.
Instead of spending a lot of effort on the definitions and (somewhat trivial) propositions in the first half of the paper, the authors should spend much more time on detailing the experiments and the actual model that they used. In an effort to make the framework as general as possible, you ended up making the paper highly verbose and difficult to follow.
Please address the following points or clarify in your rebuttal if I misunderstood something:
- what precisely is the novel contribution of your work (it cannot be "compositional networks" and the propositions concerning those because these are just old concepts under new names)?
- explain precisely (and/or more directly/less convoluted) how your model used in the experiments looks like; why do you think it is better than the other methods?
- given that compositional network is a very general concept (partially ordered set imposed on subsets of the graph vertices), what is the principled set of steps one has to follow to arrive at such a compositional network tailored to a particular graph collection? isn't (or shouldn't) that be the contribution of this work? Am I missing something?
In general, you should write the paper much more to the point and leave out unnecessary math (or move to an appendix).  The paper is currently highly inaccessible.


The field of medical diagnostics contains a wealth of challenges which closely resemble classical machine learning problems; practical constraints, however, complicate the translation of these endpoints naively into classical architectures. Many tasks in radiology, for example, are largely problems of multi-label classification wherein medical images are interpreted to indicate multiple present or suspected pathologies. Clinical settings drive the necessity for high accuracy simultaneously across a multitude of pathological outcomes and greatly limit the utility of tools which consider only a subset. This issue is exacerbated by a general scarcity of training data and maximizes the need to extract clinically relevant features from available samples -ideally without the use of pre-trained models which may carry forward undesirable biases from tangentially related tasks. We present and evaluate a partial solution to these constraints in using LSTMs to leverage interdependencies among target labels in predicting 14 pathologic patterns from chest x-rays and establish state of the art results on the largest publicly available chest x-ray dataset from the NIH without pre-training. Furthermore, we propose and discuss alternative evaluation metrics and their relevance in clinical practice.

The paper proposes to combine the recently proposed DenseNet architecture with LSTMs to tackle the problem of predicting different pathologic patterns from chest x-rays. In particular, the use of LSTMs helps take into account interdependencies between pattern labels.
Strengths:
- The paper is very well written. Contextualization with respect to previous work is adequate. Explanations are clear. Novelties are clearly identified by the authors.
- Quantitative improvement with respect to the state the art.
Weaknesses:
- The paper does not introduce strong technical novelties -- mostly, it seems to apply previous techniques to the medical domain. It could have been interesting to know if there are more insights / lessons learned in this process. This could be of interest for a broader audience. For instance, what are the implications of using higher-resolution images as input to DenseNet / decreasing the number of layers? How do the features learned at different layers compare to the ones of the original network trained for image classification? How do features of networks pre-trained on ImageNet, and then fine-tuned for the medical domain, compare to features learned from medical images from scratch?
- The impact of the proposed approach on medical diagnostics is unclear. The authors could better discuss how the approach could be adopted in practice. Also, it could be interesting also to discuss how the results in Table 2 and 3 compare to human classification capabilities, and if that performance would be already enough for building a computer-aided diagnosis system.
Finally -- is it expected that the ordering of the factorization in Eq. 3 does not count much (results in Table 3)? As a non-expert in the field, I'd expect that ordering between pathologic patterns matters more.


Published as a conference paper at ICLR 2018 NEUMANN OPTIMIZER: A PRACTICAL OPTIMIZATION ALGORITHM FOR DEEP NEURAL NETWORKS
Progress in deep learning is slowed by the days or weeks it takes to train large models. The natural solution of using more hardware is limited by diminishing returns, and leads to inefficient use of additional resources. In this paper, we present a large batch, stochastic optimization algorithm that is both faster than widely used algorithms for fixed amounts of computation, and also scales up substantially better as more computational resources become available. Our algorithm implicitly computes the inverse Hessian of each mini-batch to produce descent directions; we do so without either an explicit approximation to the Hessian or Hessian-vector products. We demonstrate the effectiveness of our algorithm by successfully training large ImageNet models (Inception-V3, Resnet-50, Resnet-101 and Inception-Resnet-V2) with mini-batch sizes of up to 32000 with no loss in validation error relative to current baselines, and no increase in the total number of steps. At smaller mini-batch sizes, our optimizer improves the validation error in these models by 0.8-0.9%. Alternatively, we can trade off this accuracy to reduce the number of training steps needed by roughly 10-30%. Our work is practical and easily usable by others -only one hyperparameter (learning rate) needs tuning, and furthermore, the algorithm is as computationally cheap as the commonly used Adam optimizer.

This paper presents a new 2nd-order algorithm that implicitly uses curvature information, and it shows the intuition behind the approximation schemes in the algorithms and also validates the heuristics in various experiments.  The method involves using Neumann Series and Richardson iteration to avoid Hessian-vector product in second order method for NN.  In the actual performance, the paper presents both practical efficiency and better generalization error in different deep neural networks for image classification tasks, and the authors also show differences according to different settings, e.g., Batch Size, Regularization.  The numerical examples are relatively clear and easy to figure out details.
1. While the paper presents the algorithm as an optimization algorithm, although it gets better learning performance, it would be interesting to see how well it is as an optimizer.  For example, one simple experiment would be showing how it works for convex problems, e.g., logistic regression.  Realistic DNN systems are very complex, and evaluating the method in a simple setting would help a lot in determining what if anything is novel about the method.
2. Also, for deep learning problems, it would be more convincing to see how different initialization can affect the performances.
3. Although the authors present their algorithm as a second order method at beginning, the final algorithm is kind of like a complex momentum SGD with limited memory.  Rather than simply throwing out a new method with a new name, it would be helpful to understand what the steps of this method are implicitly doing.  Please explain more about this.
4. It said that the algorithm is hyperparameter free except for learning rate.  However, it is hard to see why there is no need to tune other hyperparameters, e.g., Cubic Regularizer, Repulsive Regularizer.  The effect/sensitivity of hyperparameters for second order methods are quite different than hyperparameters for first order methods, and it is of interest to know how hyperparameters for implicit second order methods perform.
5. For Section 4.2, the well know benefit by using large batch size to train models is that it could reduce training time and epochs.  However, from Table 3, there is no such phenomenon.  Please explain.


It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequence-to-sequence models for structured prediction problems to develop policies over discretized spaces. Central to this method is the realization that complex functions over high dimensional spaces can be modeled by neural networks that predict one dimension at a time. Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions. With this parameterization, it is possible to both leverage the compositional structure of action spaces during learning, as well as compute maxima over action spaces (approximately). On a simple example task we demonstrate empirically that our method can perform global search, which effectively gets around the local optimization issues that plague DDPG. We apply the technique to off-policy (Q-learning) methods and show that our method can achieve the state-of-the-art for off-policy methods on several continuous control tasks.

The paper describes a new RL technique for high dimensional action spaces.  It discretizes each dimension of the action space, but to avoid an exponential blowup, it selects the action for each dimension in sequence.  This is an interesting approach.  The paper reformulates the MDP with a high dimensional action space into an equivalent MDP with more time steps (one per dimension) that each selects the action in one dimension.  This makes sense.
While I do like very much the model, I am perplex about the training technique.  The lower MDP is precisely the new proposed model with unidimensional actions and therefore it should be sufficient.  However, the paper also describes an upper MDP that seems to be superfluous.  The two MDPs are mathematically equivalent, but their Q-values are obtained differently (TD-0 for the upper MDP and Q-learning for the lower MDP) and yet the paper tries to minimize the Euclidean distance between them.  This is really puzzling since the different training algorithms suggest that the Q-values should be different while minimizing the Euclidean distance between them tries to make them equal.  The paper suggests that divergence occurs without the upper MDP.  This is really suspicious. The approach feels like a band-aid solution to cover a problem that the authors could not identify.  While the empirical results are good, I don't think the paper should be published until the authors figure out a principled way of training.
The proposed approach reformulates the MDP with high dimensional actions into an equivalent one with uni dimensional actions.  There is a catch.  This approach effectively hides the exponential action space into the state space which becomes exponential.  Since u contains all the actions of the previous dimensions, we are effectively increasing the state space by an exponential factor.  The paper should discuss this and explain what are the consequences in practice.  In the end, the MDP does not become simpler.
Overall, this is an interesting paper with a good idea, but the training technique is not mature enough for publication.


We propose Bayesian Deep Q-Network (BDQN), a practical Thompson sampling based Reinforcement Learning (RL) Algorithm. Thompson sampling allows for targeted exploration in high dimensions through posterior sampling, but is usually computationally expensive. We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efficiently through the Gaussian distribution. We apply our method to a wide range of Atari Arcade Learning Environments. Since BDQN carries out more efficient exploration, it is able to reach higher rewards substantially faster than a key baseline, DDQN.

The authors describe how to use Bayesian neural networks with Thompson sampling
for efficient exploration in q-learning. The Bayesian neural networks are only
Bayesian in the last layer. That is, the authors learn all the previous layers
by finding point estimates. The Bayesian learning of the last layer is then
tractable since it consists of a linear Gaussian model. The resulting method is
called BDQL. The experiments performed show that the proposed approach, after
hyper-parameter tuning, significantly outperforms the epsilon-greedy
exploration approaches such as DDQN.
Quality:
I am very concern about the authors stating on page 1 "we sample from the
posterior on the set of Q-functions". I believe this statement is not correct.
The Bayesian posterior distribution is obtained by combining an assumed
generative model for the data, data sampled from that model and some prior
assumptions. In this paper there is no generative model for the data and the
data obtained is not actually sampled from the model. The data are just targets
obtained by the q-learning rule. This means that the authors are adapting
Q-learning methods so that they look Bayesian, but in no way they are dealing
with a principled posterior distribution over Q-functions. At least this is my
opinion, I would like to encourage the authors to be more precise and show in
the paper what is the exact posterior distribution over Q-functions and show
how they approximate that distribution, taking into account that a posterior
distribution is obtained as $p(theta|D) \propto p(D|theta)p(\theta)$. In the
case addressed in the paper, what is the likelihood $p(D|\theta)$ and what are
the modeling assumptions that explain how $D$ is generated by sampling from a
model parameterized by \theta?
I am also concerned about the hyper-parameter tuning for the baselines. In
section 5 (choice of hyper-parameters) the authors describe a quite exhaustive
hyper-parameter tuning procedure for BDQL. However, they do not mention whether
they perform a similar hyper-parameter tuning for DDQN, in particular for the
parameter epsilon which will determine the amount of exploration. This makes me
wonder if the comparison in table 2 is fair. Especially, because the authors
tune the amount of data from the replay-buffer that is used to update their
posterior distribution. This will have the effect of tuning the width of their
posterior approximation which is directly related to the amount of exploration
performed by Thompson sampling. You can, therefore, conclude that the authors are
tuning the amount of exploration that they perform on each specific problem.
Is that also being done for the baseline DDQN, for example, by tuning epsilon in
each problem?
The authors also report in table 2 the scores obtained for DDQN by Osband et
al. 2016. What is the purpose of including two rows in table 2 with the same
method? It feels a bit that the authors want to hide the fact that they only
compare with a singe epsilon-greedy baseline (DDQN). Epsilon-greedy methods
have already been shown to be less efficient than Bayesian methods with
Thompson sampling for exploration in q learning (Lipton et al. 2016).
The authors do not compare with variational approaches to Bayesian learning
(Lipton et al. 2016). They indicate that since Lipton et al. "do not
investigate the Atari games, we are not able to have their method as an
additional baseline". This statement seems completely unjustified. The authors
should clearly include a description of why Lipton's approach cannot be applied
to the Atari games or include it as a baseline.
The method proposed by the authors is very similar to Lipton's approach. The
only difference is that Lipton et al. use variational inference with a
factorized Gaussian distribution to approximate the posterior on all the
network weights. The authors by contrast, perform exact Bayesian inference, but
only on the last layer of their neural network. It would be very useful to know
whether the exact linear Gaussian model in the last layer proposed by the
authors has advantages with respect to a variational approximation on all the
network weights. If Lipton's method would be expensive to apply to large-scale
settings such as the Atari games, the authors could also compare with that
method in smaller and simpler problems.
The plots in Figure 2 include performance in terms of episodes. However, it
would also be useful to know how much is the extra computational costs of
the proposed method. One could imagine that computing the posterior
approximation in equation 6 has some additional cost. How do BDQN and DDQN
compare when one takes into account running time and not episode count into
account?
Clarity:
The paper is clearly written. However, I found a lack of motivation for the
specific design choices made to obtain equations 9 and 10. What is a_t in
equation 9? The parameters \theta are updated just after equation 10 by
following the gradient of the loss in which the weights of the last layer are
fixed to a posterior sample, instead of the posterior mean. Is this update rule
guaranteed to produce convergence of \theta? I could imagine that at different
times, different posterior samples of the weights will be used to compute the
gradients. Does this create any instability in learning?
I found the paragraph just above section 5 describing the maze-like
deterministic game confusing and not very useful. The authors should improve
this paragraph.
Originality:
The proposed approach in which the weights in the last layer of the neural
network are the only Bayesian ones is not new. The same method was proposed in
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., ... &
Adams, R. (2015, June). Scalable Bayesian optimization using deep neural
networks. In International Conference on Machine Learning (pp. 2171-2180).
which the authors fail to cite. The use of Thompson sampling for efficient
exploration in deep Q learning is also not new since it has been proposed by
Lipton et al. 2016. The main contribution of the paper is to combine these two
methods (equations 6-10) and evaluate the results in the large-scale setting of
ATARI games, showing that it works in practice.
Significance:
It is hard to determine how significant the work is since the authors only
compare with a single baseline and leave aside previous work on efficient
exploration with Thompson sampling based on variational approximations.
As far as the method is described, I believe it would be impossible to
reproduce their results because of the complexity of the hyper-parameter tuning
performed by the authors. I would encourage the authors to release code that can
directly generate Figure 2 and table 2.


Published as a conference paper at ICLR 2018 LEARNING AWARENESS MODELS
We consider the setting of an agent with a fixed body interacting with an unknown and uncertain external world. We show that models trained to predict proprioceptive information about the agent's body come to represent objects in the external world. In spite of being trained with only internally available signals, these dynamic body models come to represent external objects through the necessity of predicting their effects on the agent's own body. That is, the model learns holistic persistent representations of objects in the world, even though the only training signals are body signals. Our dynamics model is able to successfully predict distributions over 132 sensor readings over 100 steps into the future and we demonstrate that even when the body is no longer in contact with an object, the latent variables of the dynamics model continue to represent its shape. We show that active data collection by maximizing the entropy of predictions about the bodytouch sensors, proprioception and vestibular information-leads to learning of dynamic models that show superior performance when used for control. We also collect data from a real robotic hand and show that the same models can be used to answer questions about properties of objects in the real world. Videos with qualitative results of our models are available at https://goo.gl/mZuqAV.

Summary:
The paper describes a system which creates an internal representation of the scene given observations, being this internal representation advantageous over raw sensory input for object classification and control. The internal representation comes from a recurrent network (more specifically, a sequence2sequence net) trained to maximize the likelihood of the observations from training
Positive aspects:
The authors suggest an interesting hypothesis: an internal representation of the world which is useful for control could be obtained just by forcing the agent to be able to predict the outcome of its actions in the world. This hypothesis would enable robots to train it in a self-supervised manner, which would be extremely valuable.
Negative aspects:
Although the premise of the paper is interesting, its execution is not ideal. The formulation of the problem is unclear and difficult to follow, with a number of important terms left undefined. Moreover, the experiment task is too simplistic; from the results, it's not clear whether the representation is anything more than trivial accumulation of sensory input
- Lack of clarity:
-- what exactly is the "generic cost" C in section 7.1?
-- why are both f and z parameters of C? f is directly a function of z. Given that the form of C is not explained, seems like f could be directly computing as part of C.
-- what is the relation between actions a in section 7.1 and u in section 4?
-- How is the minimization problem of u_{1:T} solved?
-- Are the authors sure that they perform gathering information through "maximizing uncertainty" (section 7.1)? This sounds profoundly counterintuitive. Maximizing the uncertainty in the world state should result in minimum information about the worlds state. I would assume this is a serious typo, but cannot confirm given that the relation between the minimize cost C and the Renyi entropy H is not explicitely stated.
-- When the authors state that "The learner trains the model by maximum likelihood" in section 7.1, do they refer to the prediction model or the control model? It would seem that it is the control model, but the objective being "the same as in section 6" points in the opposite direction
-- What is the method for classifying and/or regressing given the features and internal representation? This is important because, if the method was a recurrent net with memory, the differences between the two representations would probably be minimal.
- Simplistic experimental task:
My main intake from the experiments is that having a recurrent network processing the sensory input provides some "memory" to the system which reduces uncertainty when sensory data is ambiguous. This is visible from the fact that the performance from both systems is comparable at the beginning, but degrades for sensory input when the hand is open. This could be achievable in many simple ways, like modeling the classification/regression problem directly with an LSTM for example. Simpler modalities of providing a memory to the system should be used as a baseline.
Conclusion:
Although the idea of learning an internal representation of the world by being able to predict its state from observations is interesting, the presented paper is a) too simplistic in its experimental evaluation and b) too unclear about its implementation. Consequently, I believe the authors should improve these aspects before the article is valuable to the community


A PAC-BAYESIAN APPROACH TO SPECTRALLY-NORMALIZED MARGIN BOUNDS FOR NEURAL NETWORKS
We present a generalization bound for feedforward neural networks with ReLU activations in terms of the product of the spectral norm of the layers and the Frobenius norm of the weights. The key ingredient is a bound on the changes in the output of a network with respect to perturbation of its weights, thereby bounding the sharpness of the network. We combine this perturbation bound with the PAC-Bayes analysis to derive the generalization bound.

This paper combines a simple PAC-Bayes argument with a simple perturbation analysis (Lemma 2) to get a margin based generalization error bound for ReLU neural networks (Theorem 1) which depends on the product of the spectral norms of the layer parameters as well as their Frobenius norm. The main contribution of the paper is the simple proof technique to derive Theorem 1, much simpler than the one use in the very interesting work [Bartlett et al. 2017] (appearing at NIPS 2017) which got an analogous bound but with a dependence on the l1-norm of the layers instead of the Frobenius norm. The authors make a useful comparison between these bounds in Section 3 showing that none is dominating the others, but still analyzing their properties in terms of structural properties of the weight matrices.
I enjoyed reading this paper. One could think that it makes a somewhat incremental contribution with respect to the more complete work (both theory and practice) from [Bartlett et al. 2017]. Nevertheless, the simplicity and elegance of the proof as well as the result might be useful for the community to get progress on the theoretical analysis of NNs.
The paper is well written, though I make some suggestions for the camera ready version below to improve clarity.
I verified most of the math.
== Detailed suggestions ==
1) The authors should specify in the abstract and in the introduction that they are analyzing feedforward neural networks *with ReLU activation functions* so that the current context of the result is more transparent. It is quite unclear how one could generalize the Theorem 1 to arbitrary activation functions phi given the crucial use of the homogeneity of the ReLU at the beginning of p.4. Though the proof of Lemma 2 only appears to be using the 1-Lipschitzness property of phi as well as phi(0) =0. (Unless they can generalize further; I also suggest that they explicitly state in the (interesting) Lemma 2 that it is for the ReLU activations (like they did in Theorem 1)).
2) A footnote (or citation) could be useful to give a hint on how the inequality 1/e beta^(d-1) <= tilde{beta}^(d-1) <= e beta^(d-1) is proven from the property |beta-tilde{beta}|<= 1/d beta (middle of p.4).
3) Equation (3) -- put the missing 2 subscript for the l2 norm of |f_(w+u)(x) - f_w(x)|_2 on the LHS (for clarity).
4) One extra line of derivation would be helpful for the reader to rederive the bound|w|^2/2sigma^2  <= O(...) just above equation (4). I.e. first doing the expansion keeping the beta terms and Frobenius norm sum, and then going directly to the current O(...) term.
5) bottom of p.4: use hat{L}_gamma = 1 instead of L_gamma =1 for more clarity.
6) Top of p.5: the sentence "Since we need tilde{beta} to satisfy (...)" is currently awkwardly stated. I suggest instead to say that "|tilde{beta}- beta| <= 1/d (gamma/2B)^(1/d) is a sufficient condition to have the needed condition |tilde{beta}-beta| <= 1/d beta over this range, thus we can use a cover of size dm^(1/2d)."
7) Typo below (6): citetbarlett2017...
8) Last paragraph p.5: "Recalling that W_i is *at most* a hxh matrix" (as your result do not require constant size layers and covers the rectangular case).


Published as a conference paper at ICLR 2018 MINIMAX CURRICULUM LEARNING: MACHINE TEACHING WITH DESIRABLE DIFFICULTIES AND SCHEDULED DIVERSITY
We introduce and study minimax curriculum learning (MCL), a new method for adaptively selecting a sequence of training subsets for a succession of stages in machine learning. The subsets are encouraged to be small and diverse early on, and then larger, harder, and allowably more homogeneous in later stages. At each stage, model weights and training sets are chosen by solving a joint continuousdiscrete minimax optimization, whose objective is composed of a continuous loss (reflecting training set hardness) and a discrete submodular promoter of diversity for the chosen subset. MCL repeatedly solves a sequence of such optimizations with a schedule of increasing training set size and decreasing pressure on diversity encouragement. We reduce MCL to the minimization of a surrogate function handled by submodular maximization and continuous gradient methods. We show that MCL achieves better performance and, with a clustering trick, uses fewer labeled samples for both shallow and deep models. Our method involves repeatedly solving constrained submodular maximization of an only slowly varying function on the same ground set. Therefore, we develop a heuristic method that utilizes the previous submodular maximization solution as a warm start for the current submodular maximization process to reduce computation while still yielding a guarantee.

This paper introduces MiniMax Curriculum learning, as an approach for adaptively train models by providing it different subsets of data. The authors formulate the learning problem as a minimax problem which tries to choose diverse example and "hard" examples, where the diversity is captured via a Submodular Loss function and the hardness is captured via the Loss function. The authors formulate the problem as an iterative technique which involves solving a minimax objective at every iteration. The authors argue the convergence results on the minimax objective subproblem, but do not seem to give results on the general problem. The ideas for this paper are built on existing work in Curriculum learning, which attempts to provide the learner easy examples followed by harder examples later on. The belief is that this learning style mimics human learners.
Pros:
- The analysis of the minimax objective is novel and the proof technique introduces several interesting ideas.
- This is a very interesting application of joint convex and submodular optimization, and uses properties of both to show the final convergence results
- Even through the submodular objective is only approximately solvable, it still translates into a convergence result
- The experimental results seem to be complete for the most part. They argue how the submodular optimization does not really affect the performance and diversity seems to empirically bring improvement on the datasets tried.
Cons:
- The main algorithm MCL is only a hueristic. Though the MiniMax subproblem can converge, the authors use this in somewhat of a hueristic manner.
- It seems somewhat hand wavy in the way the authors describe the hyper parameters of MCL, and it seems unclear when the algorithm converge and how to increase/decrease it over iterations
- The objective function also seems somewhat non-intuitive. Though the experimental results seem to indicate that the idea works, I think the paper does not motivate the loss function and the algorithm well.
- It seems to me the authors have experimented with smaller datasets (CIFAR, MNIST, 20NewsGroups). This being mainly an empirical paper, I would have expected results on a few larger datasets (e.g. ImageNet, CelebFaces etc.), particularly to see if the idea also scales to these more real world larger datasets.
Overall, I would like to see if the paper could have been stronger empirically. Nevertheless, I do think there are some interesting ideas theoretically and algorithmically. For this reason, I vote for a borderline accept.


Independent causal mechanisms are a central concept in the study of causality with implications for machine learning tasks. In this work we develop an algorithm to recover a set of (inverse) independent mechanisms relating a distribution transformed by the mechanisms to a reference distribution. The approach is fully unsupervised and based on a set of experts that compete for data to specialize and extract the mechanisms. We test and analyze the proposed method on a series of experiments based on image transformations. Each expert successfully maps a subset of the transformed data to the original domain, and the learned mechanisms generalize to other domains. We discuss implications for domain transfer and links to recent trends in generative modeling.

This paper describes a setting in which a system learns collections of inverse-mapping functions that transform altered inputs to their unaltered "canonical" counterparts, while only needing unassociated and separate sets of examples of each at training time.  Each inverse map is an "expert" E akin to a MoE expert, but instead of using a feed-forward gating on the input, an expert is selected (for training or inference) based on the value of a distribution-modeling function c applied to the output of all experts:  The expert with maximum value c(E(x)) is selected.  When c is an adversarially trained discriminator network, the experts learn to model the different transformations that map altered images back to unaltered ones.  This is demonstrated using MNIST with a small set of synthetic translations and noise.
The fact that these different inverse maps arise under these conditions is interesting --- and Figure 5 is quite convincing in showing how each expert generalizes.  However, I think the experimental conditions are very limited:  Only one collection of transformations is studied, and on MNIST digits only.  In particular, I found the fact that only one of ten transformations can be applied at a time (as opposed to a series of multiple transforms) to be restrictive.  This is touched on in the conclusion, but to me it seems fundamental, as any real-world new example will undergo significantly more complex processes with many different variables all applied at once.
Another direction I think would be interesting, is how few examples are needed in the canonical distribution?  For example, in MNIST, could the canonical distribution P be limited to just one example per digit (or just one example per mode / style of digit, e.g. "2" with loop, and without loop)?  The different handwriters of the digits, and sampling and scanning process, may themselves constitute in-the-wild transformations that might be inverted to single (or few) canonical examples --- Is this possible with this mechanism?
Overall, it is nice to see the different inverse maps arise naturally in this setting.  But I find the single setting limiting, and think the investigation could be pushed further into less restricted settings, a couple of which I mention above.
Other comments:
- c is first described to be any distribution model, e.g. the autoencoder described on p.5.  But it seems that using such a fixed, predefined c like the autoencoder may lead to collapse:  What is preventing an expert from learning a single constant mode that has high c value?  The adversarially trained c doesn't suffer from this, because presumably the discriminator will be able to learn the difference between a single constant mode output and the distribution P.  But if this is the case, it seems a critical part of the system, not a simple implementation choice as the text seems to say.
- The single-net baseline is good, but I'd like to get a clearer picture of its results.  p.8 says this didn't manage to "learn more than one inverse mechanism" --- Does that mean it learns to invert a single mechanism (that is always translates up, for example, when presented an image)?  Or that it learned some mix of transforms that didn't seem to generalize as well?  Or does it have some other behavior?  Also, I'm not entirely clear on how it was trained wrt c --- is argmax(c(E(x)) always just the single expert?  Is c also trained adversarially?  And if so, is the approximate identity initialization used?


Determining the number of latent dimensions is a ubiquitous problem in machine learning. In this study, we introduce a novel method that relies on SVD to discover the number of latent dimensions. The general principle behind the method is to compare the curve of singular values of the SVD decomposition of a data set with the randomized data set curve. The inferred number of latent dimensions corresponds to the crossing point of the two curves. To evaluate our methodology, we compare it with competing methods such as Kaisers eigenvalue-greater-than-one rule (K1), Parallel Analysis (PA), Velicers MAP test (Minimum Average Partial). We also compare our method with the Silhouette Width (SW) technique which is used in different clustering methods to determine the optimal number of clusters.
The result on synthetic data shows that the Parallel Analysis and our method have similar results and more accurate than the other methods, and that our methods is slightly better result than the Parallel Analysis method for the sparse data sets.

The authors propose the use of bootstrapping the data (random sampling entries with replacement) to form surrogate data for which they can evaluate the singular value spectrum of the SVD of the matrix to the singular values of the bootstrapped data, thereby determining the number of latent dimensions in PCA by the point in which the singular values are no greater than the bootstrapped sampled values.  The procedure is contrasted to some existing methods for determining the number of latent components and found to perform similarly to another procedure based on bootstrapping correlation matrices, the PA procedure.
Pros:
Determining the number of components is an important problem that the authors here address.
Cons:
I find the paper poorly written and the methodology not sufficiently rooted in the existing literature. There are many approaches to determining the number of latent components in PCA that needs to be discussed and constrasted including:
Cross-validation:
http://scholar.google.dk/scholar_url?url=http%3A%2F%2Fwww.academia.edu%2Fdownload%2F43416804%2FGeneralizable_Patterns_in_Neuroimaging_H20160306-9605-1xf9c9h.pdf&hl=da&sa=T&oi=gga&ct=gga&cd=0&ei=rjkXWrzKKImMmAH-xo7gBw&scisig=AAGBfm2iRQhmI2EHEO7Cl6UZoRbfAxDRng&nossl=1&ws=1728x1023
Variational Bayesian PCA:
https://www.microsoft.com/en-us/research/publication/variational-principal-components/
Furthermore, the idea of bootstrapping for the SVD has been discussed in prior publications and the present work need to be related to these prior works. This includes:
Milan, Luis, and Joe Whittaker. “Application of the Parametric Bootstrap to Models That Incorporate a Singular Value Decomposition.” Journal of the Royal Statistical Society. Series C (Applied Statistics), vol. 44, no. 1, 1995, pp. 31–49. JSTOR, JSTOR, www.jstor.org/stable/2986193.
Fisher A, Caffo B, Schwartz B, Zipunnikov V. Fast, Exact Bootstrap Principal Component Analysis for p > 1 million. Journal of the American Statistical Association. 2016;111(514):846-860. doi:10.1080/01621459.2015.1062383.
Including the following package in R for performing bootstrapped SVD: https://cran.r-project.org/web/packages/bootSVD/bootSVD.pdf
The novelty of the present approach is therefore unclear given prior works on bootstrapping SVD/PCA.
Furthermore, for sparse data with missing entries there are specialized algorithms handling sparsity either using imputation or marginalization, which would be more principled to estimate the PCA parameters.
Finaly, the performance appears almost identical with the PA procedure. In fact, it seems bootstrapping the correlation matrix has a very similar effect as the proposed bootstrapping procedure. Thus, it seems the proposed procedure which is very similar in spirit to PA does not have much benefit over this procedure.
Minor comments:
Explain what SW abbreviates when introduced first.
We will see that it PA a close relationship with BSVD-> We will see that PA is closely related to BSVD
more effective than SVD under certain conditions (?). – please provide reference instead of ?
But table 4 that shows -> But table 4 shows that
We can sum up with that the result seems ->To summarize, the result seems


Neural networks are increasingly used as a general purpose approach to learning algorithms over graph structured data. However, techniques for representing graphs as real-valued vectors are still in their infancy. Recent works have proposed several approaches (e.g., graph convolutional networks), but as we show in this paper, these methods have difficulty generalizing to large graphs. In this paper we propose GRAPH2SEQ, an embedding framework that represents graphs as an infinite time-series. By not limiting the representation to a fixed dimension, GRAPH2SEQ naturally scales to graphs of arbitrary size. Moreover, through analysis of a formal computational model we show that an unbounded sequence is necessary for scalability. GRAPH2SEQ is also reversible, allowing full recovery of the graph structure from the sequence. Experimental evaluations of GRAPH2SEQ on a variety of combinatorial optimization problems show strong generalization and strict improvement over state of the art.

This paper proposes a novel way of embedding graph structure into a sequence that can have an unbounded length.
There has been a significant amount of prior work (e.g. d graph convolutional neural networks) for signals supported on a specific graph. This paper on the contrary tries to encode the topology of a graph using a dynamical system created by the graph and randomization.
The main theorem is that the created dynamical system can be used to reverse engineer the graph topology for any digraph.
As far as I understood, the authors are doing essentially reverse directed graphical model learning. In classical learning of directed graphical models (or causal DAGs) one wants to learn the structure of a graph from observed data created by this graph inducing conditional independencies on data. This procedure is creating a dynamical system that (following very closely previous work) estimates conditional directed information for every pair of vertices u,v and can find if an edge is present from the observed trajectory.
The recovery algorithm is essentially previous work (but the application to graph recovery is new).
The authors state:
``Estimating conditional directed information efficiently from samples is itself an active area of research Quinn et al. (2011), but simple plug-in estimators with a standard kernel density estimator will be consistent.''
One thing that is missing here is that the number of samples needed could be exponential in the degrees of the graph. Therefore, it is not clear at all that high-dimensional densities or directed information can be estimated from a number of samples that is polynomial in the dimension (e.g. graph degree).
This is related to the second limitation, that there is no sample complexity bounds presented only an asymptotic statement.
One remark is that there are many ways to represent a finite graph with a sequence that can be decoded back to the graph (and of course if there is no bound on the graph size, there will be no bound on the size of the sequence). For example, one could take the adjacency matrix and sequentially write down one row after the other (perhaps using a special symbol to indicate 'next row'). Many other simple methods can be obtained also, with a size of sequence being polynomial (in fact linear) in the size of the graph. I understand that such trivial representations might not work well with RNNs but they would satisfy stronger versions of Theorem 1 with optimal size.
On the contrary it was not clear how the proposed sequence will scale in the graph size.
Another remark is that it seems that GCNN and this paper solve different problems.
GCNNs want to represent graph-supported signals (on a fixed graph) while this paper tries to represent the topology of a graph, which seems different.
The experimental evaluation was somewhat limited and that is the biggest problem from a practical standpoint. It is not clear why one would want to use these sequences for solving MVC. There are several graph classification tasks that try to use the graph structure (as well as possibly other features) see eg the bioinformatics
and other applications. Literature includes for example:
Graph Kernels by S.V.N. Vishwanathan et al.
Deep graph kernels (Yanardag & Vishwanathan and graph invariant kernels (Orsini et al.),
which use counts of small substructures as features.
The are many benchmarks of graph classification tasks where the proposed representation could be useful but significantly more validation work would be needed to make that case.


Published as a conference paper at ICLR 2018 WORD TRANSLATION WITHOUT PARALLEL DATA
State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or EnglishChinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available 1 .

This paper presents a new method for obtaining a bilingual dictionary, without requiring any parallel data between the source and target languages. The method consists of an adversarial approach for aligning two monolingual word embedding spaces, followed by a refinement step using frequent aligned words (according to the adversarial mapping). The approach is evaluated on single word translation, cross-lingual word similarity, and sentence translation retrieval tasks.
The paper presents an interesting approach which achieves good performance. The work is presented clearly, the approach is well-motivated and related to previous studies, and a thorough evaluation is performed.
My one concern is that the supervised approach that the paper compares to is limited: it is trained on a small fixed number of anchor points, while the unsupervised method uses significantly more words. I think the paper's comparisons are valid, but the abstract and introduction make very strong claims about outperforming "state-of-the-art supervised approaches". I think either a stronger supervised baseline should be included (trained on comparable data as the unsupervised approach), or the language/claims in the paper should be softened. The same holds for statements like "... our method is a first step ...", which is very hard to justify. I also do not think it is necessary to over-sell, given the solid work in the paper.
Further comments, questions and suggestions:
- It might be useful to add more details of your actual approach in the Abstract, not just what it achieves.
- Given you use trained word embeddings, it is not a given that the monolingual word embedding spaces would be alignable in a linear way. The actual word embedding method, therefore, has a big influence on performance (as you show). Could you comment on how crucial it would be to train monolingual embedding spaces on similar domains/data with similar co-occurrence statistics, in order for your method to be appropriate?
- Would it be possible to add weights to the terms in eq. (6), or is this done implicitly?
- How were the 5k source words for Procrustes supervised baseline selected?
- Have you considered non-linear mappings, or jointly training the monolingual word embeddings while attempting the linear mapping between embedding spaces?
- Do you think your approach would benefit from having a few parallel training points?
Some minor grammatical mistakes/typos (nitpicking):
- "gives a good performance" -> "gives good performance"
- "Recent works", "several works", "most works", etc. -> "recent studies", "several studies", etc.
- "i.e, the improvements" -> "i.e., the improvements"
The paper is well-written, relevant and interesting. I therefore recommend that the paper be accepted.


Inspired by the success of generative adversarial networks (GANs) in image domains, we introduce a novel hierarchical architecture for learning characteristic topological features from a single arbitrary input graph via GANs. The hierarchical architecture consisting of multiple GANs preserves both local and global topological features, and automatically partitions the input graph into representative stages for feature learning. The stages facilitate reconstruction and can be used as indicators of the importance of the associated topological structures. Experiments show that our method produces subgraphs retaining a wide range of topological features, even in early reconstruction stages. This paper contains original research on combining the use of GANs and graph topological analysis.

Quality: The work has too many gaps for the reader to fill in. The generator (reconstructed matrix) is supposed to generate a 0-1 matrix (adjacency matrix) and allow backpropagation of the gradients to the generator. I am not sure how this is achieved in this work. The matrix is not isomorphic invariant and the different clusters don’t share a common model. Even implicit models should be trained with some way to leverage graph isomorphisms and pattern similarities between clusters. How can such a limited technique be generalizing? There is no metric in the results showing how the model generalizes, it may be just overfitting the data.
Clarity: The paper organization needs work; there are also some missing pieces to put the NN training together. It is only in Section 2.3 that the nature of G_i^\prime becomes clear, although it is used in Section 2.2. Equation (3) is rather vague for a mathematical equation. From what I understood from the text, equation (3) creates a binary matrix from the softmax output using an indicator function. If the output is binary, how can the gradients backpropagate? Is it backpropagating with a trick like the Gumbel-Softmax trick of Jang, Gu, and Poole 2017 or Bengio’s path derivative estimator? This is a key point not discussed in the manuscript.
And if I misunderstood the sentence “turn re_G into a binary matrix” and the values are continuous, wouldn’t the discriminator have an easy time distinguishing the generated data from the real data. And wouldn’t the generator start working towards vanishing gradients in its quest to saturate the re_G output?
Originality: The work proposes an interesting approach: first cluster the network, then learning distinct GANs over each cluster. There are many such ideas now on ArXiv but it would be unfair to contrast this approach with unpublished work. There is no contribution in the GAN / neural network aspect. It is also unclear whether the model generalizes. I don’t think this is a good fit for ICLR.
Significance: Generating graphs is an important task in in relational learning tasks, drug discovery, and in learning to generate new relationships from knowledge bases. The work itself, however, falls short of the goal. At best the generator seems to be working but I fear it is overfitting. The contribution for ICLR is rather minimal, unfortunately.
Minor:
GTI was not introduced before it is first mentioned in the into.
Y. Bengio, N. Leonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.3432, 2013.


Published as a conference paper at ICLR 2018 i-REVNET: DEEP INVERTIBLE NETWORKS
It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. An analysis of i-RevNets learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.

In this paper, the authors propose deep architecture that preserves mutual information between the input and the hidden representation and show that the loss of information can only occur at the final layer. They illustrate empirically that the loss of information can be avoided on large-scale classification such as ImageNet and propose to build an invertible deep network that is capable of retaining the information of the input signal through all the layers of the network until the last layer where the input could be reconstructed.
The authors demonstrate that progressive contraction and separation of the information can be obtained while at the same time allowing an exact reconstruction of the signal.
As it requires a special care to design an invertible architecture, the authors architecture is based on the recent reversible residual network (RevNet) introduced in (Gomez et al., 2017) and an invertible down-sampling operator introduced in (Shi et al., 2016). The inverse (classification) path of the network uses the same convolutions as the forward (reconstructing) one. It also uses subtraction operations instead of additions in the output computation in order to reconstruct intermediate and input layers.
To show the effectiveness of their approach on large-scale classification problem, the authors report top-1 error rates on the validation set of ILSVRC-2012. The obtained result is competitive with the original Resnet and the RevNet models. However, the proposed approach is expensive in terms of parameter budget as it requires almost 6.5 times more parameters than the RevNet and the Resnet architectures. Still, the classification and the reconstructing results are quite impressive as the work is the first empirical evidence that learning invertible representation that preserves information about the input is possible on large-scale classification tasks. Worth noting that recently, (Shwartz-Ziv and Tishby) demonstrated, not on large-scale datasets but on small ones, that an optimal representation for a classification task must reduce as much uninformative variability as possible while maximizing the mutual information between the desired output and its representation in order discriminate as much as possible between classes. This is called “information bottleneck principle”. The submitted paper shows that this principle is not a necessary condition large-scale classification.
The proposed approach is potentially of great benefit. It is also simple and easy to understand. The paper is well written and the authors position their work with respect to what has been done before. The spectral analysis of the differential operator in section 4.1 provide another motivation for the “hard-constrained” invertible architecture. Section 4.2 illustrates the ability of the network to reconstruct input signals. The visualization obtained suggests that network performs linear separation between complex learned factors. Section 5 shows that even when using either an SVM or a Nearest Neighbor classifier on n extracted features from a layer in the network, both classifiers progressively improve with deeper layers. When the d first principal components are used to summarize the n extracted features, the SVM and NN classifier performs better when d is bigger. This shows that the deeper the network gets, the more linearly separable and contracted the learned representations are.
In the conclusion, the authors state the following: “The absence of loss of information is surprising, given the wide believe, that discarding information is essential for learning representations that generalize well to unseen data”. Indeed, the authors have succeed in showing that this is not necessarily the case. However, the loss of information might be necessary to generalize well on unseen data and at the same time minimize the parameter budget for a given classification task.


A fundamental challenge in designing brain-computer interfaces (BCIs) is decoding behavior from time-varying neural oscillations. In typical applications, decoders are constructed for individual subjects and with limited data leading to restrictions on the types of models that can be utilized. Currently, the best performing decoders are typically linear models capable of utilizing rigid timing constraints with limited training data. Here we demonstrate the use of Long Short-Term Memory (LSTM) networks to take advantage of the temporal information present in sequential neural data collected from subjects implanted with electrocorticographic (ECoG) electrode arrays performing a finger flexion task. Our constructed models are capable of achieving accuracies that are comparable to existing techniques while also being robust to variation in sample data size. Moreover, we utilize the LSTM networks and an affine transformation layer to construct a novel architecture for transfer learning. We demonstrate that in scenarios where only the affine transform is learned for a new subject, it is possible to achieve results comparable to existing state-of-the-art techniques. The notable advantage is the increased stability of the model during training on novel subjects. Relaxing the constraint of only training the affine transformation, we establish our model as capable of exceeding performance of current models across all training data sizes. Overall, this work demonstrates that LSTMs are a versatile model that can accurately capture temporal patterns in neural data and can provide a foundation for transfer learning in neural decoding.

The paper describes an approach to use LSTM’s for finger classification based on ECOG. and a transfer learning extension of which two variations exists. From the presented results, the LSTM model is not an improvement over a basic linear model. The transfer learning models performs better than subject specific models on a subset of the subjects. Overall, I think the problem Is interesting but the technical description and the evaluation can be improved. I am not confident in the analysis of the model. Additionally, the citations are not always correct and some related work is not referenced at all. For the reasons above, I am not willing to recommend the paper for acceptance at his point.
The paper tackles a problem that is challenging and interesting. Unfortunately, the dataset size is limited.
This is common for brain data and makes evaluation much more difficult.
 The paper states that all hyper-parameters were optimized on 75% of subject B data.
The actual model training was done using cross-validation.
So far this approach seems more or less correct but in this case I would argue that subject B should not be considered for evaluation since its data is heavily used for hyper-parameter optimization and the results obtained on this subject are at risk of being biased.
Omitting subject B from the analysis, each non-transfer learning method  performs best on one of the remaining subjects.
Therefore it is not clear that an LSTM model is an improvement.
For transfer learning (ignoring B again) only C and D are improved but it is unclear what the variance is.
In the BCI community there are many approaches that use transfer learning with linear models. I think that it would be interesting how linear model transfer learning would fare in this task.
A second issue that might inflate the results is the fact that the data is shuffled randomly. While this is common practice for most machine learning tasks, it is dangerous when working with brain data due to changes in the signal over time. As a result, selecting random samples might inflate the accuracy compared to having a proper train and test set that are separated in time. Ideally the cross-validation should be done using contiguous folds.
I am not quite sure whether it should be possible to have an accuracy above chance level half a second before movement onset? How long does motor preparation take? I am not familiar with this specific subject, but a quick search gave me a reaction time for sprinters of .15 seconds. Is it possible that cue processing activity was used to obtain the classification result? Please discuss this effect because I am do not understand why it should be possible to get above chance level accuracy half a second before movement onset.
There are also several technical aspects that are not clear to me. I am confident that I am unable to re-implement the proposed method and their baseline given the information provided.
LDA baseline:
—————————
For the LDA baseline, how is the varying sequence length treated?
Ledoit wolf analytic  regularization is used, but it isn not referenced. If you use that method, cite the paper.
The claim that LDA works for structured experimental tasks but not in naturalistic scenarios and will not generalize when electrode count and trial duration increases is a statement that might be true. However, it is never empirically verified.  Therefore this statement should not be in the paper.
HMM baseline
—————————
How are the 1 and the 2 state HMM used w.r.t. the 5 classes? It is unclear to me how they are used exactly. Is there a single HMM per class? Please be specific.
LSTM Model
—————
What is the random and language model initialization scheme? I can only find the sequence auto-encoder in the Dai and Le paper.
Model analysis
——————————-
It is widely accepted in the neuroimaging community that linear weight vectors should not be interpreted directly. It is actually impossible to do this.  Therefore this section should be completely re-done. Please read the following paper on this subject.
Haufe, Stefan, et al. "On the interpretation of weight vectors of linear models in multivariate neuroimaging." Neuroimage 87 (2014): 96-110.
References
————
Ledoit wolf regularization is used but not cited. Fix this.
There is no citation for the random/language model initialization of the LSTM model. I have no clue how to do this without proper citation.
Le at al (2011) are referenced for auto-encoders. This is definitely not the right citation.
Rumelhart, Hinton, & Williams, 1986a; Bourlard & Kamp, 1988; Hinton & Zemel, 1994 and Bengio, Lamblin, Popovici, & Larochelle, 2007; Ranzato, Poultney, Chopra, & LeCun, 2007 are probably all more relevant.
Please cite the relevant work on affine transformations for transfer learning especially the work by morioka et al who also learn an input transferm.
Morioka, Hiroshi, et al. "Learning a common dictionary for subject-transfer decoding with resting calibration." NeuroImage 111 (2015): 167-178.


INVESTIGATING HUMAN PRIORS FOR PLAYING VIDEO GAMES
What makes humans so good at solving seemingly complex video games? Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efficient decision making. This paper investigates the role of human priors for solving video games. Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors. We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors. We find that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes. Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efficient game-play.

Overall:
I really enjoyed reading this paper and think the question is super important. I have some reservations about the execution of the experiments as well as some of the conclusions drawn. For this reason I am currently a weak reject (weak because I believe the question is very interesting). However, I believe that many of my criticisms can be assuaged during the rebuttal period.
Paper Summary:
For RL to play video games, it has to play many many many many times. In fact, many more times than a human where prior knowledge lets us learn quite fast in new (but related) environments. The authors study, using experiments, what aspects of human priors are the important parts.
The authors’ Main Claim appears to be: “While common wisdom might suggest that prior knowledge about game semantics such as ladders are to be climbed, jumping on spikes is dangerous or the agent must fetch the key before reaching the door are crucial to human performance, we find that instead more general and high-level priors such as the world is composed of objects, object like entities are used as subgoals for exploration, and things that look the same, act the same are more critical.”
Overall, I find this interesting. However, I am not completely convinced by some of the experimental demonstrations.
Issue 0: The experiments seem underpowered / not that well analyzed.
There are only 30 participants per condition and so it’s hard to tell whether the large differences in conditions are due to noise and what a stable ranking of conditions actually looks like. I would recommend that the authors triple the sample size and be more clear about reporting the outcomes in each of the conditions.
It’s not clear what the error bars in figure 1 represent, are they standard deviations of the mean? Are they standard deviations of the data? Are they confidence intervals for the mean effect?
Did you collect any extra data about participants? One potentially helpful example is asking how familiar participants are with platformer video games. This would give at least some proxy to study the importance of priors about “how video games are generally constructed” rather than priors like “objects are special”.
Issue 1: What do you mean by “objects”?
The authors interpret the fact that performance falls so much between conditions b and c to mean that human priors about “objects are special” are very important. However, an alternative explanation is that people explore things which look “different” (ie. Orange when everything else is black).
The problem here comes from an unclear definition of what the authors mean by an “object” so in revision I would like authors to clarify what precisely they mean by a prior about “the world is composed of objects” and how this particular experiment differentiates “object” from a more general prior about “video games have clearly defined goals, there are 4 clearly defined boxes here, let me try touching them.”
This is important because a clear definition will give us an idea for how to actually build this prior into AI systems.
Issue 2: Are the results here really about “high level” priors?
There are two ways to interpret the authors’ main claim: the strong version would maintain that semantic priors aren’t important at all.
There is no real evidence here for the strong version of the claim. A real test would be to reverse some of the expected game semantics and see if people perform just as well as in the “masked semantics” condition.
For example, suppose we had exactly the same game and N different types of objects in various places of the game where N-1 of them caused death but 1 of them opened the door (but it wasn’t the object that looked like a key). My hypothesis would be that performance would fall drastically as semantic priors would quickly lead people in that direction.
Thus, we could consider a weaker version of the claim: semantic priors are important but even in the absence of explicit semantic cues (note, this is different from having the wrong semantic cues as above) people can do a good job on the game. This is much more supported by the data, but still I think very particular to this situation. Imagine a slight twist on the game:
There is a sword (with a lock on it), a key, a slime and the door (and maybe some spikes). The player must do things in exactly this order: first the player must get the key, then they must touch the sword, then they must kill the slime, then they go to the door. Here without semantic priors I would hypothesize that human performance would fall quite far (whereas with semantics people would be able to figure it out quite well).
Thus, I think the authors’ claim needs to be qualified quite a bit. It’s also important to take into account how much work general priors about video game playing (games have goals, up jumps, there is basic physics) are doing here (the authors do this when they discuss versions of the game with different physics).


Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms. In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization with pruning to create blocks of weights with zeros. Using these techniques, we can create block-sparse RNNs with sparsity ranging from 80% to 90% with a small loss in accuracy. This technique allows us to reduce the model size by roughly 10×. Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count. Our technique works with a variety of block sizes up to 32×32. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

Thanks to the authors for their response.
Though the paper presents an interesting approach, but it relies heavily on heuristics (such as those mentioned in the initial review) without a thorough investigation of scenarios in which this might not work. Also, it might be helpful to investigate if there ways to better group the variables for group  lasso regularization. The paper therefore needs further improvements towards following a more principled approach.
=====================================
This paper presents methods for inducing sparsity in terms of blocks of weights in neural networks which aims to combine benefits of sparsity and faster access based on computing architectures. This is achieved by pruning blocks of weights and group lasso regularization. It is demonstrated empirically that model size can be reduced by upto 10 times with some loss in prediction accuracy.
Though the paper presents some interesting evaluations on the impact of block based sparsity in RNNs, some of the shortcomings of the paper seem to be :
- The approach taken consists of several heuristics rather than following a more principled approach such as taking the maximum of the weights in a block to represent that block and stop pruning till 40% training has been achieved. Also, the algorithm for computing the pruning threshold is based on a new set of hyper-parameters. It is not clear under what conditions the above settings will (not) work.
 - For the group lasso method, since there are many ways to group the variable, it is not clear how the variables are grouped. Is there a reasoning behind a particular grouping of the variables. Individually, group lasso does not seem to work, and gives much worse results. The reasons for worse performance could be investigated. It is possible that important weights are in different groups, and group sparsity is forcing some of them to be zero, and hence leading to worse results. It would be insightful to explain the kind of solver used for group lasso regularization, and if that works for large-scale problems.
 - The results for various kinds of sparsity are unclear in the sense that it is not clear how to set the block size a-priori for having minimum reduction in accuracy and still significant sparsity without having to repeat the process for various choices.
Overall, the paper does not seem to present novel ideas, and is mainly focused on evaluating the impact of block-based sparsity instead of weight pruning by Han etal. As mentioned in Section 2, regularization has been used earlier to achieve sparsity in deep networks. In this view the significance over existing work is relatively narrow, and no explicit comparison with existing methods is provided. It is possible that an existing method leads to pruning method such as by Han etal. leads to 8x decrease in model size while retaining the accuracy, while the proposed method leads to 10x decrease while also decreasing the accuracy by 10%. Scenarios like these need to be evaluated to understand the impact of the method proposed in this paper.


We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn 'distributional similarity' in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space. To validate our hypothesis, we focus on the 'image' side of image captioning, and vary the input image representation but keep the RNN text generation model of a CNN-RNN constant. We propose a sparse bag-of-objects vector as an interpretable representation to investigate our distributional similarity hypothesis. We found that image captioning models (i) are capable of separating structure from noisy input representations; (ii) experience virtually no significant performance loss when a high dimensional representation is compressed to a lower dimensional space; (iii) cluster images with similar visual and linguistic information together; (iv) are heavily reliant on test sets with a similar distribution as the training set; (v) repeatedly generate the same captions by matching images and 'retrieving' a caption in the joint visual-textual space. Our experiments all point to one fact: that our distributional similarity hypothesis holds. We conclude that, regardless of the image representation, image captioning systems seem to match images and generate captions in a learned joint image-text semantic subspace.

This paper analyzes the effect of image features on image captioning. The authors propose to use a model similar to that of Vinyals et al., 2015 and change the image features it is conditioned on. The MSCOCO captioning and Flickr30K datasets are used for evaluation.
Introduction
- The introduction to the paper could be made clearer - the authors talk about the language of captioning datasets being repetitive, but that fact is neither used or discussed later.
- The introduction also states that the authors will propose ways to improve image captioning. This is never discussed.
Captioning Model and Table 1
- The authors use greedy (argmax) decoding which is known to result in repetitive captions. In fact, Vinyals et al. note this very point in their paper. I understand this design choice was made to focus more on the image side, rather than the decoding (language) side, but I find it to be very limiting. In this regime of greedy decoding it is hard to see any difference between the different ConvNet features used for captioning - Table 1 shows meteor scores within 0.19 - 0.22 for all methods.
- Another effect (possibly due to greedy decoding + choice of model), is that the numbers in Table 1 are rather low compared to the COCO leaderboard. The top 50 entries have METEOR scores >= 0.25, while the maximum METEOR score reported by the authors is 0.22. Similar trend holds for other metrics like BLEU-4.
- The results of Table 5 need to be presented and interpreted in the light of this caveat of greedy decoding.
Experimental Setup and Training Details
- How was the model optimized? No training details are provided. Did you use dropout? Were hyperparamters fixed for training across different feature sizes of VGG19 and ResNet-152? What is the variance in the numbers for Table 1?
Main claim of the paper
Devlin et al., 2015 show a simple nearest neighbor baseline which in my opinion shows this more convincingly. Two more papers from the same group which use also make similar observations - tweaking the image representation makes image captioning better: (1) Fang et al., 2015: Multiple-instance Learning using bag-of-objects helps captioning (2) Misra et al. 2016 (not cited): label noise can be modeled which helps captioning. This claim has been both made and empirically demonstrated earlier.
Metrics for evaluation
- Anderson et al., 2016 (not cited) proposed the SPICE metric and also showed how current metrics including CiDER may not be suitable for evaluating image captions. The COCO leaderboard also uses this metric as one of its evaluation metrics. If the authors are evaluating on the test set and reporting numbers, then it is odd that they `skipped' reporting SPICE numbers.
Choice of Datasets
- If we are thoroughly evaluating the effect of image features, doing so on other datasets is very important. Visual Genome (Krishnan et al., not cited) and SIND (Huang et al., not cited) are two datasets which are both larger than Flickr30k and have different image distributions from MSCOCO. These datasets should show whether using more general features (YOLO-9k) helps.
The authors should evaluate on these datasets to make their findings stronger and more valuable.
Minor comments
- Figure 1 is hard to read on paper. Please improve it.
- Figure 2 is hard to read even on screen. It is really interesting, so improving the quality of this figure will really help.