Published as a conference paper at ICLR 2018 IMPROVING THE IMPROVED TRAINING OF WASSERSTEIN GANS: A CONSISTENCY TERM AND ITS DUAL EFFECT
Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difficult to train. This issue is formally analyzed by Arjovsky & Bottou (2017), who also propose an alternative direction to avoid the caveats in the minmax two-player training of GANs. The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the 1-Lipschitz continuity of the discriminator. In this paper, we propose a novel approach to enforcing the Lipschitz continuity in the training procedure of WGANs. Our approach seamlessly connects WGAN with one of the recent semi-supervised learning methods. As a result, it gives rise to not only better photo-realistic samples than the previous methods but also state-of-the-art semi-supervised learning results. In particular, our approach gives rise to the inception score of more than 5.0 with only 1,000 CIFAR-10 images and is the first that exceeds the accuracy of 90% on the CIFAR-10 dataset using only 4,000 labeled images, to the best of our knowledge.

Summary:
The paper proposes a new regularizer for wgans, to be combined with the traditional gradient penalty. The theoretical motivation is bleak, and the analysis contains some important mistakes. The results are very good, as noticed by the comments, the fact that the method is also less susceptible to overfitting is also an important result, though this might be purely due to dropout. One of the main problems is that the largest dataset used is CIFAR, which is small. Experiments on something like bedrooms or imagenet would make the paper much stronger.
If the authors fix the theoretical analysis and add evidence in a larger dataset I will raise the score.
Detailed comments:
- The motivation of 1.2 and the sentence "Arguably, it is fairly safe to limit our scope to the manifold that supports the real data distribution P_r and its surrounding regions" are incredibly wrong. First of all, it should be noted that the duality uses 1-Lip in the entire space between Pr and Pg, not in Pr alone. If the manifolds are not extremely close (such as in the beginning of training), then the discriminator can be almost exactly 1 in the real data, and 0 on the fake. Thus the discriminator would be almost exactly constant (0-Lip) near the real manifold, but will fail to be 1-lip in the decision boundary, this is where interpolations fix this issue. See Figure 2 of the wgan paper for example, in this simple example an almost perfect discriminator would have almost 0 penalty.
- In the 'Potential caveats' section, the implication that 1-Lip may not be enforced in non-examined samples is checkable by an easy experiment, which is to look for samples that have gradients of the critic wrt the input with norm > 1. I performed the exp in figure 8 and saw that by taking a slightly higher lambda, one reaches gradients that are as close to 1 as with ct-gan. Since ct-gan uses an extra regularizer, I think the authors need some stronger evidence to support the claim that ct-gan better battles this 'potential caveat'.
- It's important to realize that the CT regularizer with M' = 1 (1-Lip constraint) will only be positive for an almost 1-Lip function if x and x' are sampled when x - x' has a very similar direction than the gradient at x. This is very hard in high dimensional spaces, and when I implemented a CT regularizer indeed the ration of eq (4) was quite less than the norm of the gradient. It would be useful to plot the value of the CT regularizer (the eq 4 version) as the training iterations progresses. Thus the CT regularizer works as an overall Lipschitz penalty, as opposed to penalizing having more than 1 for the Lipschitz constant. This difference is non-trivial and should be discussed.
- Line 11 of the algorithm is missing L^(i) inside the sum.
- One shouldn't use MNIST for anything else than deliberately testing an overfitting problem. Figure 4 is thus relevant, but the semi-supervised results of MNIST or the sample quality experiments give hardly any evidence to support the method.
- The overfitting result is very important, but one should disambiguate this from being due to dropout. Comparing with wgangp + dropout is thus important in this experiment.
- The authors should provide experiments in at least one larger dataset like bedrooms or imagenet (not faces, which is known to be very easy). This would strengthen the paper quite a bit.


We propose GraphGAN -the first implicit generative model for graphs that enables to mimic real-world networks. We pose the problem of graph generation as learning the distribution of biased random walks over a single input graph. Our model is based on a stochastic neural network that generates discrete output samples, and is trained using the Wasserstein GAN objective. GraphGAN enables us to generate sibling graphs, which have similar properties yet are not exact replicas of the original graph. Moreover, GraphGAN learns a semantic mapping from the latent input space to the generated graph's properties. We discover that sampling from certain regions of the latent space leads to varying properties of the output graphs, with smooth transitions between them. Strong generalization properties of GraphGAN are highlighted by its competitive performance in link prediction as well as promising results on node classification, even though not specifically trained for these tasks.

This paper proposes a WGAN formulation for generating graphs based on random walks. The proposed generator model combines node embeddings, with an LSTM architecture for modeling the sequence of nodes visited in a random walk; the discriminator distinguishes real from fake walks.
The model is learned from a single large input graph (for three real-world networks) and evaluated against one baseline generative graph model: degree-corrected stochastic block models.
The primary claims of the paper are as follows:
i) The proposed approach is a generative model of graphs, specifically producing "sibling" graphs
ii) The learned latent representation provides an interpretation of generated graph properties
iii) The model generalizes well in terms of link and node classification
The proposed method is novel and the incorporated ideas are quite interesting (e.g., discriminating real from fake random walks, generating random walks from node embeddings and LSTMs). However, from a graph generation perspective, the problem formulation and evaluation do not sufficiently demonstrate the utility of proposed method.
First, wrt claim (i) the problem of generating "sibling" graphs is ill-posed. Statistical graph models are typically designed to generate a probability distribution over all graphs with N nodes and, as such, are evaluated based on how well they model that distribution. The notion of a "sibling" graph used in this paper is not clearly defined, but it seems to only be useful if the sibling graphs are likely under the distribution. Unfortunately, the likelihood of the sampled graphs is not explicitly evaluated. On the other hand, since many of the edges are shared the "siblings" may be nearly isomorphic to the input graph, which is not useful from a graph modeling perspective.
For claim (i), the comparison to related work is far from sufficient to demonstrate its utility as a graph generation model. There are many graph models that are superior to DC-SBM, including KPGMs, BETR, ERGMs, hierarchical random graph models and latent space models. Moreover, a very simple baseline to assess the LSTM component of the model, would be to produce a graph by sampling links repeatedly from the latent space of node embeddings.
Next, the evaluation wrt to claim (ii) is novel and may help developers understand the model characteristics. However, since the properties are measured based on a set of random walks it is still difficult to interpret the impact on the generated graphs (since an arbitrary node in the final graph will have some structure determined from each of the regions). Do the various regions generate different parts of the final graph structure (i.e., focusing on only a subset of the nodes)?
Lastly, the authors evaluate the learned model on link and node prediction tasks and state that the model's so-so performance supports the claim that the model can generalize. This is the weakest claim of the paper. The learned node embeddings appear to do significantly worse than node2vec, and the full model is worse than DC-SBM. Given that the proposed model is transductive (when there is significant edge overlap) it should do far better than DC-SBM which is inductive.
Overall, while the paper includes a wide range of experimental evaluation, they are aimed too broadly (and the results are too weak) to support any specific claim of the work. If the goal is to generate transductively (with many similar edges), then it would be better to compare more extensively to alternative node embedding and matrix factorization approaches, and assess the utility of the various modeling choices (e.g., LSTM, in/out embedding). If the goal is to generate inductively, over the full distribution of graphs, then it would be better to (i) assess whether the sampled graphs are isomorphic, and (ii) compare more extensively to alternative graph models (many of which have been published since 2010).


Published as a conference paper at ICLR 2018 REINFORCEMENT LEARNING ON WEB INTERFACES USING WORKFLOW-GUIDED EXPLORATION
Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates. This has been a notable problem in training deep RL agents to perform web-based tasks, such as booking flights or replying to emails, where a single mistake can ruin the entire sequence of actions. A common remedy is to "warmstart" the agent by pre-training it to mimic expert demonstrations, but this is prone to overfitting. Instead, we propose to constrain exploration using demonstrations. From each demonstration, we induce high-level "workflows" which constrain the allowable actions at each time step to be similar to those in the demonstration (e.g., "
Step 1: click on a textbox; Step 2: enter some text"). Our exploration policy then learns to identify successful workflows and samples actions that satisfy these workflows. Workflows prune out bad exploration directions and accelerate the agent's ability to discover rewards. We use our approach to train a novel neural policy designed to handle the semi-structured nature of websites, and evaluate on a suite of web tasks, including the recent World of Bits benchmark. We achieve new state-of-the-art results, and show that workflow-guided exploration improves sample efficiency over behavioral cloning by more than 100x.

Summary:
The authors propose a method to make exploration in really sparse reward tasks more efficient. They propose a method called Workflow Guided Exploration (WGE) which is learnt from demonstrations but is environment agnostic. Episodes are generated by first turning demonstrations to a workflow lattice. This lattice encodes actions which are in some sense similar to those in the demonstration. By rolling out episodes which are randomly sampled from this set of similar actions for each encountered state, it is claimed that other methods like Behavor Cloning + RL (BC-then-RL) can be outperformed in terms of number of sample complexity since high reward episodes can be sampled with much higher probability.
A novel NN architecture (DOMNet) is also presented which can embed structured documents like HTML webpages.
Comments:
- The paper is well-written and relevant literature is cited and discussed.
- My main concern is that while imitation learning and inverse reinforcement learning are mentioned and discussed in related work section as classes of algorithms for incorporating prior information there is no baseline experiment using either of these methods. Note that the work of Ross and Bagnell, 2010, 2011 (cited in the paper) establish theoretically that Behavior Cloning does not work in such situations due to the non-iid data generation process in such sequential decision-making settings (the mistakes grow quadratically in the length of the horizon). Their proposed algorithm DAgger fixes this (the mistakes by the policy are linear in the horizon length) by using an iterative procedure where the learnt policy from the previous iteration is executed and expert demonstrations on the visited states are recorded, the new data thus generated is added to the previous data and a new policy retrained. Dagger and related methods like Aggrevate provide sample-efficient ways of exploring the environment near where the initial demonstrations were given. WGE is aiming to do the same: explore near demonstration states.
- The problem with putting in the replay buffer only episodes which yield high reward is that extrapolation will inevitably lead the learnt policy towards parts of the state space where there is actually low reward but since no support is present the policy makes such mistakes.
- Therefore would be good to have Dagger or a similar imitation learning algorithm be used as a baseline in the experiments.
- Similar concerns with IRL methods not being used as baselines.
Update: Review score updated after discussion with authors below.


We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance. We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and provide a concrete formulation for doing so.

`The papers aims to provide a quality measure/test for GANs.   The objective is ambitious an deserve attention. As GANs are minimizing some f-divergence measure, the papers remarks that computing a  Wasserstein distance between two distributions made of a sum of Diracs is not a degenerate case and is tractable. So they propose evaluate the current approximation of a distribution learnt by a GAN by using this distance as a baseline performance (in terms of W distance and computed on a hold out dataset).
A first remark is that the papers does not clearly develop the interest of puting things a trying to reach a treshold of performance in W distance rather than just trying to minimize the desired f-divergence. More specifically as they assess the performance in terms of W distance I would would be tempted to just minimize the given criterion. This would be very interesting to have arguments on why being better than the "Dirac estimation" in terms of W2 distance would lead to better performance for others tasks (as other f-divergences or image generation).
According to the authors the core claims are:
"1/ We suggest a formalisation of the goal of GAN training (/generative modelling more broadly) in terms of divergence minimisation. This leads to a natural, testable notion of generalisation. "
Formalization in terms of divergence minimization is not new (see O. Bousquet & all https://arxiv.org/pdf/1701.02386.pdf ) and I do not feel like this paper actually performs any "test" (in a statistical sense). In my opinion the contribution is more about exhibiting a baseline which has to be defeated for any algorithm interesting is learning the distribution in terms of W2 distance.
"2/ We use this test to evaluate the success of GAN algorithms empirically, with the Wasserstein distance as our divergence."
Here the distance does not seems so good because the performance in generation does not seems to only be related to W2 distance. Nevertheless, there is interesting observations in the paper about the sensitivity of this metric to the bluring of pictures. I would enjoyed more digging in this direction. The authors proposes to solve this issue by relying to an embedded space where the L2 distance makes more sense for pictures (DenseNet). This is of course very reasonable but I would expect anyone working on distribution over picture to work with such embeddings. Here I'm not sure if this papers opens a new way to improve the embedding making use on non labelled data. One could think about allowing the weights of the embeddings to vary while f-divergence is minimized but this is not done in the submitted work.
 "3/ We find that whether our proposed test matches our intuitive sense of GAN quality depends heavily on the ground metric used for the Wasserstein distance."
This claim is highly biased by who is giving the "intuitive sense". It would be much better evaluated thought a mechanical turk test.
 "4/ We discuss how to use these insights to improve the design of WGANs more generally."
As our understanding of the GANs dynamics are very coarse, I feel this is not a good thing to claim that "doing xxx should improve things" without actually trying it.


Recurrent neural networks (RNNs) are powerful models for sequential data. They can approximate arbitrary computations, and have been used successfully in domains such as text and speech. However, the flexibility of RNNs makes them susceptible to overfitting and regularization is important. We develop a noise-based regularization method for RNNs. The idea is simple and easy to implement: we inject noise in the hidden units of the RNN and then maximize the original RNN's likelihood averaged over the injected noise. On a language modeling benchmark, our method achieves better performance than its deterministic RNN and variational dropout (Gal and Ghahramani, 2016).

The RNN transition function is: h_t+1 = f(h_t,x_t)
This paper proposes using a stochastic transition function instead of a deterministic one.
i.e h_{t+1} \sim expfam(mean = f(h_t,x_t), gamma) where expfam denotes a distribution from the exponential family.
The experimental results consider text modeling (evaluating on perplexity) on Penn Treebank and Wikitext-2. The method of regularization is compared to a reimplementation of Variational Dropout and no regularization.
The work is written clearly and easy to follow.
Overall, the core idea in this work is interesting but underexplored.
* As of when I read this paper, all results on this work used 200 hidden units realizing results that were well off from the state of the art results on Penn Tree Bank (as pointed out by the external reader).
The authors responded by stating that this was done to achieve a relative comparison.  A more interesting comparison, in addition to the ones presented, would be to see how well each method performs while not controlling for hidden layer size. Then, it might be that restricting the number of hidden dimensions is required for the RNN without any regularization but for both Variational Dropout and Noisin, one obtains better results with a larger the hidden dimension.
* The current experimental setup makes it difficult to assess when the proposed regularization is useful. Table 2 suggests the answer is sometimes and Table 3 suggests its marginally useful when the RNN size is restricted.
* How does the proposed method's peformance compare to Zoneout https://arxiv.org/pdf/1606.01305.pdf?
* Clarifying the role of variational inference: I could be missing something but I don't see a good reason why the prior (even if learned) should be close to the true posterior under the model. I fear the bound in Section (3) [please include equation numbers in the paper] could be quite loose.
* What is the rationale for not comparing to the model proposed in [Chung et. al] where there is a stochastic and deterministic component to the transition function? In what situations do we expect the fully stochastic transition here to work better than a model that has both? Presumably, some aspect of the latent variable + RNN model could be expressed by having a small variance for a subset of the dimensions and large one for the others
but since gamma is the same across all dimensions of the model, I'm not sure this feature can be incorporated into the current approach. Such a comparison would also empirically verify what happens when learning with the prior versus doing inference with an approximate posterior helps.
* The regularization is motivated from the point of view of sampling the hidden states to be from the exponential family, but all the experiments provided seem to use a Gaussian distribution. This paper would be strengthened by a discussion and experimentation with other kinds of distributions in the exponential family.


EMPIRICAL RISK LANDSCAPE ANALYSIS FOR UNDER- STANDING DEEP NEURAL NETWORKS
This work aims to provide comprehensive landscape analysis of empirical risk in deep neural networks (DNNs), including the convergence behavior of its gradient, its stationary points and the empirical risk itself to their corresponding population counterparts, which reveals how various network parameters determine the convergence performance. In particular, for an l-layer linear neural network consisting of d i neurons in the i-th layer, we prove the gradient of its empirical risk uniformly converges to the one of its population risk, at the rate of
Here d is the total weight dimension, s is the number of nonzero entries of all the weights and the magnitude of weights per layer is upper bounded by r. Moreover, we prove the one-to-one correspondence of the non-degenerate stationary points between the empirical and population risks and provide convergence guarantee for each pair. We also establish the uniform convergence of the empirical risk to its population counterpart and further derive the stability and generalization bounds for the empirical risk. In addition, we analyze these properties for deep nonlinear neural networks with sigmoid activation functions. We prove similar results for convergence behavior of their empirical risk gradients, non-degenerate stationary points as well as the empirical risk itself. To our best knowledge, this work is the first one theoretically characterizing the uniform convergence of the gradient and stationary points of the empirical risk of DNN models, which benefits the theoretical understanding on how the neural network depth l, the layer width d i , the network size d, the sparsity in weight and the parameter magnitude r determine the neural network landscape.

Overall, this work seems like a reasonable attempt to answer the question of how the empirical loss landscape relates to the true population loss landscape.  The analysis answers:
1) When empirical gradients are close to true gradients
2) When empirical isolated saddle points are close to true isolated saddle points
3) When the empirical risk is close to the true risk.
The answers are all of the form that if the number of training examples exceeds a quantity that grows with the number of layers, width and the exponential of the norm of the weights with respect to depth, then empirical quantities will be close to true quantities.  I have not verified the proofs in this paper (given short notice to review) but the scaling laws in the upper bounds found seem reasonably correct.
Another reviewer's worry about why depth plays a role in the convergence of empirical to true values in deep linear networks is a reasonable worry, but I suspect that depth will necessarily play a role even in deep linear nets because the backpropagation of gradients in linear nets can still lead to exponential propagation of errors between empirical and true quantities due to finite training data.  Moreover the loss surface of deep linear networks depends on depth even though the expressive capacity does not.   An analysis of dynamics on this loss surface was presented in Saxe et. al. ICLR 2014 which could be cited to address that reviewer's concern.  However, the reviewer's suggestion that the results be compared to what is known more exactly for simple linear regression is a nice one.
Overall, I believe this paper is a nice contribution to the deep learning theory literature. However,  it would even better to help the reader with more intuitive statements about the implications of their results for practice, and the gap between their upper bounds and practice, especially given the intense interest in the generalization error problem.   Because their upper bounds look similar to those based on Rademacher complexity or VC dimension (although they claim theirs are a little tighter) - they should put numbers in to their upper bounds taken from trained neural networks, and see what the numerical evaluation of their upper bounds turn out to be in situations of practical interest where deep networks show good generalization performance despite having significantly less training data than number of parameters.  I suspect their upper bounds will be loose, but still  - it would be an excellent contribution to the literature to quantitatively compare theory and practice with bounds that are claimed to be slightly tigher than previous bounds.  Even if they are loose - identifying the degree of looseness could inspire interesting future work.


We show how an ensemble of Q * -functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the Q-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark.

This paper paper uses an ensemble of networks to represent the uncertainty in deep reinforcement learning.
The algorithm then chooses optimistically over the distribution induced by the ensemble.
This leads to improved learning / exploration, notably better than the similar approach bootstrapped DQN.
There are several things to like about this paper:
- It is a clear paper, with a simple message and experiments that back up the claims.
- The proposed algorithm is simple and could be practical in a lot of settings and even non-DQN variants.
- It is interesting that Bootstrapped DQN gets such poor performance, this suggests that it is very important in the original paper https://arxiv.org/abs/1602.04621 that "ensemble voting" is applied to the test evaluation... (why do you think this is by the way, do you think it has something to do with the data being *more* off-policy / diverse under a TS vs UCB scheme?)
On the other hand:
- The novelty/scope of this work is somewhat limited... this is more likely (valuable) incremental work than a game-changer.
- Something feels wrong/hacky/incomplete about just doing "ensemble" for uncertainty without bootstrapping/randomization... if we had access to more powerful optimization techniques then this certainly wouldn't be sensible - I think that you should mention that you are heavily reliant on "random initialization + SGD/Adam + specific network architecture" to maintain this idea of uncertainty. For example, this wouldn't work for linear value functions!
- I think the original bootstrapped DQN used "ensemble voting" at test time, so maybe you should change the labels or the way this is introduced/discussed. It's definitely very interesting that *essentially* the learning benefit is coming from ensembling (rather than "raw" bootstrapped DQN) and UCB still looks like it does better.
- I'm not convinced that page 4 and the "Bayesian" derivation really add too much value to this paper... alternatively, maybe you could introduce the actual algorithm first (train K models in parallel) and then say "this is similar to particle filter" and add the mathematical derivation after, rather than as if it was some complex formula derived. If you want to reference some justification/theory for ensemble-based uncertainty approximation you might consider https://arxiv.org/pdf/1705.07347.pdf instead.
- I think this paper might miss the point of the "bigger" problem of efficient exploration in RL... or even how to get "deep" exploration with deep RL. Yes this algorithm sees improvements across Atari, but it's not clear why/if this is a step change versus simply increasing the amount of replay or tuning the learning rate.  (Actually I do believe this algorithm can demonstrate deep exploration... but it looks like we're not seeing the big improvements on the "sub-human" games you might hope.)
Overall I do think this is a pretty good short paper/evaluation of UCB-ensembles on Atari.
The scope/insight of the paper isn't groundbreaking, but I think it delivers a clear short message on the Atari benchmark.
Perhaps this will encourage people to dig deeper into some of these issues... I vote accept.


Published as a conference paper at ICLR 2018 PROXIMAL BACKPROPAGATION
We propose proximal backpropagation (ProxProp) as a novel algorithm that takes implicit instead of explicit gradient steps to update the network parameters during neural network training. Our algorithm is motivated by the step size limitation of explicit gradient descent, which poses an impediment for optimization. ProxProp is developed from a general point of view on the backpropagation algorithm, currently the most common technique to train neural networks via stochastic gradient descent and variants thereof. Specifically, we show that backpropagation of a prediction error is equivalent to sequential gradient descent steps on a quadratic penalty energy, which comprises the network activations as variables of the optimization. We further analyze theoretical properties of ProxProp and in particular prove that the algorithm yields a descent direction in parameter space and can therefore be combined with a wide variety of convergent algorithms. Finally, we devise an efficient numerical implementation that integrates well with popular deep learning frameworks. We conclude by demonstrating promising numerical results and show that ProxProp can be effectively combined with common first order optimizers such as Adam.

The paper uses a lesser-known interpretation of the gradient step of a composite function (i.e., via reverse mode automatic differentiation or backpropagation), and then replaces one of the steps with a proximal step. The proximal step requries the solution of a positive-definite linear system, so it is approximated using a few iterations of CG. The paper provides theory to show that their proximal variant (even with the CG approximations) can lead to convergent algorithms (and since practical algorithms are not necessarily globally convergent, most of the theory shows that the proximal variant has similar guarantees to a standard gradient step).
On reading the abstract and knowing quite a bit about proximal methods, I was initially skeptical, but I think the authors have done a good job of making their case. It is a well-written, very clear paper, and it has a good understanding of the literature, and does not overstate the results. The experiments are serious, and done using standard state-of-the-art tools and architectures. Overall, it is an interesting idea, and due to the current focus on neural nets, it is of interest even though it is not yet providing substantial improvements.
The main drawback of this paper is that there is no theory to suggest the ProxProp algorithm has better worst-case convergence guarantees, and that the experiments do not show a consistent benefit (in terms of time) of the method. On the one hand, I somewhat agree with the authors that "while the running time is higher... we expect that it can be improved through further engineering efforts", but on the other hand, the idea of nested algorithms ("matrix-free" or "truncated Newton") always has this issue. A very similar type of ideas comes up in constrained or proximal quasi-Newton methods, and I have seen many papers (or paper submissions) on this style of method (e.g., see the 2017 SIAM Review paper on FWI by Metivier et al. at https://doi.org/10.1137/16M1093239). In every case, the answer seems to be that it can work on *some problems* and for a few well chosen parameters, so I don't yet buy that ProxProp is going to make a huge savings on a wide-range of problems.
In brief: quality is high, clarity is high, originality is high, and significance is medium.
Pros: interesting idea, relevant theory provided, high-quality experiments
Cons: no evidence that this is a "break-through" idea
Minor comments:
- Theorems seemed reasonable and I have no reason to doubt their accuracy
- No typos at all, which I find very unusual. Nice job!
- In Algo 1, it would help to be more explicit about the updates (a), (b), (c), e.g., for (a), give a reference to eq (8), and for (b), reference equations (9,10).  It's nice to have it very clear, since "gradient step" doesn't make it clear what the stepsize is, and if this is done in a "Jacob-like" or "Gauss-Seidel-like" fashion. (c) has no reference equation, does it?
- Similarly, for Algo 2, add references. In particular, tie in the stepsizes tau and tau_theta here.
- Motivation in section 4.1 was a bit iffy. A larger stepsize is not always better, and smaller is not worse. Minimizing a quadratic f(x) = .5||x||^2 will converge in one step with a step-size of 1 because this is well-conditioned; on the flip side, slow convergence comes from lack of strong convexity, or with strong convexity, ill-conditioning of the Hessian (like a stiff ODE).
- The form of equation (6) was very nice, and you could also point out the connection with backward Euler for finite-difference methods. This was the initial setting of analysis for most of original results that rely on the proximal operator (e.g., Lions and Mercier 1970s).
- Eq (9), this is done component-wise, i.e., Hadamard product, right?
- About eq (12), even if softmax cross-entropy doesn't have a closed-form prox (and check the tables of Combettes and Pesquet), because it is separable (if I understand correctly) then it ought to be amenable to solving with a handful of Newton iterations which would be quite cheap.
Prox tables (see also the new edition of Bauschke and Combettes' book): P. L. Combettes and J.-C. Pesquet, "Proximal splitting methods in signal processing," in: Fixed-Point Algorithms for Inverse Problems in Science and Engineering (2011) http://www4.ncsu.edu/~pcombet/prox.pdf
- Below prop 4, discussing why not to make step (b) proximal, this was a bit vague to me. It would be nice to expand this.
- Page 6 near the top, to apply the operator, in the fully-connected case, this is just a matrix multiply, right? and in a conv net, just a convolution? It would help the reader to be more explicit here.
- Section 5.1, 2nd paragraph, did you swap tau_theta and tau, or am I just confused? The wording here was confusing.
- Fig 2 was not that convincing since the figure with time showed that either usual BackProp or the exact ProxProp were best, so why care about the approximate ProxProp with a few CG iterations? The argument of better generalization is based on very limited experiments and without any explanation, so I find that a weak argument (and it just seems weird that inexact CG gives better generalization).  The right figure would be nice to see with time on the x-axis as well.
- Section 5.2, this was nice and contributed to my favorable opinion about the work. However, any kind of standard convergence theory for usual SGD requires the stepsize to change per iteration and decrease toward zero. I've heard of heuristics saying that a fixed stepsize is best and then you just make sure to stop the algorithm a bit early before it diverges or behaves wildly -- is that true here?
- Final section of 5.3, about the validation accuracy, and the accuracy on the test set after 50 epochs. I am confused why these are different numbers. Is it just because 50 epochs wasn't enough to reach convergence, while 300 seconds was? And why limit to 50 epochs then? Basically, what's the difference between the bottom two plots in Fig 3 (other than scaling the x-axis by time/epoch), and why does ProxProp achieve better accuracy only in the right figure?


Many deep reinforcement learning approaches use graphical state representations, this means visually distinct games that share the same underlying structure cannot effectively share knowledge. This paper outlines a new approach for learning underlying game state embeddings irrespective of the visual rendering of the game state. We utilise approaches from multi-task learning and domain adaption in order to place visually distinct game states on a shared embedding manifold. We present our results in the context of deep reinforcement learning agents.

In this paper, the authors propose a new approach for learning underlying structure of visually distinct games.
The proposed approach combines convolutional layers for processing input images, Asynchronous Advantage Actor Critic for deep reinforcement learning task and adversarial approach to force the embedding representation to be independent of the visual representation of games.
The network architecture is suitably described and seems reasonable to learn simultaneously similar games, which are visually distinct. However, the authors do not explain how this architecture can be used to do the domain adaptation.
Indeed, if some games have been learnt by the proposed algorithm, the authors do not precise what modules have to be retrained to learn a new game. This is a critical issue, because the experiments show that there is no gain in terms of performance to learn a shared embedding manifold (see DA-DRL versus baseline in figure 5).
If there is a gain to learn a shared embedding manifold, which is plausible, this gain should be evaluated between a baseline, that learns separately the games, and an algorithm, that learns incrementally the games.
Moreover, in the experimental setting, the games are not similar but simply the same.
My opinion is that this paper is not ready for publication. The interesting issues are referred to future works.


A SCALABLE LAPLACE APPROXIMATION FOR NEURAL NETWORKS
We leverage recent insights from second-order optimisation for neural networks to construct a Kronecker factored Laplace approximation to the posterior over the weights of a trained network. Our approximation requires no modification of the training procedure, enabling practitioners to estimate the uncertainty of their models currently used in production without having to retrain them. We extensively compare our method to using Dropout and a diagonal Laplace approximation for estimating the uncertainty of a network. We demonstrate that our Kronecker factored method leads to better uncertainty estimates on out-of-distribution data and is more robust to simple adversarial attacks. Our approach only requires calculating two square curvature factor matrices for each layer. Their size is equal to the respective square of the input and output size of the layer, making the method efficient both computationally and in terms of memory usage. We illustrate its scalability by applying it to a state-of-the-art convolutional network architecture.

This paper uses recent progress in the understanding and approximation of curvature matrices in neural networks to revisit a venerable area- that of Laplace approximations to neural network posteriors. The Laplace method requires two stages - 1) obtaining a point estimate of the parameters followed by 2) estimation of the curvature. Since 1) is close to common practice it raises the appealing possibility of adding 2) after the fact, although the prior may be difficult to interpret in this case. A pitfall is that the method needs the point estimate to fall in a locally quadratic bowl or to add regularisation to make this true. The necessary amount of regularisation can be large as reported in section 5.4.
The paper is generally well written. In particular the mathematical exposition attains good clarity. Much of the mathematical treatment of the curvature was already discussed by Martens and Grosse and Botev et al in previous works. The paper is generally well referenced.
Given the complexity of the method, I think it would have helped to submit the code in anonymized form at this point.There are also some experiments not there that would improve the contribution. Figure 1 should include a comparison to Hamiltonian Monte Carlo and the full Laplace approximation (It is not sufficient to point to experiments in Hernandez-Lobato and Adams 2015 with a different model/prior). The size of model and data would not be prohibitive for either of these methods in this instance. All that figure 1 shows at the moment is that the proposed approximation has smaller predictive variance than the fully diagonal variant of the method.
It would be interesting (but perhaps not essential) to compare the Laplace approximation to other scalable methods from the literature such as that of Louizos and Welling 2016 which uses also used matrix normal distributions. It is good that the paper includes a modern architecture with a more challenging dataset. It is a shame the method does not work better in this instance but the authors should not be penalized for reporting this. I think a paper on a probabilistic method should at some point evaluate log likelihood in a case where the test distribution is the same as the training distribution. This complements experiments where there is dataset shift and we wish to show robustness. I would be very interested to know how useful the implied marginal likelihoods of the approximation where, as suggested for further work in the conclusion.


NEURAL-GUIDED DEDUCTIVE SEARCH FOR REAL- TIME PROGRAM SYNTHESIS FROM EXAMPLES
Synthesizing user-intended programs from a small number of input-output examples is a challenging problem with several important applications like spreadsheet manipulation, data wrangling and code refactoring. Existing synthesis systems either completely rely on deductive logic techniques that are extensively handengineered or on purely statistical models that need massive amounts of data, and in general fail to provide real-time synthesis on challenging benchmarks. In this work, we propose Neural Guided Deductive Search (NGDS), a hybrid synthesis technique that combines the best of both symbolic logic techniques and statistical models. Thus, it produces programs that satisfy the provided specifications by construction and generalize well on unseen examples, similar to data-driven systems. Our technique effectively utilizes the deductive search framework to reduce the learning problem of the neural component to a simple supervised learning setup. Further, this allows us to both train on sparingly available real-world data and still leverage powerful recurrent neural network encoders. We demonstrate the effectiveness of our method by evaluating on real-world customer scenarios by synthesizing accurate programs with up to 12× speed-up compared to state-of-the-art systems.

The paper presents a branch-and-bound approach to learn good programs
(consistent with data, expected to generalise well), where an LSTM is
used to predict which branches in the search tree should lead to good
programs (at the leaves of the search tree). The LSTM learns from
inputs of program spec + candidate branch (given by a grammar
production rule) and ouputs of quality scores for programms. The issue
of how greedy to be in this search is addressed.
In the authors' set up we simply assume we are given a 'ranking
function' h as an input (which we treat as black-box). In practice
this will simply be a guess (perhaps a good educated one) on which
programs will perform correctly on future data. As the authors
indicate, a more ambitious paper would consider learning h, rather
than assuming it as a given.
The paper has a number of positive features. It is clearly written
(without typo or grammatical problems). The empirical evaluation
against PROSE is properly done and shows the presented method working
as hoped. This was a competent approach to an interesting (real)
problem. However, the 'deep learning' aspect of the paper is not
prominent: an LSTM is used as a plug-in and that is about it. Also,
although the search method chosen was reasonable, the only real
innovation here is to use the LSTM to learn a search heuristic.
The authors do not explain what "without attention" means.
I think the authors should mention the existence of (logic) program
synthesis using inductive logic programming. There are also (closely
related) methods developed by the LOPSTR (logic-based program
synthesis and transformation) community. Many of the ideas here are
reminiscent of methods existing in those communities (e.g. top-down search
with heuristics). The use of a grammar to define the space of programs
is similar to the "DLAB" formalism developed by researchers at KU
Leuven.
ADDED AFTER REVISIONS/DISCUSSIONS
The revised paper has a number of improvements which had led me to give it slightly higher rating.


HIERARCHICAL DENSITY ORDER EMBEDDINGS
By representing words with probability densities rather than point vectors, probabilistic word embeddings can capture rich and interpretable semantic information and uncertainty (Vilnis & McCallum, 2014;Athiwaratkun & Wilson, 2017). The uncertainty information can be particularly meaningful in capturing entailment relationships -whereby general words such as "entity" correspond to broad distributions that encompass more specific words such as "animal" or "instrument". We introduce density order embeddings, which learn hierarchical representations through encapsulation of probability distributions. In particular, we propose simple yet effective loss functions and distance metrics, as well as graph-based schemes to select negative samples to better learn hierarchical probabilistic representations. Our approach provides state-of-the-art performance on the WORDNET hypernym relationship prediction task and the challenging HYPERLEX lexical entailment dataset -while retaining a rich and interpretable probabilistic representation.

The paper introduces a novel method for modeling hierarchical data. The work builds on previous approaches, such as Vilnis and McCallum's Word2Gauss and Vendrov's Order Embeddings, to establish a partial order over probability densities via encapsulation, which allows it to model hierarchical information. The aim is to learn embeddings from supervised structured data, such as WordNet. The work also investigates various schemes for selecting negative samples. The evaluation consists of hypernym detection on WordNet and graded lexical entailment, in the shape of HyperLex. This is good work: it is well written, the experiments are thorough and the proposed method is original and works well.
Section 3 could use some more signposting. Especially for 3.3 it would be good to explain (either at the beginning of section 3, or the beginning of section 3.3) why these measures matter and what is going to be done with them.
It's good that LEAR is mentioned and compared against, even though it was very recently published. Please do note that the authors' names are misspelled: Vuli\'c not Vulic, Mrk\v{s}i\'c instead of Mrksic.
If I am not mistaken, the Vendrov WordNet test set is a set of positive pairs. I would like to see more details on how the evaluation is done here: presumably, the lower I set the threshold, the higher my score? Or am I missing something?
It would be useful to describe exactly the extent to which supervision is used - the method only needs positive and negative links, and does not require any additional order information (i.e., WordNet strictly contains more information than what is being used).
I don't see what Socher et al. (2013) has to do with the loss in equation (7). Or did they invent the margin loss?
Word2gauss also evaluates on similarity and relatedness datasets. Did you consider doing that here too?
"hypothesis proposed by Santus et al. which says" is not a valid reference.


Workshop track -ICLR 2018 BUILDING GENERALIZABLE AGENTS WITH A REALIS- TIC AND RICH 3D ENVIRONMENT
Teaching an agent to navigate in an unseen 3D environment is a challenging task, even in the event of simulated environments. To generalize to unseen environments, an agent needs to be robust to low-level variations (e.g. color, texture, object changes), and also high-level variations (e.g. layout changes of the environment). To improve overall generalization, all types of variations in the environment have to be taken under consideration via different level of data augmentation steps. To this end, we propose House3D, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset . The diversity in House3D opens the door towards scene-level augmentation, while the label-rich nature of House3D enables us to inject pixel-& task-level augmentations such as domain randomization (Tobin et al., 2017) and multi-task training. Using a subset of houses in House3D, we show that reinforcement learning agents trained with an enhancement of different levels of augmentations perform much better in unseen environments than our baselines with raw RGB input by over 8% in terms of navigation success rate. House3D is publicly available at

Building rich 3D environments where to run simulations is a very interesting area of research.
Strengths:
1.	The authors propose a virtual environment of indoor scenes having a much larger scale compared to similar interactive environments and access to multiple visual modalities. They also show how the number of available scenes greatly impacts generalization in navigation based tasks.
2.	The authors provide a thorough analysis on the contribution of different feature types (Mask, Depth, RGB) towards the success rate of the goal task. The improvements and generalization brought by the segmentation and depth masks give interesting insights towards building new navigation paradigms for real-world robotics.
Weaknesses:
1.	The authors claim that the proposed environment allows for multiple applications and interactions, however from the description in section 3, the capacities of the simulator beyond navigation are unclear.
The dataset proposed, Home3D, adds a number of functionalities over the SUNCG dataset. The SUNCG dataset provides a large number of 3D scanned houses.  The most important contributions with respect to SUNCG are:
- An efficient renderer: an important aspect.
- Introducing physics: this is very interesting, unfortunately the contribution here is very small. Although I am sure the authors are planing to move beyond the current state of their implementation, the only physical constraint currently implemented is an occupancy rule and collision detection. This is not technically challenging.
Therefore, the added novelty with respect to SUNCG is very limited.
2.	The paper presents the proposed task as navigation from high level task description, but given that the instructions are fixed for a given target, there are only 5 possible instructions which are encoded as one-hot vectors. Given this setting, it is unclear the need for a gated attention mechanism. While this limited setting allows for a clear generalization analysis, it would have been good to study a setting with more complex instructions, allowing to evaluate instructions not seen during training.
3.	While the authors make a good point showing generalization towards unseen scenes, it would have been good to also show generalization towards real scenarios, demonstrating the realistic nature of House3D and the advantages of using non-RGB features.
4.	It would have been good to report an analysis on the number of steps performed by the agent before reaching its goal on the success cases. It seems to me that the continuous policy would be justified in this setting.
Comments
-	It is unclear to me how the reward shaping addition helps generalize to unseen houses at test time, as suggested by the authors.
-	I miss a reference to (https://arxiv.org/pdf/1609.05143.pdf) beyond the AI-THOR environment, given that they also approach target driven navigation using an actor-critic approach.
The paper proposes a new realistic indoor virtual environment, having a much larger number of scenes than similar environments. From the experiments shown, it seems that the scale increase, together with the availability of features such as Segmentation and Depth improve generalization in navigation tasks, which makes it a promising framework for future work on this direction. However, the task proposed seems too simple considering the power of this environment, and the models used to solve the task don’t seem to bring  relevant novelties from previous approaches. (https://arxiv.org/pdf/1706.07230.pdf)


EVALUATING THE ROBUSTNESS OF NEURAL NET- WORKS: AN EXTREME VALUE THEORY APPROACH
The robustness of neural networks to adversarial examples has received great attention due to security implications. Despite various attack approaches to crafting visually imperceptible adversarial examples, little has been developed towards a comprehensive measure of robustness. In this paper, we provide a theoretical justification for converting robustness analysis into a local Lipschitz constant estimation problem, and propose to use the Extreme Value Theory for efficient evaluation. Our analysis yields a novel robustness metric called CLEVER, which is short for Cross Lipschitz Extreme Value for nEtwork Robustness. The proposed CLEVER score is attack-agnostic and computationally feasible for large neural networks. Experimental results on various networks, including ResNet, Inceptionv3 and MobileNet, show that (i) CLEVER is aligned with the robustness indication measured by the 2 and ∞ norms of adversarial examples from powerful attacks, and (ii) defended networks using defensive distillation or bounded ReLU indeed achieve better CLEVER scores. To the best of our knowledge, CLEVER is the first attack-independent robustness metric that can be applied to any neural network classifier.

Summary
========
The authors present CLEVER, an algorithm which consists in evaluating the (local) Lipschitz constant of a trained network around a data point. This is used to compute a lower-bound on the minimal perturbation of the data point needed to fool the network.
The method proposed in the paper already exists for classical function, they only transpose it to neural networks. Moreover, the lower bound comes from basic results in the analysis of Lipschitz continuous functions.
Clarity
=====
The paper is clear and well-written.
Originality
=========
This idea is not new: if we search for "Lipschitz constant estimation" in google scholar, we get for example
Wood, G. R., and B. P. Zhang. "Estimation of the Lipschitz constant of a function." (1996)
which presents a similar algorithm (i.e., estimation of the maximum slope with reverse Weibull).
Technical quality
==============
The main theoretical result in the paper is the analysis of the lower-bound on \delta, the smallest perturbation to apply on
a data point to fool the network. This result is obtained almost directly by writing the bound on Lipschitz-continuous function
 | f(y)-f(x) | < L || y-x ||
where x = x_0 and y = x_0 + \delta.
Comments:
- Lemma 3.1: why citing Paulavicius and Zilinskas for the definition of Lipschitz continuity? Moreover, a Lipschitz-continuous function does not need to be differentiable at all (e.g. |x| is Lipschitz with constant 1 but sharp at x=0). Indeed, this constant can be easier obtained if the gradient exists, but this is not a requirement.
- (Flaw?) Theorem 3.2 : This theorem works for fixed target-class since g = f_c - f_j for fixed g. However, once g = min_j f_c - f_j, this theorem is not clear with the constant Lq. Indeed, the function g should be
g(x) = min_{k \neq c} f_c(x) - f_k(x).
Thus its Lipschitz constant is different, potentially equal to
L_q = max_{k} \| L_q^k \|,
where L_q^k is the Lipschitz constant of f_c-f_k. If the theorem remains unchanged after this modification, you should clarify the proof. Otherwise, the theorem will work with the maximum over all Lipschitz constants but the theoretical result will be weakened.
- Theorem 4.1: I do not see the purpose of this result in this paper. This should be better motivated.
Numerical experiments
====================
Globally, the numerical experiments are in favor of the presented method. The authors should also add information about the time it takes to compute the bound, the evolution of the bound in function of the number of samples and the distribution of the relative gap between the lower-bound and the best adversarial example.
Moreover, the numerical experiments look to be realized in the context of targeted attack. To show the real effectiveness of the approach, the authors should also show the effectiveness of the lower-bound in the context of non-targeted attack.
#######################################################
Post-rebuttal review
---------------------------
Given the details the authors provided to my review, I decided to adjust my score. The method is simple and shows to be extremely effective/accurate in practice.
Detailed answers:
1) Indeed, I was not aware that the paper only focuses on one dimensional functions. However, they still work with less assumption, i.e., with no differential functions. I was pointing out the similarities between their approach and your: the two algorithms (CLEVER and Slope) are basically the same, and using a limit you can go from "slope" to "gradient norm".
In any case, I have read the revision and the additional numerical experiment to compare Clever with their method is a good point.
2) " Overall, our analysis is simple and more intuitive, and we further facilitate numerical calculation of the bound by applying the extreme value theory in this work. "
This is right. I am just surprised is has not been done before, since it requires only few lines of derivation. I searched a bit but it is not possible to find any kind of similar results. Moreover, this leads to good performances, so there is no needs to have something more complex.
3) "The usual Lipschitz continuity is defined in terms of L2 norm and the extension to an arbitrary Lp norm is not straightforward"
Indeed, people usually use the Lipschitz continuity using the L2norm, but the original definition is wider.
Quickly, if you have a differential, scalar function from a space E -> R, then the gradient is a function from space E to E*, the dual of the space E.
Let || . || the norm of space E. Then, || . ||* is the dual norm of ||.||, and also the norm of E*.
In that case, Lipschitz continuity writes
f(x)-f(y) <= L || x-y ||, with L >= max_{x in E*} || f'(x) ||*
In the case where || . || is an \ell-p norm, then || . ||* is an \ell-q norm; with 1/p+1/q = 1.
If you are interested, there is a clear and concise explanation in the introduction of this paper: Accelerating the cubic regularization of Newton’s method on convex problems, by Yurii Nesterov.
I have no additional remarks for 4) -> 9), since everything is fixed in the new version of the paper.


Reinforcement learning in environments with large state-action spaces is challenging, as exploration can be highly inefficient. Even if the dynamics are simple, the optimal policy can be combinatorially hard to discover. In this work, we propose a hierarchical approach to structured exploration to improve the sample efficiency of on-policy exploration in large state-action spaces. The key idea is to model a stochastic policy as a hierarchical latent variable model, which can learn low-dimensional structure in the state-action space, and to define exploration by sampling from the low-dimensional latent space. This approach enables lower sample complexity, while preserving policy expressivity. In order to make learning tractable, we derive a joint learning and exploration strategy by combining hierarchical variational inference with actor-critic learning. The benefits of our learning approach are that 1) it is principled, 2) simple to implement, 3) easily scalable to settings with many actions and 4) easily composable with existing deep learning approaches. We demonstrate the effectiveness of our approach on learning a deep centralized multi-agent policy, as multi-agent environments naturally have an exponentially large state-action space. In this setting, the latent hierarchy implements a form of multi-agent coordination during exploration and execution (MACE). We demonstrate empirically that MACE can more efficiently learn optimal policies in challenging multi-agent games with a large number (∼ 20) of agents, compared to conventional baselines. Moreover, we show that our hierarchical structure leads to meaningful agent coordination.

This paper proposes an approach to improve exploration in multiagent reinforcement learning by allowing the policies of the individual agents to be conditioned on an external coordination signal \lambda. In order to find such parametrized policies, the approach combines deep RL with a variational inference approach (ELBO optimization). The paper presents an empirical evaluation, which seems encouraging, but that is also somewhat difficult to interpret given the lack of comparison to other state-of-the-art methods.
Overall, the paper seems interesting, but (in addition to the not completely convincing empirical evaluation), it has two main weaknesses: lack of clarity and grounding in related literature.
=Issues with clarity=
-"This problem has two equivalent solutions".
This is not so clear; depending on the movement of the preys it might well be that the optimal solution will switch to the other prey in certain cases?
-It is not clear what is really meant with the term "structured exploration". It just seems to mean 'improved'?
-It is not clear that the improvements are due to exploration; my feeling is that is is due to improved statistical strength on a more abstract state feature (which is learned), not unlike:
Geramifard, Alborz, et al. "Online discovery of feature dependencies." Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011.
However, there is no clear indication that there is an improved exploration policy.
-The problem setting is not quite clear:
The paper first introduces "multi-agent RL", which seems to correspond to a "stochastic game" (also "Markov game"), but then moves on to restrict to the "fully cooperative setting" (which would make it a "Multiagent MDP", Boutilier '96).
It subsequently says it deals only with deterministic problems (which would reduce the problem further to a learning version of a multiagent classical planning problem), but in the experiments do consider stochastically moving preys.
-The paper says the problem is fully observable, but fails to make explicit if this is *individually* fully observable, or jointly. I am assuming the former, but is it not clear how the agents observe this full state in the experimental evaluation.
This is actually a crucial confusion, as it completely changes the interpretation of what the approach does: in the individually observable case, the approach is adding a redundant source of information which is more abstract and thus seems to facilitate faster learning. In the latter case, where agents would have individual observations, it is actually providing the agents with more information.
As such, I would really encourage the authors to better define the task they are considering. E.g., by building on the taxonomies of problems that researchers have developed in the community focusing on decentralized POMDPs, such as:
Goldman, Claudia V., and Shlomo Zilberstein. "Decentralized control of cooperative systems: Categorization and complexity analysis." (2004).
-"Compared to the single-agent RL setting, multi-agent RL poses unique difficulties. A central issue
is the exploration-exploitation trade-off"
That now in particular happens to be a central issue in single agent RL too.
-"Finding the true posteriors P (λ t |s t ) ∝ P (s t |λ t )P (λ t ) is intractable in general"
The paper did not explain how this inference task is required to solve the RL problem.
-In general, I found the technical description impossible to follow, even after carefully looking at the appending. For instance, (also) there the term P (λ |s ) is suddenly introduced without explaining what the term exactly is? Why is the term P(a|λ) not popping up here? That also needs to be optimized, right? I suppose \phi is the parameter vector of the variational approximation, but it is never really stated. The various shorthand notations introduced for clarity do not help at all, but only make the formulas very cryptic.
-The main text is not readable since definitions, e.g., L(Q_r,\tehta,\phi), that are in the appendix are now missing.
-It is not clear to me how the second term of (10) is now estimated?
-"Shared (shared actor-critic): agents share a deterministic hidden layer,"
What kind of layer is this exactly? How does it relate to \lambda ?
-"The key difference is that this model does not sample from the shared hidden layer"
Why would sampling help? Given that we are dealing with a fully observable multiagent MDP, there is no inherent need to randomize at all? (there should be a optimal deterministic joint policy?)
-"There is shared information between the agents"
What information is referred to exactly?
Also: It is not quite clear if for these domains cloned would be better than completely independent learners (without shared weights)?
-I can't seem to find anywhere what is the actual shape (or type? I am assuming a vector of reals) of the used \lambda.
-in figure 5, rhs, what is being shown exactly? What do the colors mean? Why does there seem to be a \lambda *per* agent now?
=Related work=
I think the paper could/should be hugely improved in this respect.
The idea of casting MARL as inference has also been considered by:
Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments.
M Liu, C Amato, EP Anesta, JD Griffith, JP How - AAAI, 2016
Stick-breaking policy learning in Dec-POMDPs
M Liu, C Amato, X Liao, L Carin, JP How
International Joint Conference on Artificial Intelligence (IJCAI) 2015
Wu, F.; Zilberstein, S.; and Jennings, N. R. 2013. Monte-carlo
expectation maximization for decentralized POMDPs. In Proc.
of the 23rd Int’l Joint Conf. on Artificial Intelligence (IJCAI-
13).
I do not think that these explicitly make use of a mechanism to coordinate the policies, since they address to true Dec-POMDP setting where each agent only gets its own observations, but in the Dec-POMDP literature, there also is the notion of 'correlation device', which is an additional controller (say corresponding to a dummy agent), which of which the states can be observed by other agents and used to condition their actions on:
Bernstein DS, Hansen EA, Zilberstein S. Bounded policy iteration for decentralized POMDPs. InProceedings of the nineteenth international joint conference on artificial intelligence (IJCAI) 2005 Jun 6 (pp. 52-57).
(and clearly this could be directly included in the aforementioned learning approaches).
This notion of a correlation device also highlights to potential relation to methods to learn/compute correlated equilibria. E.g.,:
Greenwald A, Hall K, Serrano R. Correlated Q-learning. In ICML 2003 Aug 21 (Vol. 3, pp. 242-249).
A different connection between MARL and inference can be found in:
Zhang, Xinhua and Aberdeen, Douglas and Vishwanathan, S. V. N., "Conditional Random Fields for Multi-agent Reinforcement Learning", in (New York, NY, USA: ACM, 2007), pp. 1143--1150.
The idea of doing something hierarchical of course makes sense, but also here there are a number of related papers:
-putting "hierarchical multiagent" in google scholar finds works by Ghavamzadeh et al., Saira & Mahadevan, etc.
-Victor Lesser has pursued coordination for better exploration with a number of students.
I suppose that Guestrin et al.'s classical paper:
Guestrin, Carlos, Michail Lagoudakis, and Ronald Parr. "Coordinated reinforcement learning." ICML. Vol. 2. 2002.
would deserve a citation, and the MARL field is moving ahead fast, an explanation of the differences with COMA:
Counterfactual Multi-Agent Policy Gradients
J Foerster, G Farquhar, T Afouras, N Nardelli, S Whiteson
AAAI 2018
is probably also warranted.


Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers. We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small (100k-1.2M parameter) image classification networks. Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts. Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth.

This paper performs an analysis of shortcut connections in ResNet-like architectures. The authors hypothesize that the success of shortcut connections comes from the combination of linear and non-linear features at each layer and propose to substitute the identity shortcuts with a convolutional one (without non-linearity). This alternative is referred to as tandem block. Experiments are performed on a variety of image classification tasks such as CIFAR-10, CIFAR-100, SVHN and Fashion MNIST.
The paper is well structured and easy to follow. The main contribution of the paper is the comparison between identity skip connections and skip connections with one convolutional layer.
My main concerns are related to the contribution of the paper and experimental pipeline followed to perform the comparison. First, the idea of having convolutional shortcuts was already explored in the ResNet paper (see https://arxiv.org/pdf/1603.05027.pdf). Second, given Figures 3-4-5-6, it would seem that the authors are monitoring the performance on the test set during training. Moreover, results on Table 2 are reported as the ones with “the highest test accuracy achieved with each tandem block”. Could the authors give more details on how the hyperparameters of the architectures/optimization were chosen and provide more information on how the best results were achieved?
In section 3.5, the authors mention that batchnorm was not useful in their experiments, and was more sensitive to the learning rate value. Do the authors have any explanation/intuition for this behavior?
In section 4, authors claim that their results are competitive with the best published results for a similar number of parameters. It would be beneficial to add the mentioned best performing models in Table 2 to back this statement. Moreover, it seems that in some cases such as SVHN the differences between all the proposed blocks are too minor to draw any strong conclusions. Could those differences be due to, for example, luck in picking the initialization seed? How many times was each experiment run? If more than once, what was the std?
The experiments were performed on relatively shallow networks (8 to 26 layers). I wonder how the conclusions drawn scale to much deeper networks (of 100 layers for example) and on larger datasets such as ImageNet.
Figures 3-5 are not referenced nor discussed in the text.
Following the design of the tandem blocks proposed in the paper, I wonder why the tandem block B3x3(2,w) was not included.
Finally, it might be interesting to initialize the convolutions in the shortcut connections with the identity, and check what they have leant at the end of the training.
Some typos that the authors might want to fix:
- backpropegation -> backpropagation (Introduction, paragraph 3)
- dropout is a kind of regularization as well (Introduction, second to last paragraph)
- nad -> and (Sect 3.1. paragraph 1)


Bayesian posterior inference is prevalent in various machine learning problems. Variational inference provides one way to approximate the posterior distribution, however its expressive power is limited and so is the accuracy of resulting approximation. Recently, there has a trend of using neural networks to approximate the variational posterior distribution due to the flexibility of neural network architecture. One way to construct flexible variational distribution is to warp a simple density into a complex by normalizing flows, where the resulting density can be analytically evaluated. However, there is a trade-off between the flexibility of normalizing flow and computation cost for efficient transformation. In this paper, we propose a simple yet effective architecture of normalizing flows, ConvFlow, based on convolution over the dimensions of random input vector. Experiments on synthetic and real world posterior inference problems demonstrate the effectiveness and efficiency of the proposed method.

In this paper, the authors propose a type of Normalizing Flows (Rezende and Mohamed, 2015) for Variational Autoencoders (Kingma and Welling, 2014; Rezende et al., 2014) they call Convolutional Normalizing Flows.
More particularly, it aims at extending on the Planar Flow scheme proposed in Rezende and Mohamed (2015). The authors notice an improvement through their method over Normalizing Flows, IWAE with diagonal gaussian approximation, and standard Variational Autoencoders.
As noted by AnonReviewer3, several baselines are missing. But the authors partly address that issue in the comment section for the MNIST dataset.
The requirement of h being bijective seems wrong. For example, if h was a rectifier nonlinearity in the zero-derivative regime, the Jacobian determinant of the ConvFlow would be 1.
More importantly, the main issue is that this paper might need to highlight the fundamental difference between their proposed method and Inverse Autoregressive Flow (Kingma et al., 2016). The proposed connectivity pattern proposed for the convolution in order to make the Jacobian determinant computation is exactly the same as Inverse Autoregressive Flow and the authors seems to be aware of the order dependence of their architecture which is every similar to autoregressive models. This presentation of the paper can be misleading concerning the true innovation in the model trained. Proposing ConvFlow as a type of Inverse Autoregressive Flow would be more accurate and would allow to highlight better the innovation of the work.
Since this work does not offer additional significant insight over Inverse Autoregressive Flow, its value should be on demonstrating the efficiency of the proposed method. MNIST and Omniglot seems insufficient for that purpose given currently published work.
In the current state, I can't recommend the paper for acceptance.
Danilo Jimenez Rezende, Shakir Mohamed: Variational Inference with Normalizing Flows. ICML 2015
Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra: Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models. ICML 2014
Diederik P. Kingma, Max Welling: Auto-Encoding Variational Bayes. ICLR 2014
Diederik P. Kingma, Tim Salimans, Rafal Józefowicz, Xi Chen, Ilya Sutskever, Max Welling: Improving Variational Autoencoders with Inverse Autoregressive Flow. NIPS 2016


WASSERSTEIN AUTO-ENCODERS
We propose the Wasserstein Auto-Encoder (WAE)-a new algorithm for building a generative model of the data distribution. WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational Auto-Encoder (VAE) (Kingma & Welling, 2014). This regularizer encourages the encoded training distribution to match the prior. We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE) (Makhzani et al., 2016). Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality, as measured by the FID score.

This very well written paper covers the span between W-GAN and VAE. For a reviewer who is not an expert in the domain, it reads very well, and would have been of tutorial quality if space had allowed for more detailed explanations. The appendix are very useful, and tutorial paper material (especially A).
While I am not sure description would be enough to reproduce and no code is provided, every aspect of the architecture, if not described, if referred as similar to some previous work. There are also some notation shortcuts (not explained) in the proof of theorems that can lead to initial confusion, but they turn out to be non-ambiguous. One that could be improved is P(P_X, P_G) where one loses the fact that the second random variable is Y.
This work contains plenty of novel material, which is clearly compared to previous work:
- The main consequence of the use of Wasserstein distance is the surprisingly simple and useful Theorem 1. I could not verify its novelty, but this seems to be a great contribution.
- Blending GAN and auto-encoders has been tried in the past, but the authors claim better theoretical foundations that lead to solutions that do not rquire min-max
- The use of MMD in the context of GANs has also been tried. The authors claim that their use in the latent space makes it more practival
The experiments are very convincing, both numerically and visually.
Source of confusion: in algorithm 1 and 2, \tilde{z} is "sampled" from Q_TH(Z|xi), some one is lead to believe that this is the sampling process as in VAEs, while in reality Q_TH(Z|xi) is deterministic in the experiments.


Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.

Summary:
The manuscript presents experiments on distilling knowledge from a face classification model to student models for face alignment and verification. By selecting a good initialization strategy and guidelines for selecting appropriate targets for non-classification tasks, the authors achieve improved performance, compared to networks trained from scratch or with different initialization strategies.
Review:
The paper seems to be written in a rush.
I am not sure about the degree of novelty, as pretraining with domain-related data instead of general-purpose ImageNet data has been done before, Liu et al. (2014), for example pretrain a CNN on face classification to be used for emotion recognition. Admitted, knowledge transfer from classification to regression and retrieval tasks is not very common yet, except via pretraining on ImageNet, followed by fine-tuning on the target task.
My main concern is with the presentation of the paper. It is very hard to follow! Two reasons are that it has too many grammatical mistakes and that very often a “simple trick” or a “common trick” is mentioned instead of using a descriptive name for the method used.
Here are a few points that might help improving the work:
1) Many kind of empty phrases are repeated all over the paper, e.g. the reader is teased with mention of a “simple trick” or a “common trick”. I don’t think the phrase “breaking point”, that is repeated a couple of times, is correctly used (see https://www.merriam-webster.com/dictionary/breaking%20point for a defininition).
2) Section 4.1 does not explain the initialization but just describes motivation and notation.
3) Clarity of the approach: Using the case of alignment as an example, do you first pretrain both the teacher and student on classification, then finetune the teacher on alignment before the distillation step?
4) Table 1 mentions Fitnets, but cites Ba & Caruana (2014) instead of Romero et al. (2015)
5) The “experimental trick” you mention for setting alpha and beta, seems to be just validation, comparing different settings and picking the one yielding the highest improvements. On what partition of the data are you doing this hyperparameter selection?
6) The details of the architectures are missing, e.g. exactly what changes do you make to the architecture, when you change the task from classification to alignment or verification? What exactly is the “hidden layer” in that architecture?
7) Minor: Usually there is a space before parentheses (many citations don’t have one)
In its current form, I cannot recommend the manuscript for acceptance. I get the impression that the experimental work might be of decent quality, but the manuscript fails to convey important details of the method, of the experimental setup and in the interpretation of the results. The overall quality of the write-up has to be significantly improved.
References:
Liu, Mengyi, Ruiping Wang, Shaoxin Li, Shiguang Shan, Zhiwu Huang, and Xilin Chen. "Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild." In Proceedings of the 16th International Conference on Multimodal Interaction, pp. 494-501. ACM, 2014.


Published as a conference paper at ICLR 2018 PARAMETRIZED HIERARCHICAL PROCEDURES FOR NEURAL PROGRAMMING
Neural programs are highly accurate and structured policies that perform algorithmic tasks by controlling the behavior of a computation mechanism. Despite the potential to increase the interpretability and the compositionality of the behavior of artificial agents, it remains difficult to learn from demonstrations neural networks that represent computer programs. The main challenges that set algorithmic domains apart from other imitation learning domains are the need for high accuracy, the involvement of specific structures of data, and the extremely limited observability. To address these challenges, we propose to model programs as Parametrized Hierarchical Procedures (PHPs). A PHP is a sequence of conditional operations, using a program counter along with the observation to select between taking an elementary action, invoking another PHP as a sub-procedure, and returning to the caller. We develop an algorithm for training PHPs from a set of supervisor demonstrations, only some of which are annotated with the internal call structure, and apply it to efficient level-wise training of multi-level PHPs. We show in two benchmarks, NanoCraft and long-hand addition, that PHPs can learn neural programs more accurately from smaller amounts of both annotated and unannotated demonstrations.

I thank the authors for their updates and clarifications.  I stand by my original review and score.  I think their method and their evaluation has some major weaknesses, but I think that it still provides a good baseline to force work in this space towards tasks which can not be solved by simpler models like this.  So while I'm not super excited about the paper I think it is above the accept threshold.
--------------------------------------------------------------------------
This paper extends an existing thread of neural computation research focused on learning resuable subprocedures (or options in RL-speak).  Instead of simply input and output examples, as in most of the work in neural computation, they follow in the vein of the Neural Programmer-Interpreter (Reed and de Freitas, 2016) and Li et. al., 2017, where the supervision contains the full sequence of elementary actions in the domain for all samples, and some samples also contain the hierarchy of subprocedure calls.
The main focus of their work is learning from fewer fully annotated samples than prior work.  They introduce two new ideas in order to enable this:
1.  They limit the memory state of each level in the program heirarchy to simply a counter indicating the number of elementary actions/subprocedure calls taken so far (rather than a full RNN embedded hidden/cell state as in prior work).  They also limit the subprocedures such that they do not accept any arguments.
2.  By considering this very limited set of possible hidden states, they can compute the gradients using a dynamic program that seems to be more accurate than the approximate dynamic program used in Li et. al., 2017.
The main limitation of the work is this extremely limited memory state, and the lack of arguments.  Without arguments, everything that parameterizes the subprocedures must be in the visible world state.  In both of their domains, this is true, but this places a significant limitation on the algorithms which can be modeled with this technique.  Furthermore, the limited memory state means that the only way a subprocedure can remember anything about the current observation is to call a different subprocedure.  Again, their two evalation tasks fit into this paradigm, but this places very significant limitations on the set of applicable domains.  I would have like to see more discussion on how constraining these limitations would be in practice.  For example, it seems it would be impossible for this architecture to perform the Nanocraft task if the parameters of the task (width, height, etc.) were only provided in the first observation, rather than every observation.
None-the-less I think this work is an important step in our understanding of the learning dynamics for neural programs.  In particular, while the RNN hidden state memory used by the prior work enables the learning of more complicted programs *in theory*, this has not been shown in practice. So, it's possible that all the prior work is doing is learning to approixmate a much simpler architecture of this form.  Specifically, I think this work can act as a great base-line by forcing future work to focus on domains which cannot be easily solved by a simpler architecture of this form.  This limitation will also force the community to think about which tasks require a more complicated form of memory, and which can be solved with a very simple memory of this form.
I also have the following additional concerns about the paper:
1.  I found the current explanation of the algorithm to be very difficult to understand.  It's extremely difficult to understand the core method without reading the appendix, and even with the appendix I found the explanation of the level-by-level decomposition to be too terse.
2.  It's not clear how their gradient approximation compares to the technique used by Li et. al.  They obviously get better results on the addition and Nanocraft domains, but I would have liked a more clear explanation and/or some experiments providing insights into what enables these improvements (or at least an admission by the authors that they don't really understand what enabled the performance improvements).


Published as a conference paper at ICLR 2018 LEARNING APPROXIMATE INFERENCE NETWORKS FOR STRUCTURED PREDICTION
ABSTRACT Structured prediction energy networks (SPENs; Belanger & McCallum 2016) use neural network architectures to define energy functions that can capture arbitrary dependencies among parts of structured outputs. Prior work used gradient descent for inference, relaxing the structured output to a set of continuous variables and then optimizing the energy with respect to them. We replace this use of gradient descent with a neural network trained to approximate structured argmax inference. This "inference network" outputs continuous values that we treat as the output structure. We develop large-margin training criteria for joint training of the structured energy function and inference network. On multi-label classification we report speed-ups of 10-60x compared to (Belanger et al., 2017) while also improving accuracy. For sequence labeling with simple structured energies, our approach performs comparably to exact inference while being much faster at test time. We then demonstrate improved accuracy by augmenting the energy with a "label language model" that scores entire output label sequences, showing it can improve handling of long-distance dependencies in part-of-speech tagging. Finally, we show how inference networks can replace dynamic programming for test-time inference in conditional random fields, suggestive for their general use for fast inference in structured settings.

= Quality =
Overall, the authors do a good job of placing their work in the context of related research, and employ a variety of non-trivial technical details to get their methods to work well.
= Clarity =
Overall, the exposition regarding the method is good. I found the setup for the sequence tagging experiments confusing, tough. See more comments below.
= Originality / Significance =
The paper presents a clever idea that could help make SPENs more practical. The paper's results also suggest that we should be thinking more broadly about how to using complicated structured distributions as teachers for model compression.
= Major Comment =
I'm concerned by the quality of your results and the overall setup of your experiments. In particular, the principal contribution of the sequence tagging experiments seems top be different than what is advertised earlier on in the paper.
Most of your empirical success is obtained by taking a pretrained CRF energy function and using this as a teacher model to train a feed-forward inference network. You have have very few experiments using a SPEN energy function parametrization that doesn't correspond to a CRF, even though you could have used an arbitrary convnet, RNN, etc. The one exception is when you use the tag language model. This is a good idea, but it is pretrained, not trained using the saddle-point objective you introduce. In fact, you don't have any results demonstrating that the saddle-point approach is better than simpler alternatives.
It seems that you could have written a very different paper about model compression with CRFs that would have been very interesting and you could've have used many of the same experiments. It's unclear why SPENs are so important. The idea of amortizing inference is perhaps more general. My recommendation is that you either rebrand the paper to be more about general methods for amortizing structured prediction inference using model compression or do more fine-grained experiments with SPENs that demonstrate empirical gains that leverage their flexible deep-network-based energy functions.
= Minor Comments =
* You should mention 'Energy Based GANs"
* I don't understand "This approach performs backpropagation through each step of gradient descent, permitting more stable training but also evidently more overfitting." Why would it overfit more? Simply because training was more stable? Couldn't you prevent overfitting by regularizing more?
* You spend too much space talking about specific hyperparameter ranges, etc. This should be moved to the appendix. You should also add a short summary of the TLM architecture to the main paper body.
* Regarding your footnote discussing using a positive vs. negative sign on the entropy regularization term, I recommend checking out "Regularizing neural networks by penalizing confident output distributions."
* You should add citations for the statement "In these and related settings, gradient descent has started to be replaced by inference networks."
* I didn't find Table 1 particularly illuminating. All of the approaches seem to perform about the same. What conclusions should I make from it?
* Why not use KL divergence as your \Delta function?
* Why are the results in Table 5 on the dev data?
* I was confused by Table 4. First of all, it took me a very long time to figure out that the middle block of results corresponds to taking a pretrained CRF energy and amortizing inference by training an inference network. This idea of training with a standard loss (conditional log lik.) and then amortizing inference post-hoc was not explicitly introduced as an alternative to the saddle point objective you put forth earlier in the paper. Second, I was very surprised that the inference network outperformed Viterbi (89.7 vs. 89.1 for the same CRF energy). Why is this?
* I'm confused by the difference between Table 6 and Table 4? Why not just include the TLM results in Table 4?


Over the past decade a wide spectrum of machine learning models have been developed to model the neurodegenerative diseases, associating biomarkers, especially non-intrusive neuroimaging markers, with key clinical scores measuring the cognitive status of patients. Multi-task learning (MTL) has been extensively explored in these studies to address challenges associated to high dimensionality and small cohort size. However, most existing MTL approaches are based on linear models and suffer from two major limitations: 1) they cannot explicitly consider upper/lower bounds in these clinical scores; 2) they lack the capability to capture complicated non-linear effects among the variables. In this paper, we propose the Subspace Network, an efficient deep modeling approach for non-linear multi-task censored regression. Each layer of the subspace network performs a multi-task censored regression to improve upon the predictions from the last layer via sketching a low-dimensional subspace to perform knowledge transfer among learning tasks. Under mild assumptions, for each layer the parametric subspace can be recovered using only one pass of training data. Empirical results demonstrate that the proposed subspace network quickly picks up correct parameter subspaces, and outperforms state-of-the-arts in predicting neurodegenerative clinical scores using information in brain imaging.

This work proposes a multi task learning framework for the modeling of clinical data in neurodegenerative diseases.
Differently from previous applications of machine learning in neurodegeneration modeling, the proposed approach models the clinical data accounting for the bounded nature of cognitive tests scores. The framework is represented by a feed-forward deep architecture analogous to a residual network. At each layer a low-rank constraint is enforced on the linear transformation, while the cost function is specified in order to differentially account for the bounds of the predicted variables.
The idea of explicitly accounting for the boundedness of clinical scores is interesting, although the assumption of the proposed model is still incorrect: clinical scores are defined on discrete scales. For this reason the Gaussian assumption for the cost function used in the method is still not appropriate for the proposed application.
Furthermore, while being the main methodological drive of this work, the paper does not show evidence about improved predictive performance and generalisation when accounting for the boundedness of the regression targets.
The proposed algorithm is also generally compared with respect to linear methods, and the authors could have provided a more rigorous benchmark including standard non-linear prediction approaches (e.g. random forests, NN, GP, …).
Overall, the proposed methods seems to provide little added value to the large amount of predictive methods proposed so far for prediction in neurodegenerative disorders. Moreover, the proposed experimental paradigm appears flawed. What is the interest of predicting baseline (or 6 months at best) cognitive scores (relatively low-cost and part of any routine clinical assessment) from brain imaging data (high-cost and not routine)?
Other remarks.
- In section 2.2 and 4 there is some confusion between iteration indices and samples indices “i”.
- Contrarily to what is stated in the introduction, the loss functions proposed in page 3 (first two formulas) only accounts for the lower bound of the predicted variables.
-  Figure 2, synthetic data. The scale of the improvement of the subspace difference is quite tiny, in the order of 1e-2 when compared to U, and of 1e-5 across iterations. The loss function of Figure 2.b also does not show a strong improvement across iterations, while indicating a rather large instability of the optimisation procedure. These aspects may be a sign of convergence issues.
- The dimensionality of the subspace representation importantly depends on the choice of the rank R of U and V. This is a crucial parameters that is however not discussed nor analysed in the paper.
- The synthetic example of page 7 is quite misleading and potentially biased towards the proposed model. The authors are generating the synthetic data according to the model, and it is thus not surprising that they managed to obtain the best performance.  In particular, due to the nonlinear nature of (1), all the competing linear models are expected to perform poorly in this kind of setting.
- The computation time for the linear model shown in Table 3 is quite surprising (~20 minutes for linear regression of 5k observations). Is there anything that I am missing?