MMD GAN: Towards Deeper Understanding of Moment Matching Network
Generative moment matching network (GMMN) is a deep generative model that differs from Generative Adversarial Network (GAN) by replacing the discriminator in GAN with a two-sample test based on kernel maximum mean discrepancy (MMD). Although some theoretical guarantees of MMD have been studied, the empirical performance of GMMN is still not as competitive as that of GAN on challenging and large benchmark datasets. The computational efficiency of GMMN is also less desirable in comparison with GAN, partially due to its requirement for a rather large batch size during the training. In this paper, we propose to improve both the model expressiveness of GMMN and its computational efficiency by introducing adversarial kernel learning techniques, as the replacement of a fixed Gaussian kernel in the original GMMN. The new approach combines the key ideas in both GMMN and GAN, hence we name it MMD GAN. The new distance measure in MMD GAN is a meaningful loss that enjoys the advantage of weak ⇤ topology and can be optimized via gradient descent with relatively small batch sizes. In our evaluation on multiple benchmark datasets, including MNIST, CIFAR-10, CelebA and LSUN, the performance of MMD GAN significantly outperforms GMMN, and is competitive with other representative GAN works.

This paper presents the MMD-GAN, a generative architecture which combines the benefits of generative adversarial nets (GANs) and generative moment matching nets (GMMNs). The architecture is based on MMD, but with a kernel function learned by an adversary trying to maximize MMD. There's a bit of theory showing non-degeneracy, and experiments demonstrate that the generated samples are competitive with W-GANs.
Overall, I think this is a strong paper which presents a simple idea and executes it well. The proposed method is intuitively appealing. The modeling choices are well motivated, the writing is clear, the theoretical justification is nice, and relationships to other methods are clearly highlighted. The experiments compare against strong baselines on challenging datasets. Being able to make GMMNs competitive with WGANs is impressive, since to the best of my knowledge there was still a large performance gap between them.
While the results seem significantly better than the baselines, the paper could use more analysis of where this improvement comes from. How does the MMD-GAN eliminate the need for large mini-batches?  (Don't the samples in each mini-batch still need to essentially tile the space in order to achieve low MMD?)  Is there an explanation for why it outperforms the WGAN?
Minor comments:
- Section 3 describes the method in relation to the GAN, but isn't the WGAN a more direct match?  If I understand right, the MMD-GAN is essentially a kernelized W-GAN.
- "Also, the gradient has to be bounded, which can be done by clipping phi." Is this the same as the Lipschitz constraint from WGANs?  If so, why not use the regularization method from the "improved WGAN" paper, rather than the original version (which the authors found was less effective)?
- If the claim is that the proposed method prevents mode dropping, estimating log-likelihoods is probably a stronger quantitative way to test it. E.g., see Wu et al., 2017.
- The experiment of Section 5.3 doesn't seem to be measuring stability. (I assume this refers to avoiding the degenerate solutions that regular GANs fall into?)


Self-supervised Learning of Motion Capture
Current state-of-the-art solutions for motion capture from a single camera are optimization driven: they optimize the parameters of a 3D human model so that its re-projection matches measurements in the video (e.g. person segmentation, optical flow, keypoint detections etc.). Optimization models are susceptible to local minima. This has been the bottleneck that forced using clean green-screen like backgrounds at capture time, manual initialization, or switching to multiple cameras as input resource. In this work, we propose a learning based motion capture model for single camera input. Instead of optimizing mesh and skeleton parameters directly, our model optimizes neural network weights that predict 3D shape and skeleton configurations given a monocular RGB video. Our model is trained using a combination of strong supervision from synthetic data, and self-supervision from differentiable rendering of (a) skeletal keypoints, (b) dense 3D mesh motion, and (c) human-background segmentation, in an end-to-end framework. Empirically we show our model combines the best of both worlds of supervised learning and test-time optimization: supervised learning initializes the model parameters in the right regime, ensuring good pose and surface initialization at test time, without manual effort. Self-supervision by back-propagating through differentiable rendering allows (unsupervised) adaptation of the model to the test data, and offers much tighter fit than a pretrained fixed model. We show that the proposed model improves with experience and converges to low-error solutions where previous optimization methods fail.

This work addresses the problem of motion capture for monocular videos in the wild, where we are interested in estimating human poses in RGB videos. Instead of directly optimizing the parameters of a 3D human model to reduce the error measured by the metrics such as segmentation, optical flow, or the coordinates of key points, this paper proposes an end-to-end trainable framework to predict the parameters of the SMPL model. The optimization objective includes the projection error of key points, optical flow, and foreground-background segmentation. The model is tested on Surreal dataset and H3.6M dataset, compared with a pre-trained model and a direct optimization framework.
[Strengths]
- The paper proposes a set of promising objectives which can be utilized to fine-tune the models on real-world datasets, which are pre-trained on synthetic datasets.
- Figures are well-drawn and do help readers understand the proposed framework.
[Weakness]
- This paper is poorly written.
   - The motivation, the contribution, and the intuition of this work can barely be understood based on the introduction.
   - Sharing the style of citations and bullet items is confusing.
   - Representing a translation vector with the notation $t^{i} = [x^{i}, y^{i},  z^{i} ] ^{T}$ is usually more preferred.
- The experimental results are not convincing.
   - The descriptions of baseline models are unclear.
   - Comparing the performance of the methods which directly optimize the mesh parameters, rotation, translation, and focal length according to the metric provided (projection error) doesn't make sense since the objective is in a different domain of the measurement of the performance.
   - Comparing the performance of the model only pre-trained on synthetic data is unfair; instead, demonstrating that the proposed three projection errors are important is more preferred. In other words, providing the performance of the models pre-trained on synthetic data but fine-tuned on real-world datasets with different losses is necessary.
- The reason of claiming that it is a supervised learning framework is unclear. In my opinion, the supervision signals are still labeled.
[Reproducibility]
The proposed framework is very simple and well explained with sufficient description of network parameters and optimization details. I believe it's trivial to reproduce the results.
[Overall]
In term of the proposed framework, this paper only shows the improvement gained of fine-tuning the model based on the proposed losses defined by the reprojection errors of key points, optical flow, and foreground-background segmentation.
Taking into account that this work does show that fine-tuning the model pre-trained on synthetic datasets on real-world video clips improves the performance especially, it's still a convicting article.
In sum, as far as I am concerned this work makes a contribution but is insufficient.


From which world is your graph?
Discovering statistical structure from links is a fundamental problem in the analysis of social networks. Choosing a misspecified model, or equivalently, an incorrect inference algorithm will result in an invalid analysis or even falsely uncover patterns that are in fact artifacts of the model. This work focuses on unifying two of the most widely used link-formation models: the stochastic blockmodel (SBM) and the small world (or latent space) model (SWM). Integrating techniques from kernel learning, spectral graph theory, and nonlinear dimensionality reduction, we develop the first statistically sound polynomial-time algorithm to discover latent patterns in sparse graphs for both models. When the network comes from an SBM, the algorithm outputs a block structure. When it is from an SWM, the algorithm outputs estimates of each node's latent position.

(I have been asked by the area chairs to provide an extra set of eyes for this paper. Because this review is entering the discussion after the rebuttal phase, discussants were asked to consider anything raised here in light of this lack of an opportunity to give a rebuttal, as it is possible that I have misunderstood things that a rebuttal would have addressed.)
The work presents itself as contributing a unification of two widely studied random graph models, the stochastic block model and small world model, and developing an efficient algorithm for recovering latent representations for this unified model. A problem with this claim is that, as is acknowledged in the work, this “new” unified approach is really just a (sampled) graphon model, as the authors make clear on Lines 113-114. It’s plain to any reader familiar with graphons that adaptations of both the SBM and small world model fit within the graphon framework. The SBM-graphon connection is obvious and well-studied; the SW-graphon connection is obvious in-so-much-as the small world model is really just a latent space model, though most attention for latent space graphons has been on e.g. logistic latent space models. Denoting that model the "small world” model invokes expectations of navigability results, so in general the results would be better explained in terms of a latent space model with an inverse polynomial distance kernel. All that said, the work is considered on it's merits as a contribution to the graphon literature.
Detailed comments:
1) The way the paper presents it’s results does a disservice to their strengths. The paper contributes a careful analysis of a complex algorithm that appears to bring earlier results for sampled dense graphons over to sparse graphons (Line 255: "We generalize the results in [24] for the sparse graph case.”) That said, while the results in [24] are explicitly given for dense graphs, they are sketched there for the sparse case as well (in the conclusions, with the comment "A precise statement and formulation of the results in Section 4 for the sparse setting might require some care, but should be for the most part straightforward.”).
It is possible that the details of the sketch in [24] are not nearly as “straight-forward” as the authors there claim. But engaging in a clear dialogue with the literature is a reasonable requirements for acceptance, in the eyes of this reviewer. If the paper had a (1) title and abstract and (2) related work section that made clear the position of the paper in the sampled graphon literature, specifically related to [24], that would _greatly_ improve the ability of a reader to appreciate the work.
As is, the results pertaining to recovering both SBM and SWM networks claimed in the abstract and introduction are essentially results achieved by [24], except for (a) the sparse graph case (the polynomial small world kernel is a universal kernel in the language of the results of that work) and (b) the pairwise distance recovery. I will now focus on the pairwise distance recovery. Basically, [24] is quite general, as is the core of this work. And the focus of the title and introduction on mis-specification between the SBM and SWM feels like the wrong problem to focus on when putting forward this work.
2) Connecting the extended analysis of [24] -- roughly, to get the kernel PCA to work in the sparse regime -- is a nice if modest contribution. More significant is connecting the output of this machinery to MDS machinery to obtain the pairwise distances. This appears to be a novel point, and it seems quite useful. The analysis here does rely on some specific details of the small-world kernel, but it would be really helpful if the authors clarified what the general conditions are on the latent space kernel for some version of pairwise distance recovery to go through. It seems that the long-range links allowed by the heavy polynomial tails may be critical; is that so? Do any version of the results work for kernels with exponential tails?
3) In line with the odd focus of the introduction, the empirical comparisons made in the work are a bit off, but I'm willing to let this slip as the work isn't really an empirical paper, and should get positive "credit" for even trying to do empirical work here. The empirical work focuses on comparisons with algorithms that aren’t quite trying to do what the present algorithm is trying to do, while not benchmarking itself against methods that are designed for some of the same tasks. First, the SBA algorithm of Airoldi-Costa-Chan and SAS algorithm of Chen-Airoldi, and related network histogram methods are specifically designed for the estimation problem in Figure 4. But not compared. Second, the methods used in comparison are not really reasonable. Majority vote label propagation from [56] is an algorithm for categorical labels, not continuous embeddings. It’s not clear why [54], a paper about conformal Isomap, is called a random walk label propagation method, so that might be a crossed-up citation. It is likely that the modularity method [55] is finding a core-periphery cut with it’s main eigenvector; for this reason it would be interesting to see how a semi-supervised method does (e.g. ZGL, given the labels for the presidential candidates discussed as being given to other methods).
As an overall comment, the empirical question is a terrific one, with high quality data, but the evaluation is not up to the standards of a thorough empirical investigation. But again, I realize that the main contribution of this work is theoretical, but I still want to register what I see as issues. I am most concerned about the lack of comparison to the other methods in the estimation in Figure 4 and Figure S9.
3) As a small comment about Figure (S)9:  for the methods under consideration it would be more honest to show the performance of the Abraham et al. algorithm in Fig 9d and the Newman spectral algorithm in Fig 9b, as opposed to just showing the cross-objectives (where the two methods fail).
4) One point that confused this reader for some time was the fact that the paper studies sparse graphons without any discussion of the tricky issues for such models. It should be made very clear that the model under consideration is the Bollobas-Janson-Riordan (2007) model (also studied by Bickel-Chen-Levina (2011), which is a sampled dense graphon. Some pointers to the emerging literature on sparse graphons defined through edge-exchangeable ideas (Crane-Dempsey 2016 building on Caron-Fox 2014, Veitch-Roy 2015, Borgs et al. 2016) would be useful to provide.
5) The related work section overlooks that [22], Airoldi et al., makes computational improvements (in terms of complexity) on the USVT method (but still for dense graphs).
6) Ref [19], the latent space model of Hoff et al., is pegged as a “Graphon-based technique” where it should be raised as a broad citation for latent space models such as the small world model discussed herein.
7) The discussion of results for both the unipartite and bipartite graph model is nice. The bipartite model discussed here is sometimes called an affiliation network, see "Affiliation networks” by Lattanzi and Sivakumar, perhaps worth connecting to.
I think there are a lot of interesting connections made in the work. The management of errors in the isomap algorithm for recovering latent space graph models is a nice contribution.


Speeding Up Latent Variable Gaussian Graphical Model Estimation via Nonconvex Optimization
We study the estimation of the latent variable Gaussian graphical model (LVGGM), where the precision matrix is the superposition of a sparse matrix and a low-rank matrix. In order to speed up the estimation of the sparse plus low-rank components, we propose a sparsity constrained maximum likelihood estimator based on matrix factorization, and an efficient alternating gradient descent algorithm with hard thresholding to solve it. Our algorithm is orders of magnitude faster than the convex relaxation based methods for LVGGM. In addition, we prove that our algorithm is guaranteed to linearly converge to the unknown sparse and low-rank components up to the optimal statistical precision. Experiments on both synthetic and genomic data demonstrate the superiority of our algorithm over the state-ofthe-art algorithms and corroborate our theory.

The paper considers learning the dependency structure of Gaussian graphical models where some variables are latent. Directly applying the usual assumption of sparsity in the precision matrix is difficult because variables that appear correlated might actually both depend on a common latent variable. Previously, Chandrasekaran et al. proposed estimating the model structure by decomposing the full precision matrix into the sum of of a sparse matrix and a low-rank matrix. Likelihood is maximized while the components of the sparse matrix are penalized with an l1 regularizer and the low-rank matrix is penalized with a nuclear norm. Computing the proximal operator to update the low-rank component requires performing SVD in O(d^3) time at each iteration.
The authors propose replacing the low-rank component with its Cholesky decomposition ZZ^T and finding Z directly. The main technical contribution is that they prove linear converge of an alternating gradient algorithm, up to optimal statistical precision.
Experiments show that by avoiding the need for SVD at each iteration, the proposed method is orders of magnitude faster than the method of Chandrasekaran et al. or a related approach that uses ADMM to estimate the model. It is also more accurate on synthetic and gene network data.
Is Huang, Jianhua Z., et al. "Covariance matrix selection and estimation via penalised normal likelihood." Biometrika 93.1 (2006): 85-98 related? They propose regularizing the Cholesky factorization for precision matrix estimation, although without any latent variables. Their motivation is that it avoids needing to check that the estimated precision matrix is PSD, which takes O(d^3) time.


Convolutional Gaussian Processes
We present a practical way of introducing convolutional structure into Gaussian processes, making them more suited to high-dimensional inputs like images. The main contribution of our work is the construction of an inter-domain inducing point approximation that is well-tailored to the convolutional kernel. This allows us to gain the generalisation benefit of a convolutional kernel, together with fast but accurate posterior inference. We investigate several variations of the convolutional kernel, and apply it to MNIST and CIFAR-10, where we obtain significant improvements over existing Gaussian process models. We also show how the marginal likelihood can be used to find an optimal weighting between convolutional and RBF kernels to further improve performance. This illustration of the usefulness of the marginal likelihood may help automate discovering architectures in larger models.

The paper presents a method to incorporate convolutional structure in GPs, in order to make them more competitive for image classification problems. This is achieved by constructing a convolutional kernel through a latent function g() shared by different patches in the image.
Approximate inference is based on variational methods and very promising results are given in MNIST and CIFAR datasets.
The paper is clearly written and  it presents a quite extensive set of experimental results. Regarding the main construction of the convolutional kernel (see eq. 13 and 14) I am not sure if
the concept of inter-domain inducing variables is really necessary or useful here. The latent function of the GP model is essentially g() which is used inside the likelihood through this patch-sharing, i.e. through the definition of f(x). Under this viewpoint the inducing variables are just points of the latent function, i.e. points of g(), and you don't really need to refer to inter-domains in order to describe the variational sparse approximation. Also having
presented the method based on g() (and discussing f(x) only as a complementary viewpoint) will probably make it easier for the reader to understand the weighted and colour extensions.
Can you clarify further the arguments in lines 215-220? In particular, it is not clear why you avoid the inversion of a 2 M × 2 M matrix given that q(u1,u2) is coupled. Section 3 in Supplementary material, that tries to clarify this point, is very unclear and it contains also mistakes (e.g. in equation 13).
Overall I believe that this is a good paper since it improves a lot on GP classification by incorporating convolutional structure. Still of course GPs cannot compete with Deep NNs, but this paper introduces some new ideas and it makes good steps forward to making GP classifiers more competitive for image classification.


Distral: Robust Multitask Reinforcement Learning
Most deep reinforcement learning algorithms are data inefficient in complex and rich environments, limiting their applicability to many scenarios. One direction for improving data efficiency is multitask learning with shared neural network parameters, where efficiency may be improved through transfer across related tasks. In practice, however, this is not usually observed, because gradients from different tasks can interfere negatively, making learning unstable and sometimes even less data efficient. Another issue is the different reward schemes between tasks, which can easily lead to one task dominating the learning of a shared model. We propose a new approach for joint training of multiple tasks, which we refer to as Distral (distill & transfer learning). Instead of sharing parameters between the different workers, we propose to share a "distilled" policy that captures common behaviour across tasks. Each worker is trained to solve its own task while constrained to stay close to the shared policy, while the shared policy is trained by distillation to be the centroid of all task policies. Both aspects of the learning process are derived by optimizing a joint objective function. We show that our approach supports efficient transfer on complex 3D environments, outperforming several related methods. Moreover, the proposed learning process is more robust to hyperparameter settings and more stable-attributes that are critical in deep reinforcement learning.

The paper presents an approach to performing transfer between multiple reinforcement learning tasks by regularizing the policies of different tasks towards a central policy, and also encouraging exploration in these policies.  The approach relies on KL-divergence regularization.  The idea is straightforward and well explained.  There are no theoretical results regarding the learning speed or quality of the policies obtained (though these are soft, so clearly there would be some performance loss compared to optimal).  The evaluation shows slightly better results that A3C baselines in both some simple mazes and deep net learning tasks.
While the paper is well written, and the results are generally positive, the performance improvements are modest.  It would have been nice also to quantify the performance in *runtime* instead of just in terms of number of samples used (since the networks architectures are different).  In the tabular case, it would really have been nice to show some more detailed experiments to understand the type of behaviour obtained.  Eg. what happens if goals are sampled uniformly (as in these experiments) vs all sampled in one or the other of the extreme rooms, and what happens as the number of sampled tasks increases. Also, another baseline which does not seem to be included but would be very useful is to simply regularize task policies towards literally a "centroid" of the suggested actions, without learning a distilled policy.  One could simply average the policies of the different tasks and regularize towards that quantity.  It is not clear if this would be competitive with the proposed approach, but conceptually it should lead to very similar results.  If the number of tasks increases, one would expect the distilled policy would in fact simply become uniform - is this the case?
Finally, It would really have been nice to see some attempt at theory in the tabular case. It seems to me that one ought to be able to bound the loss of this approach wrt optimal policies using a variational approach, since the optimization seems to suggest that the algorithm computes a variational approximation to what would be the optimal policy for a distribution of MDPs. Such a result would make the paper more interesting conceptually and give some confidence in its applicability beyond the domains used in the experiments.
Smaller suggestions:
- The title refers to "robustness" which in RL is a technical term regarding optimization wrt an unknown distribution of dynamics (using typically min-max algorithms). This is not the type of approach proposed here, so some clarification is needed.


Robust Optimization for Non-Convex Objectives
We consider robust optimization problems, where the goal is to optimize in the worst case over a class of objective functions. We develop a reduction from robust improper optimization to stochastic optimization: given an oracle that returns ↵-approximate solutions for distributions over objectives, we compute a distribution over solutions that is ↵-approximate in the worst case. We show that derandomizing this solution is NP-hard in general, but can be done for a broad class of statistical learning tasks. We apply our results to robust neural network training and submodular optimization. We evaluate our approach experimentally on corrupted character classification and robust influence maximization in networks.

Summary:
This work provides a method to optimize the worst-case loss over a set of different class of non-convex objective functions. In the real world, observed signals may contain various source of noise, which requires the learning method to be robust enough to the noise. The author converted the robust improper optimization to a Bayesian optimization problem. The proposed learning procedure consists of two iterative steps: the learner tries to minimize the loss with an alpha-approximate Bayesian oracle; the adversary aims to find a distribution over losses that maximize the loss of the learner. The author provide detailed proofs for the convergence and the theoretical bounds of the algorithm. Specifically, a robust statistical learning and a robust submodular maximization cases were discussed. Supportive application experiments were performed to validate the method. The result shows that the proposed method has substantial improvements in the worst-case scores.
Quality:
The paper is well written with theoretical proofs and supportive experiments. The author compared the proposed method with many modifications to support their choice.
Clarity:
The paper is well organized. The experiment details are clearly written.
Originality:
There are a lot of works in robust optimization in machine learning. This work focuses on more general setting of robust optimization where loss functions are non-convex. The application of alpha-approximate Bayesian oracle is a novel approach.
Significance:
This work provides a general approach for robust optimization of non-convex objective functions. The method is validated from both a neural network learning task and an influence maximization task and it substantially outperformed the baseline approaches. It leads the state-of-art in robust optimization for non-convex problems.


Model evidence from nonequilibrium simulations
The marginal likelihood, or model evidence, is a key quantity in Bayesian parameter estimation and model comparison. For many probabilistic models, computation of the marginal likelihood is challenging, because it involves a sum or integral over an enormous parameter space. Markov chain Monte Carlo (MCMC) is a powerful approach to compute marginal likelihoods. Various MCMC algorithms and evidence estimators have been proposed in the literature. Here we discuss the use of nonequilibrium techniques for estimating the marginal likelihood. Nonequilibrium estimators build on recent developments in statistical physics and are known as annealed importance sampling (AIS) and reverse AIS in probabilistic machine learning. We introduce estimators for the model evidence that combine forward and backward simulations and show for various challenging models that the evidence estimators outperform forward and reverse AIS.

This paper discusses tackling marginal likelihood
estimation, a central problem in Bayesian model selection, using
insights provided by so-called nonequilibrium fluctuation
theorems. While Jarzynksi's Equality (JE) allows a natural sequential
Monte Carlo estimator of log Z, the estimator can be high-variance if
the resultant sequence of nonequilibrium distributions have
insufficient overlap, which is a common problem for complex,
multimodal problems. The fluctuation theorem allows (under generally
non-realistic assumptions; more shortly) a bounding of the estimate no
matter how fast the annealing schedule is. In addition to providing
bounds, the reverse work distribution can be incorporated into
estimators for log Z itself that are (hopefully) an improvement over
the one provided by JE. Experiments are run on various statistical
models.
I found this to be an interesting paper and enjoyed reading it. My
main reservations are concerned with the feasibility of simulating
the reverse process in most situations that are
practically-challenging. It appears that the authors do recognize this
limitation, but provide a brief and not overly satisfying discussion
at the end of p.4. Most of the experimental results focus on problems
with a simple low-energy structure (the ferromagnetic Ising and Potts
models, and mixture models with 3 components) The RBM example is a bit
more intriguing, but when trained on MNIST, one can reasonably infer
the existence of a relatively small number of modes. In general, the
method assumes access to an oracle providing a perfectly-trained
model, where the data points do indeed represent the modes of the
distribution (1000 sweeps of Gibbs sampling at low temperature will
move only locally and cannot be expected to move the state too far
from the training points.) While the agreement of the current method
and PT log Z results in this case is encouraging, the fit of the model
is generally difficult to verify in practice- indeed that's the
reason for conducting the simulation in the first place.
To phrase the objections in another way, optimization is generally a
very different problem from sampling, as represented by the
distinction between problems in NP and those in #P. There exist
several interesting types of problems for which computing the ground
state energy is tractable, but counting the number of such minima (obtaining the
ground state entropy) is not. Generating samples uniformly from the
ground state distribution, as would be required for the reverse
methodologies, will then be problematic.
Another issue was that on first reading, it was not immediately clear
what the paper's contributions are; presumably, it is the combined
forward/reverse weighted estimator presented on page 5? The remaining
ones have been previously introduced in the literature. While valid,
I'm ambivalent as to whether this contribution warrants an entire NIPS
paper. Certainly, the connection between marginal likelihood and free
energy estimation is widely-known among Bayesian computational
statisticians.
Despite these misgivings, I still found the paper to be quite cool and
am encouraged to see progress in this research area.
Technical correctness:
-------------------
The paper appears to be technically correct to my knowledge.
Clarity:
-------
The paper is quite well-written and easy to follow for someone
familiar with Sequential Monte Carlo and Bayesian statistics. The
presentation could be better-adapted to the NIPS context, however, as
physics-flavored notions such as "work" may seem distant to a machine
learning audience.
Specific issues:
---------------
252-253: In what sense is simulation of marginal RBM 3x faster than
that of the joint? The joint simulation can be parallelized and
simulated on a GPU due to the conditional independence structure of
the RBM. The statement should be clarified.


Accelerated consensus via Min-Sum Splitting
We apply the Min-Sum message-passing protocol to solve the consensus problem in distributed optimization. We show that while the ordinary Min-Sum algorithm does not converge, a modified version of it known as Splitting yields convergence to the problem solution. We prove that a proper choice of the tuning parameters allows Min-Sum Splitting to yield subdiffusive accelerated convergence rates, matching the rates obtained by shift-register methods. The acceleration scheme embodied by Min-Sum Splitting for the consensus problem bears similarities with lifted Markov chains techniques and with multi-step first order methods in convex optimization.

In this paper, the authors present an accelerated variant of the Min-Sum message-passing protocol for solving consensus problems in distributed optimization. The authors use the reparametrization techniques proposed in [Ruozzi and Tatikonda, 2013] and establish rates of convergence for the Min-Sum Splitting algorithm for solving consensus problems with quadratic objective functions. The main tool used for the analysis is the construction of an auxiliary linear process that tracks the evolution of the Min-Sum Splitting algorithm.
The main contributions of the paper can be summarized as follows: (i) provide analysis for the Min-Sum splitting algorithm using a new proof technique based on the introduction of an auxiliary process, (ii) design a Min-Sum protocol for consensus problems that achieves better convergence than previously established results, and (iii) show the connection between the proposed method, and lifted Markov chains and multi-step methods in convex optimization.
The motivation and contributions of the paper are clear. The paper is well written and easy to follow, however, it does contain several typos and grammatical mistakes (listed below). The proofs of Propositions 1 and 2, and Theorem 1 appear to be correct.
Typos and Grammatical errors:
- Line 34: “…with theirs neighbors…” -> “…with their neighbors…”
- Line 174: “double-stochastic” -> “doubly-stochastic”
- Line 183: “… can be casted as…” -> “… can be cast as…”
- Line 192: “…class of graph with…” -> “…class of graphs with…”
- Line 197: “…which seems to…” -> “…which seem to…”
- Line 206: “…additional overheads…” -> “…additional overhead…”
- Line 225: “…pugging…” -> “…plugging…”
- Line 238: “…are seen to…” -> “…are able to…”
- Line 240: “…both type of…” -> “…both types of…”
- Line 248: “…also seen to…” -> “…also shown to…”
- Line 279-280: “…to convergence to…” -> “…to converge to…”
- Line 300: “…,which scales like…” -> “…,which scale like…”
- Line 302: “…for the cycle,…” -> “…for cycle graphs,…”
Other minor comments:
- Lines 220 and 221: Do you mean “Lagrangian” and “Lagrange multipliers” instead of “Laplacian” and “Laplace multipliers”?
- The authors present 3 algorithms, and the quantities involved are not always explained or described. For example, what is R_{vw} and r_{vw} in Algorithm 2? Also, in Algorithm 2, the quantities \hat{R}^0 and \hat{r}^0 do not appear to be initialized. Moreover, since the auxiliary linear process is key to the analysis and the central idea of the paper, the authors show clearly state which variables correspond to this in Algorithm 3.
The paper also appears to be missing several references. More specifically:
- Lines 41 and 43: (Sub)gradient methods for consensus optimization. There are several more references that could be included:
-- Bertsekas and Tsitsiklis, Parallel and distributed computation: numerical methods, 1989
-- Sundhar Ram Srinivasan et. al., Incremental stochastic subgradient algorithms for convex optimization, 2009
-- Wei Shi, Extra: An exact first-order algorithm for decentralized consensus optimization, 2015
(and, of course, many more)
- Line 170: “The original literature…”
- Line 229: work by Polyak (Heavy-ball)
- Line 232: work by Nesterov
It would be interesting and useful if the authors could answer/comment and address in the paper the following:
- Although the paper is a theoretical paper, the authors should comment on the practicality of the method, and when such a method should be used as opposed to other distributed methods for consensus optimization.
- What are the limitations of the Min-Sum Splitting method?
- What is the intuition behind using the auxiliary process in the Min-Sum Splitting method?
- The results provided in this paper are for consensus problems with quadratic objective functions. Can this framework be extended to solve more general consensus problems that often arise in Machine Learning?
- The authors should also clearly state why such an approach is of interest in the context of Machine Learning and for the Machine Learning community.
In summary, this paper is a purely theoretical paper in which the authors establish rates of convergence using a new proof technique and show the connections between their method and well-established methods in the literature. Overall, the ideas presented in this paper are interesting, however, the practicality of the method and intuition behind the results are missing, as well as some justification for the  importance of this result for the Machine Learning community.


Monte-Carlo Tree Search by Best Arm Identification
Recent advances in bandit tools and techniques for sequential learning are steadily enabling new applications and are promising the resolution of a range of challenging related problems. We study the game tree search problem, where the goal is to quickly identify the optimal move in a given game tree by sequentially sampling its stochastic payoffs. We develop new algorithms for trees of arbitrary depth, that operate by summarizing all deeper levels of the tree into confidence intervals at depth one, and applying a best arm identification procedure at the root. We prove new sample complexity guarantees with a refined dependence on the problem instance. We show experimentally that our algorithms outperform existing elimination-based algorithms and match previous special-purpose methods for depth-two trees.

This paper treats the problem of Monte Carlo Tree Search in depth-k game trees. Even though the problem was considered last year for depth-two trees in 'Optimal best arm identification with fixed confidence' (COLT 2016), the present paper considerably extends this previous work that could not directly be generalized to higher depths.
They propose a new type of algorithm that can be run with different Best Arm Identification (BAI) algorithms and they provide the analysis for LUCB and UGapE. The idea relies on two main ingredients : First, the BAI algorithm leads the learner in its first decision departing from the root of the tree and then, the subsequent 'representative leaf' is sampled. This leaf is chosen by following a somehow greedy path along the tree, each time choosing the action that maximizes (resp. minimizes) the expected value.
This line of work sheds a new light on MCTS problems by bringing the tools developed in the framework of Best Arm Identification. The proposed method has several qualities :
1. It is fairly intuitive and easy to understand for the reader having some ground knowledge of the BAI literature;
2. It is computationally efficient, allowing to conduct convincing experiments;
3. It outperforms the existing algorithm FindTopWinner (see comment below).
The authors also prove a problem-dependent lower bound following the now classic technics from e.g. (Garivier and Kaufmann, 2016), (Kaufmann et al., 2015). Even though the final result of Lemma 7 is not fully explicit, the optimization problem can be numerically solved. This allows in particular to have a comparison basis for the experimental section, on top of existing algorithms. Also, as mentioned in the comments of Lemma 7, such a lower bound result gives a hope towards finally obtaining an asymptotically  optimal algorithm following the Track-And-Stop technique from (Garivier and Kaufmann, 2016). This is indeed an exciting perspective, coming with a few other new problems opened by this paper.
I only have minor comments:
- The Lemmas 13 and 14 seems to be the key tools that allow to control the behavior of the algorithm with respect to the choice of the 'representative leaf'. However, this intuition is not really clear in the proof of Theorem 3 that is sketched in the main paper. Space requirements don't make it easy but this could make the idea of the proof even clearer.
- The 'update information about the arms' in the Figure 2 is quite unclear. Actually, the exact action of the algorithm is a bit tricky, I had to read several times to understand whether the algorithm only chooses the first step or all the path of actions and where the information/feedback finally shows up. This could be made a bit clearer in the algorithm pseudo-code maybe.
- l 255 : 'uniformLY at random';
- Lemma 11 in Appendix, please chose i or k to index s;
Overall, I strongly recommend accepting this paper as it opens a new path towards the understanding of MCTS problems and may lead to future major advances.


Summary:
"A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning" presents a novel scalable algorithm that is shown to converge to better behaviours in partially-observable Multi-Agent Reinforcement Learning scenarios compared to previous methods. The paper begins with describing the problem, mainly that training reinforcement learning agents independently (i.e. each agent ignores the behaviours of the other agents and treats them as part of the environment) results in policies which can significantly overfit to only the agent behaviours observed during training time, failing to generalize when later set against new opponent behaviours. The paper then describes its solution, a generalization of the Double Oracle algorithm. The algorithm works using the following process: first, given a set of initial policies for each player, an empirical payoff tensor is created and from that a meta-strategy is learnt for each player which is the mixture over that initial policy set which achieves the highest value. Then each player i in the game is iterated, and a new policy is trained against policies sampled from the meta-strategies of the other agents not equal to i. After training, this new policy is added to player i's set of policies. After all agents have trained a new policy, the new entries within the empirical payoff tensor are computed and the meta-strategies are updated. Finally, the process repeats again, training another policy for each agent and so on. This algorithm, called "Policy-Space Response Oracles" (PSRO), is shown to be a generalization of several previous algorithms including independent RL, fictitious play and double oracle. An extension is then developed, called Deep Cognitive Hierarchies (DCH), which operates only on a tractable hierarchy of asynchronously updated policies instead of having meta-strategies learnt over the entire set of all previous policies. PSRO and DCH are then empirically evaluated on several 2D grid world games and Leduc poker. To identify overfitting "joint policy correlation" matrices are constructed on which the "average proportional reduction" metric is computed. This metric makes clear that independent RL does very poorly at generalizing to new opponent behaviours, even when those opponents were trained under identical conditions. Finally, it is shown that PSRO and DCH converge to close to an approximate Nash Equilibrium much faster than Neural Fictitious Self-Play, a related multi-agent algorithm which also uses deep neural network function approximators.
Qualitative Assessment:
I think the paper presents an interesting and novel solution to the important problem of learning generalizable behaviours in multi-agent environments. The algorithm developed is also scalable and lends itself to be used efficiently with modern function approximators like deep neural networks. The experimental results clearly demonstrate the problem with the naive method of independent RL, with the development of the "joint policy correlation" matrices showing that independent RL gets far lower reward when set against a novel behaviour. It is interesting to see how brittle RL algorithms are if other agents are treated simply as part of the environment. The connections between PSRO and previous methods such as double oracle and fictitious play are informative and give an intuition on what each method is actually computing in an easily comparable way. The related work cited is extensive and covers a wide variety of methods. While it quickly covers a lot of ground, the writing is still clear and easy to follow. The hyperparameter settings are given in the appendix to help others reproduce the work.
I have some clarifying questions:
1) Is there a strict reason for using a hierarchy of policies in DCH? Why restrict levels from only considering oracles in previous levels? I understand there is some intuition in that it resembles Camerer, Ho, & Chong’s cognitive hierarchy model, but I was wondering if there was another reason to impose that constraint.
2) In Figure 5 (a) and (b), it seems like NFSP has not plateaued completely. Does NFSP converge to roughly the same NashConv as PSRO/DCH, or a slightly better solution? It would be informative to see the long term performance (until convergence) of all methods.
3) The DCH algorithm seems to have a large number of hyperparameters, with RL agents and decoupled meta-solvers needing to be trained asynchronously. How sensitive is the algorithm's performance to hyperparameters? Did it require a large hyperparameter search to get it working?
Typos:
1) There is a sentence fragment at line 280.
2) At line 266, should the upper-case sigma be an upper-case delta (meta-strategy mixture)?


Model-based Bayesian inference of neural activity and connectivity from all-optical interrogation of a neural circuit
Population activity measurement by calcium imaging can be combined with cellular resolution optogenetic activity perturbations to enable the mapping of neural connectivity in vivo. This requires accurate inference of perturbed and unperturbed neural activity from calcium imaging measurements, which are noisy and indirect, and can also be contaminated by photostimulation artifacts. We have developed a new fully Bayesian approach to jointly inferring spiking activity and neural connectivity from in vivo all-optical perturbation experiments. In contrast to standard approaches that perform spike inference and analysis in two separate maximum-likelihood phases, our joint model is able to propagate uncertainty in spike inference to the inference of connectivity and vice versa. We use the framework of variational autoencoders to model spiking activity using discrete latent variables, low-dimensional latent common input, and sparse spike-and-slab generalized linear coupling between neurons. Additionally, we model two properties of the optogenetic perturbation: off-target photostimulation and photostimulation transients. Using this model, we were able to fit models on 30 minutes of data in just 10 minutes. We performed an all-optical circuit mapping experiment in primary visual cortex of the awake mouse, and use our approach to predict neural connectivity between excitatory neurons in layer 2/3. Predicted connectivity is sparse and consistent with known correlations with stimulus tuning, spontaneous correlation and distance.

This papers proposes an inference method of (biological) neural connectivity from fluorescence (calcium ) traces. The model includes the spiking model (GLM+low-rank factor) with an external input (optical stimulation) and a fluorescence model. The inference methods is based on variational Bayes, where the approximate posterior is modeled using a neural network.
Novelty and originality: The methods in this paper are adequately novel and original, nicely combining various elements from previous work.
Technical issues: My main problem with this paper is that I can't really be sure that the proposed method is actually working well. It is very good that the authors tested their method on real data, but since there is no ground truth, I it is hard to estimate the quality of the inferred weights (see footnote (1) below). Given that there is no ground truth, what would really help to convince the reader that everything is working as it should, is some basic check on simulated data to show that the inferred weights are reasonably accurate. Also, it is hard to interpret Fig 5 due to its scale: how can the reader know that an improvement from 1.02 to 1.05 in the ELBO is significant?
Clarity: The methods part seems a bit dense, with various details not completely clear from the text. I would suggest adding more complete details in the supplementary, such as the complete objective being optimized. This would also make it easier for the reader to understand the context of section 2.5. Also, from section 2.1 (e.g., lines 69-70) I thought that the weights are also treated as latent variables, and so have a recognition model. But since no recognition model is specified for the weights, so I guess I was wrong.
Minor issues:
1) In Figure 2A, the approximate "sum of ReLU" softplus, we have c > 0, i.e. it to does not go to zero for low firing rates. Maybe using an exponential tail in that section would improve the approximation (at is still integrates with a Gaussian, and can upper bound a logistic tail).
2) For Gaussians, the notation is supposed to be N(mu,sigma^2), not N(mu,sigma).
3) line 192: Fig 6B -> Fig 6E.
4) Figure 6 third line in caption: remove "A."
5) The index 'c' is for "cell"? This should be made clearer.
Footnote:
(1) The authors argue that their method works since the properties of the detected weights match the expected biological statistics (Fig. 6), but this does not seem convincing. Specifically, It is a good sanity check, but these results only indicate that the method does not fail completely: it is not hard to think of cases in which we get the same properties, yet the estimate weights are highly inaccurate. For example, the observed distance dependence (Fig 6B) could be related to partial mixing of fluorescence signal of nearby neurons. With stimulation this could also be related to the "area affect" of the stimulation (Fig 6F), though this is partially compensated by the method. Similarly, the correlation of weight strength to strong Spont. and Signal. correlations might be the result that all of these factors typically increase with the firing rate of the neuron.
%%% After authors' feedback %%%
The new results and clarifications address my concerns. I therefore have modified my score. I would like to see the limit that the correlation -> 1 on the synthetic data set in the revised paper, to make sure the method is consistent.


Gaussian process based nonlinear latent structure discovery in multivariate spike train data
A large body of recent work focuses on methods for extracting low-dimensional latent structure from multi-neuron spike train data. Most such methods employ either linear latent dynamics or linear mappings from latent space to log spike rates. Here we propose a doubly nonlinear latent variable model that can identify low-dimensional structure underlying apparently high-dimensional spike train data. We introduce the Poisson Gaussian-Process Latent Variable Model (P-GPLVM), which consists of Poisson spiking observations and two underlying Gaussian processes-one governing a temporal latent variable and another governing a set of nonlinear tuning curves. The use of nonlinear tuning curves enables discovery of low-dimensional latent structure even when spike responses exhibit high linear dimensionality (e.g., as found in hippocampal place cell codes). To learn the model from data, we introduce the decoupled Laplace approximation, a fast approximate inference method that allows us to efficiently optimize the latent path while marginalizing over tuning curves. We show that this method outperforms previous Laplace-approximation-based inference methods in both the speed of convergence and accuracy. We apply the model to spike trains recorded from hippocampal place cells and show that it compares favorably to a variety of previous methods for latent structure discovery, including variational auto-encoder (VAE) based methods that parametrize the nonlinear mapping from latent space to spike rates with a deep neural network.

Summary
---------
The authors propose a gaussian process latent variable model with poisson observations (P-GPLVM). The demonstrate how to perform inference using a decoupled Laplace approximation. They apply the model to simulated data as well as rat hippocampal data.
The paper is well written and clearly presented. I appreciated the nice summary of the model structures in Table 1.
I would have liked to see more applications of the technique, and/or a larger improvement on real data compared to previous methods.
Questions/concerns
--------------------
- The performance benefit of the P-GPLVM is quite small over PLDS and the GPLVM on real data. For example, in Figure 2C, there is only a weak improvement over these methods (especially given the fact that the y-scale in Fig 2C shows a small range, and lack error bars), lessening the overall impact of the results.
- How does the algorithm scale with the size of the latent dimension, the number of neurons, and the length of the recording? The examples in the paper have latent dimension <= 2, on the order of tens of neurons, and 500 time points (50s binned at 100ms). Is the technique feasible for larger datasets?
- The authors ran multiple simulations and show the mean across repeats--the authors should also show the variability (standard error or deviation) across repeats, both for the r^2 performance (error bars) as well as estimated trajectories (error bands).
- If I understand correctly, the estimated latent trajectory in both the simulation and hippocampal data tracked the animal's position? That seems surprising--what if you set the latent dimension to 3? or 4, or 1?
Minor comments
-----------------
- line 103: remove the 'as' in "our model is as a doubly nonlinear ..."


Convergence of Gradient EM on Multi-component Mixture of Gaussians
In this paper, we study convergence properties of the gradient variant of Expectation-Maximization algorithm [11] for Gaussian Mixture Models for arbitrary number of clusters and mixing coefficients. We derive the convergence rate depending on the mixing coefficients, minimum and maximum pairwise distances between the true centers, dimensionality and number of components; and obtain a near-optimal local contraction radius. While there have been some recent notable works that derive local convergence rates for EM in the two symmetric mixture of Gaussians, in the more general case, the derivations need structurally different and non-trivial arguments. We use recent tools from learning theory and empirical processes to achieve our theoretical results.

Summary: This paper derives statistical guarantees for the gradient EM algorithm applied to clustering in Gaussian mixture models, generalizing the results of [1] to multiple clusters and non-uniform cluster weights.
The first result (Theorem 1) concerns the "population EM algorithm", i.e., when the expectation step can be computed exactly rather than being estimated from data. For this case, if the cluster centers are sufficiently separated and all the estimated cluster centers are initialized sufficiently close to the true cluster centers (i.e., roughly if there is already a sufficiently clear 1-1 correspondence between the estimated and true cluster centers), then the estimated cluster centers converge linearly to the true centers. This result relies on first showing (Theorem 4) that, under these conditions, due to properties of the Gaussian distribution, the curvature of the Q function is bounded (i.e., specifically, the gradient is Lipschitz). By a standard proof, the Lipschitz gradient then implies convergence of gradient descent.
The paper then turns to showing that finite-sample gradient EM approximates population EM, based on a generalization [2] of McDiarmid's inequality (to unbounded data), and a novel bound on the Rademacher complexity of the gradient for GMMs. This in turn implies a linear convergence rate for finite-sample gradient EM, assuming sufficiently good initialization.
Finally, the paper concludes with some synthetic experiments verifying the linear convergence rate and the necessity of the assumption that the initial clusters are good.
Main Comments:
This paper proves fairly strong, tight convergence bounds on a classic clustering algorithm, significantly generalizing the results of [1]. The proofs are well-motivated and lucidly sketched. I'm not too familiar with related work in this area, but I believe the results are quite novel. Overall, this paper is a significant contribution. However, the paper needs extensive proof-reading (see some of the typos noted below).
Some thoughts:
1) The results rely on having a sufficiently good initialization of the estimated cluster centers. Although haven't thought in much detail, I suspect that (perhaps assuming the clusters centers are known to lie in a bounded region), it should be quite easy to derive a high-probability bound the number of random initializations needed to obtain such an initialization. Such a result might nicely complement the given results.
2) It is noted (Remark 4) that the altered version (from [2]) of McDiarmid's inequality used results in extra factors of sqrt(d) and log(n). I wonder if the recent variant (see [R1] below) of McDiarmid's inequality for sub-Gaussian data can give tighter results? This bound appears somewhat more similar to the original (bounded-differences) version of McDiarmid's inequality, though perhaps it is equivalent to that in [2].
The paper has a lot of typos and minor language issues:
Line 18: "loglikelihood" should be "log-likelihood"
Line 26 and Line 142: "convergence guarantees... is" should be "convergence guarantees... are"
Line 27: "converge to arbitrarily" should be "converge to an arbitrarily"
Line 30: The sentence "In [16]... is continuous." doesn't parse.
Line 52: "semi definite" should be "semi-definite"
Line 69: "Ledoux Talagrand" should be "Ledoux-Talagrand"
Line 122: "defined in 1". What is "1"? Should this be the reference [1]?
Lines 139-144: It looks like part of the related work section was randomly pasted into the middle of a sentence that should read "... contraction results [9] coupled with arguments that use..."
Line 187: "equivalent as" should be "equivalent to"
Hyphenation (e.g., "sub-Gaussian", "sub-optimal") is inconsistent throughout the paper.
[R1] Kontorovich, Aryeh. "Concentration in unbounded metric spaces and algorithmic stability." Proceedings of the 31st International Conference on Machine Learning (ICML-14). 2014.


A Bayesian Data Augmentation Approach for Learning Deep Models
Data augmentation is an essential part of the training process applied to deep learning models. The motivation is that a robust training process for deep learning models depends on large annotated datasets, which are expensive to be acquired, stored and processed. Therefore a reasonable alternative is to be able to automatically generate new annotated training samples using a process known as data augmentation. The dominant data augmentation approach in the field assumes that new training samples can be obtained via random geometric or appearance transformations applied to annotated training samples, but this is a strong assumption because it is unclear if this is a reliable generative model for producing new training samples. In this paper, we provide a novel Bayesian formulation to data augmentation, where new annotated training points are treated as missing variables and generated based on the distribution learned from the training set. For learning, we introduce a theoretically sound algorithm -generalised Monte Carlo expectation maximisation, and demonstrate one possible implementation via an extension of the Generative Adversarial Network (GAN). Classification results on MNIST, CIFAR-10 and CIFAR-100 show the better performance of our proposed method compared to the current dominant data augmentation approach mentioned abovethe results also show that our approach produces better classification results than similar GAN models.

The authors propose a Bayesian approach to handle Data Augmentation (DA) for learning deep models. Data augmentation, in this context, addresses the problem of data sparsity by generating more training samples to make the training process of a (heavily parameterized) Classifier Network (CN) robust. The standard Generative Model (GM) for obtaining the augmented training set is typically trained once and then new data are generated based on this GM and used to estimate the CN. The authors suggest to also learn the GM itself jointly with the CN as new augmented training data are produced - which is straightforward with a Bayesian approach. Using three different data sets and two classifiers,  they demonstrate that the Bayesian DA is superior to regular DA, which in turn is superior to not having DA at all (which likely suffers from some overfitting).
The authors claim in the introduction that they provide a novel Bayesian formulation of DA. While this may be true, from a Bayesian point of view their idea is quite simple, as this is what any Bayesian statistician would do. Having said that, I think this is a good paper which is clearly written. If it really is the first attempt to think about this problem in a Bayesian way it is by itself worthy of publication, but I am not sufficiently familiar with the deep learning literature to judge this.
The main problem with the pure DA is that one is heavily relying of the GM to be producing correct data, while the proposed approach would allow the GM to be improved. This makes the authors' approach more challenging as both the CN and the GM need to be trained and, in top of this, both are deep models. Hence, making inference using traditional powerful Bayesian tools seems impossible and the authors propose an algorithm which is in spirit with the EM-algorithm to learn the "latents" (here the artificial data) conditional on model parameters and vice versa. Hence my main concern regards the optimization and its robustness to different choices of tuning parameters. For example, Wilson et al. (2017) show that adaptive step-size choices (such as ADAM) are prone to find poor modes in heavily parameterized models. This model is significantly more parameterized than the standard DA, and, although you generate more data, you also have the problem of training the GM. Can you comment on this?

I think that Lemma 1 should be expressed more carefully, stating the assumptions made.
@article{wilson2017marginal,
  title={The Marginal Value of Adaptive Gradient Methods in Machine Learning},
  author={Wilson, Ashia C and Roelofs, Rebecca and Stern, Mitchell and Srebro, Nathan and Recht, Benjamin},
  journal={arXiv preprint arXiv:1705.08292},
  year={2017}
}


A Regularized Framework for Sparse and Structured Neural Attention
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real values to probabilities, suitable as an attention mechanism. Our framework includes softmax and a slight generalization of the recently-proposed sparsemax as special cases. However, we also show how our framework can incorporate modern structured penalties, resulting in more interpretable attention mechanisms, that focus on entire segments or groups of an input. We derive efficient algorithms to compute the forward and backward passes of our attention mechanisms, enabling their use in a neural network trained with backpropagation. To showcase their potential as a drop-in replacement for existing ones, we evaluate our attention mechanisms on three large-scale tasks: textual entailment, machine translation, and sentence summarization. Our attention mechanisms improve interpretability without sacrificing performance; notably, on textual entailment and summarization, we outperform the standard attention mechanisms based on softmax and sparsemax.

Summary
=======
  This paper presents a framework for implementing different sparse
  attention mechanisms by regularizing the max operator using convex
  functions. As a result, softmax and sparsemax are derived as special
  cases of this framework. Furthermore, two new sparse attention
  mechanisms are introduced that allow the model to learn to pay the
  same attention to contiguous spans. My concerns are regarding to the
  motivation of interpretability, as well as the baseline attention
  models. However, the paper is very well presented and the framework is
  a notable contribution that I believe will be useful for researchers
  working with attention mechanisms.
Strengths
=========
  - The provided framework encompasses softmax and sparsemax as special
    cases.
  - It is a drop-in replacement for softmax, so it can be readily used
    for extending a variety of existing neural networks that employ a
    softmax attention mechanism.
  - There seems to only be a small computational overhead (~25% increase
    in runtime).
  - While the attention weights produced by the sparse attention models
    are slightly more interpretable, using such sparse attention didn't
    hurt performance on RTE, MT and sentence summarization tasks.
Weaknesses
==========
  - Looking at Fig 1 and playing Devil's advocate, I would argue that
    interpretability doesn't really seem to improve that much with the
    variants of sparse attention presented here -- I could just
    threshold the weights of a normal softmax attention mechanism (or
    increase the contrast in Fig 1) and I would get a similar outcome. I
    do see however that with span-based sparse attention things start to
    look more interpretable but I have the impression that the three
    tasks are not the best fit for it. For instance, Q&A on SQuAD might
    have been a more interesting task as it asks for a span in a
    paragraph as the answer for a given question. This might have
    alleviated another weakness of the evaluation, namely that we don't
    see strong improvements when using span-based sparse attention. So
    right now the main takeaway is that we can obtain similar
    performance with slightly improved interpretability -- where I am
    still unsure how practically relevant the latter is.
  - Related to the point above, I do believe it is important to point
    out that interpretability of attention weights is problematic as the
    attentive RTE and MT (and maybe the summarization model too?) are
    not solely relying on the context vector that is obtained from the
    attention mechanism. Hence, any conclusions drawn from looking at
    the attention weights should be taken with a big grain of salt.
  - Moreover, the RTE attention baseline seems weak (though I understand
    that model is inherited from sparsemax). [35] report a test set
    accuracy of 83.5% whereas the baseline attentive RTE model here only
    achieves 81.66%, and the best performance for fusedmax reported here
    is 82.41%. In fact, what I would have liked to see is a comparison
    of a state-of-the-art attentive RTE/MT model with a drop-in
    replacement of fusedmax/oscarmax and an improvement in
    accuracy/BLEU.
Minor Comments
==============
  - I think it would be great if the limitations of this framework could
    be highlighted a bit more. For instance, I am wondering whether your
    framework could also support more global structured penalties such
    as the prior in MT that most of the attention should be along the
    diagonal and my hunch is that this would be difficult. See Cohn,
    Trevor, et al. "Incorporating structural alignment biases into an
    attentional neural translation model." arXiv preprint
    arXiv:1601.01085 (2016).
  - I believe in section 2 it could be made clearer when gradients are
    calculated by a solver and when not.


Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning
Statistical performance bounds for reinforcement learning (RL) algorithms can be critical for high-stakes applications like healthcare. This paper introduces a new framework for theoretically measuring the performance of such algorithms called Uniform-PAC, which is a strengthening of the classical Probably Approximately Correct (PAC) framework. In contrast to the PAC framework, the uniform version may be used to derive high probability regret guarantees and so forms a bridge between the two setups that has been missing in the literature. We demonstrate the benefits of the new framework for finite-state episodic MDPs with a new algorithm that is Uniform-PAC and simultaneously achieves optimal regret and PAC guarantees except for a factor of the horizon.

SUMMARY.
This paper considers the setting of finite horizon episodic learning MDPs, and studies the performance of learning algorithms in terms of PAC and regret guarantees.
The paper introduces a novel algorithm called UBEV, together with a finite time regret and PAC analysis.
DETAILED COMMENTS.
I initially thought this would be a very promising article; In the end I am quite disappointed.
A) I find Section 2 a bit spurious. The whole discussion seems to be motivated only by the fact that you make use of a refined concentration inequality based on a peeling argument instead of the crude union bound that is sometimes used in MDP analysis. Further, the UBEV algorithm is nothing but a regret minimization algorithm for undiscounted MDPs, in the spirit of UCRL, KL-UCRL and so on, with the key notable difference that it applies to the episodic setting with fixed horizon H.
I recommend removing this section 2 entirely.
B) Finite Horizon H:
The assumption of an episodic setting with a fixed, known horizon H should really be discussed.
I find it very misleading that in page 5, several differences are mentioned with other setting but this knowledge of the horizon is not even mentioned.
This vastly simplifying assumption is actually the key element that enables to obtain a simplified analysis. In particular, it naturally gives a known upper bound on the diameter (H) of the MDP, and more importantly it enables the use of Dynamic Programming, as opposed to Value Iteration, for which it is simple to enforce constrained/regularization on the MDP.
In stark contrast, computing an optimistic value while enforcing constrained on the output value in the context of an infinite horizon is a key challenge;
See algorithms such as UCRL, KL-UCRL or REGAL that consider infinite horizon (they do not resort to DP due to the infinite horizon), plus discussions there-in regarding the difficulty of enforcing constrains on the MDP in practice.
C) Scaling of the regret:
The leading term of the regret in Theorem 4 scales with sqrt{S} but it also scales with H^2 with the horizon. In this context where H upper bounds the diameter D of the MDP, this dependency seems to be loose compared to the regret of other algorithms that scale with D, not D^2. Thus it seems in the analysis you just improve on one factor but loose on another one. Once again, the fact this "detail" is not even discussed is misleading.
D) Concentration inequalities:
The main building block of this paper, apart from the greatly simplifying assumption of having a finite horizon H, is to resort to an improved concentration inequality for the empirical transition distribution, in order to properly handle the random stopping time (number of observations of a state-action pair).
Indeed the naive way to obtain concentration inequalities for the empirical mean reward or transition probability at time t, with N_t(s,a) observations, is to apply a crude union bound on all possible values of N_t(s,a), that is t values, thus creating a factor t in the log terms (log(t\delta)).
However, improved techniques based on a peeling argument  (where the values of N_t(s,a) are localized) leads to better scaling essentially moving from a t term to a log(t) factor, up to loosing some constant factors. Such argument has been used in several analyses of bandit: see e.g. the analysis of KL-UCB by [Cappe et al.], or tools by Aurelien Garivier relating this indeed to the LIL. One can also see a similar type of argument in the PhD of Sebastien Bubeck, or in even earlier papers by Tze Leung Lai. More generally, the peeling technique is largely used is mathematical statistics, and is not novel.
Now, it is an easy exercise to show that the same peeling argument leads to concentration inequalities valid not for a single t, but uniformly over all t (actually for possibly unbounded random stopping time, and thus in particular for the random number of pulls) as well; thus I do not see much novelty either.
Let us go a little more in depth in the statements (e.g. G.1) regarding tightness of the bounds that is provided here, one may first question the use of a peeling with a geometric factor 2, that is arguably the most naive and sub-optimal one, and could be improved (see e.g. analysis of KL-UCB in [Cappe et al.]). Second, at least for sub-Gaussian random variables, one may instead resort to the Laplace method: this method leads to a \sqrt{t} term in the logarithm instead of a \log(t) and thus might be considered sub-optimal. However, the constant factors that we are used when applying the peeling argument, even after optimization of all terms (which is not done here), is still larger than the one we loose with the Laplace method. As a result, the bound resulting from the peeling argument term becomes only tighter than the one resulting from the Laplace method asymptotically for huge values of t (this actually makes sense as the regime of the LIL is also a quite asymptotic regime).
Thus, I feel that the discussion regarding the llnp(t) term is incomplete and misses an important block of the literature, once again.
E) The only possibly interesting part of the paper is Lemma D.1; but as mentioned, being able to enforce the constraints on the MDP is made easy by the fact that a finite horizon is considered.
F) Minor point: the appendix seems too long for a NIPS paper.
DECISION.
The paper does not enough put in perspective its results with the existing literature, especially regarding the horizon H and the concentration inequalities.
Likewise, the scaling of the regret bounds is not discussed against other results.
These missing points tend to give the feeling that the authors oversell their contribution, and hide the more debatable aspects of their results.
This is also extremely misleading for the less expert readers that may not be able to see the loose aspects of this work.
Perhaps it is just that the paper was just written in a rush, but this is anyway bad practice.
Points B,C,D should be improved/fixed.
Reject.
FINAL DECISION
I read the authors' rebuttal.
Their answer to point C) (scaling with H) should be included in the paper, as this is an important clarification of their result.
Even though the paper is restricted to the simpler finite horizon setting, I agree that this setting is also relevant and deserves attention.
I still believe that the selling point of the paper should not be section 2 and that much more credit should be given to previous works: indeed each separate idea presented in the paper existed before, so that the paper does not introduce a novel idea; what is novel however, is the combination of these.
We should thus judge how this combination is executed, and I consider this is well done.
Overall, given that the claims and proofs look correct (even though I haven't checked everything in detail), and that the paper is globally well executed, I accept to raise my score in order to meet a consensus.


Generalizing GANs: A Turing Perspective
Recently, a new class of machine learning algorithms has emerged, where models and discriminators are generated in a competitive setting. The most prominent example is Generative Adversarial Networks (GANs). In this paper we examine how these algorithms relate to the Turing test, and derive what-from a Turing perspective-can be considered their defining features. Based on these features, we outline directions for generalizing GANs-resulting in the family of algorithms referred to as Turing Learning. One such direction is to allow the discriminators to interact with the processes from which the data samples are obtained, making them "interrogators", as in the Turing test. We validate this idea using two case studies. In the first case study, a computer infers the behavior of an agent while controlling its environment. In the second case study, a robot infers its own sensor configuration while controlling its movements. The results confirm that by allowing discriminators to interrogate, the accuracy of models is improved.

This paper proposes a generalization of GAN, which the authors refer to as Turing Learning. The idea is to let the discriminator to interact with the generator, and thus work as an active interrogator that can influence the sampling behavior of the generator. To show that this strategy is superior to existing GAN which has the discriminator to function passive responder, the authors perform two case studies of the proposed (high-level) model (interactive), comparing against models with passive discriminators. The results show that the interactive model largely outperforms the passive models, with much less variances across multiple runs.
Pros
1) The paper is written well, with clear motivation. The idea of having the discriminator to work as active interrogator to influence the generator is novel and makes perfect sense.
2) The results show that the proposed addition of discriminator-generator interaction actually helps the model work more accurately.
Cons
3) The proposed learning strategy is too general while the implemented models are too specific to each problem. I was hoping to see a specific GAN learning framework that implements the idea, but no concrete model is proposed and the interaction between the discriminator and the generator is implemented differently from model to model, in the two case studies. This severely limits the application of the model to new problems.
4) The two case studies consider toy problems, and it is not clear how this model will work in real-world problems. This gives me the impression that the work is still preliminary.
Summary
In sum, the paper presents a novel and  interesting idea that can generalize and potentially improve the existing generative adversarial learning. However, the idea is not implemented into a concrete model that can generalize to general learning cases, and is only validated with two proof-of-concept case studies on toy problems. This limits the applicability of the model to new problems. Thus, despite its novelty and the potential impact, I think this is a preliminary work at its current status, and do not strongly support for its acceptance.


Scalable Log Determinants for Gaussian Process Kernel Learning
For applications as varied as Bayesian neural networks, determinantal point processes, elliptical graphical models, and kernel learning for Gaussian processes (GPs), one must compute a log determinant of an n × n positive definite matrix, and its derivatives -leading to prohibitive O(n 3 ) computations. We propose novel O(n) approaches to estimating these quantities from only fast matrix vector multiplications (MVMs). These stochastic approximations are based on Chebyshev, Lanczos, and surrogate models, and converge quickly even for kernel matrices that have challenging spectra. We leverage these approximations to develop a scalable Gaussian process approach to kernel learning. We find that Lanczos is generally superior to Chebyshev for kernel learning, and that a surrogate approach can be highly efficient and accurate with popular kernels.

Summary of the paper:
	This paper describes two techniques that can be used to estimate (stochastically) the log determinant of a positive definite matrix and its gradients. The aim is to speed up the inference process several probabilistic models that involve such computations. The example given is a Gaussian Process in which the marginal likelihood has to be maximized to find good hyper-parameters. The methods proposed are based on using the trace of the logarithm of a matrix to approximate the log determinant. Two techniques are proposed. The first one based on Chevyshev polynomials and the second one based on the Lanczos algorithm. The first one seems to be restricted to the case in which the eigenvalues lie in the interval [-1,1]. The proposed method are compared with sparse GP approximations based on the FITC approximation and with related methods based on scaled eigenvalues on several experiments involving GPs.
Detailed comments:
Clarity:
The paper is very well written and all details seem to be clear with a few exceptions. For example, the reader may not be familiar with the logarithm of a matrix and similar advanced algebra operations. There are few typos in the paper also.  In the Eq. above (2) it is not clear what beta_m or e_m are. They have not being defined. It is also known that the Lanczos algorithm is unstable and extra computations have to be done. The authors may comment on these too.
Quality:
I think the quality of the paper is high. In particular, the authors analyze the method described in several experiments, comparing with other important and related methods. These experiments seem to indicate that the proposed method gives competitive results at a small computational cost.
Originality:
The paper a strong related work section and the methods described seem to be original.
Significance:
My main concern with this paper is its significance. While it is true that the proposed method allow for a faster evaluation of the log determinant of the covariance matrix and its gradients. One still has to compute such a matrix with squared cost in the number of samples (or squared cost in the number of inducing points, if sparse approximations are used). This limits a bit the practical applicability of the proposed method. Furthermore, it seems that the experiments carried out by the authors involve a single repetition. Thus, there are no error bars in the results and one can not say whether or not the results are statistical significant. The differences with respect to the scaled eigenvalues problems (ref. [32]) in the paper seem very small.


Rotting Bandits
The Multi-Armed Bandits (MAB) framework highlights the trade-off between acquiring new knowledge (Exploration) and leveraging available knowledge (Exploitation). In the classical MAB problem, a decision maker must choose an arm at each time step, upon which she receives a reward. The decision maker's objective is to maximize her cumulative expected reward over the time horizon. The MAB problem has been studied extensively, specifically under the assumption of the arms' rewards distributions being stationary, or quasi-stationary, over time. We consider a variant of the MAB framework, which we termed Rotting Bandits, where each arm's expected reward decays as a function of the number of times it has been pulled. We are motivated by many real-world scenarios such as online advertising, content recommendation, crowdsourcing, and more. We present algorithms, accompanied by simulations, and derive theoretical guarantees.

The paper introduces a novel variant of the multi-armed problem, motivated by practice.

      The approach is technically sound and well grounded. There are a few typos and minor concerns (listed
      below), despite which I believe this paper can be accepted.

      Decaying bandits - e.g., Komiyama and Qin 2014, Heidari, Kearns and Roth 2016 - are mentioned, but Deteriorating bandits - e.g., Gittins 1979, Mandelbaum 1987, Kaspi and Mandelbaum 1998, etc. - are not

      Can't you use algorithms from non-stationary MABs, which you say are "most related to our problem", for benchmarking? It's quite unfair to use only UCB variants.

      Line 15: "expanse" -> "expense"
      Line 19: you have skipped a number of key formulations and break-throughs between 1933 and 1985
      Line 31-32: probably a word is missing in the sentence
      Line 63: "not including" what?
      Line 68: "equal objective" -> "equivalent objective"
      Line 68: this is an unusual definition of regret, in fact this expression is usually called "suboptimality", as you only compare to the optimal policy which is based on past allocations and observations. Regret is usually defined as distance from the objective achievable if one knew the unknown parameters.
      Line 105: in the non-vanishing case, do you allow \mu_i^c = 0 or negative?
      Line 106-118: some accompanying text describing the intuition behind these expressions would be helpful
      Line 119,144: headings AV and ANV would be more reasonable than names of the heuristics
      Line 160: "w.h.p" -> "w.h.p."
      Table 1: "CTO" is ambiguous, please use the full algorithm acronyms
      Figure 1: it would be desirable to explain the apparent non-monotonicity
      Line 256: It would be helpful for the reader if this sentence appeared in the Introduction. Btw, the term "rested bandits" is not very common; instead "classic bandits" is a much more popular term


ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games
In this paper, we propose ELF, an Extensive, Lightweight and Flexible platform for fundamental reinforcement learning research. Using ELF, we implement a highly customizable real-time strategy (RTS) engine with three game environments (Mini-RTS, Capture the Flag and Tower Defense). Mini-RTS, as a miniature version of StarCraft, captures key game dynamics and runs at 40K frameper-second (FPS) per core on a laptop. When coupled with modern reinforcement learning methods, the system can train a full-game bot against built-in AIs endto-end in one day with 6 CPUs and 1 GPU. In addition, our platform is flexible in terms of environment-agent communication topologies, choices of RL methods, changes in game parameters, and can host existing C/C++-based game environments like ALE [4]. Using ELF, we thoroughly explore training parameters and show that a network with Leaky ReLU [17] and Batch Normalization [11] coupled with long-horizon training and progressive curriculum beats the rule-based built-in AI more than 70% of the time in the full game of Mini-RTS. Strong performance is also achieved on the other two games. In game replays, we show our agents learn interesting strategies. ELF, along with its RL platform, is open sourced at https://github.com/facebookresearch/ELF.

The main proposal of the paper is a real-time strategy simulator specifically designed for reinforcement learning purposes. The paper presents with several details the architecture of the simulator, along with how gaming is done on it and some experimentations with the software with some RL techniques implemented in the software.
Although I think there are good values in making with software for research, I don’t think that NIPS is the right forum for presenting technical papers on them. Machine Learning Open Source Software (MLOSS) track from JMLR or relevant workshop are much relevant for that. And in the current case, a publication in the IEEE Computational Intelligence and Games (IEEE-CIG) conference might be a much better fit.
That being said, I am also quite unsure of the high relevance of real-time strategies (RTS) games for demonstrating reinforcement learning (RL). The use of RTS for benchmarking AI is not new, it has been done several time in the past. I am not convinced it will the next Atari or Go. It lacks the simplicity and elegance of Go, while not being particularly useful for practical applications. The proposed implementation is sure designed to allow fast during simulation, making learning can be done much faster (assuming that the bottleneck was the simulator, not the learning algorithms). But such a feat does not justify a NIPS paper per se, there contribution is implementation-wise, not on learning by itself.
*** Update following rebuttal phase and comments from area chair ***
It seems that the paper still fit in the scope of the conference. I updated my evaluation accordingly, being less harsh in my scoring. However, I maintain my opinion on this paper, I still think RTS for AI is not a new proposal, and I really don't feel it will be the next Atari or Go. The work proposed appears correct in term of implementation, but in term of general usefulness, I still remain to be convinced.


Kernel functions based on triplet comparisons
Given only information in the form of similarity triplets "Object A is more similar to object B than to object C" about a data set, we propose two ways of defining a kernel function on the data set. While previous approaches construct a lowdimensional Euclidean embedding of the data set that reflects the given similarity triplets, we aim at defining kernel functions that correspond to high-dimensional embeddings. These kernel functions can subsequently be used to apply any kernel method to the data set.

The authors propose two kernel functions based on comparing triplets of data examples. The first kernel function is based on ranking of all the other objects with respect to the objects being compared. The second function is based on comparing whether the compared objects rank similarly with respect to the other objects. The idea is interesting and it is potentially impactful to create such “ordinal” kernels, where the data are not embedded in metric spaces.
Comments:
1. The authors should provide their main claims somewhere and demonstrate it well in the paper. Why should anybody use a triplet comparison kernel? From what I understand, with triplet comparison kernel, less number of triplets are sufficient to capture the same information that a full pairwise similarity matrix can capture (sec. 2.2). It is also faster. Is that correct?
2. How is the first kernel (k_1) and the one proposed in (Jiao & Vert 2015) different?
3. In Figure 3, can you quantify how well the proposed kernel performs with respect to computing all pairwise similarities? The synthetic datasets are not hard, and it may also be useful to demonstrate some harder case?
4. The kernels seem to have time advantages compared to other embedding approaches such as GNMDS (from the experiments in Sec. 4.2). However in terms of clustering performance they are outperformed by embedding approaches (the authors claim that the performance is close for all algorithms). How many landmark points were chosen for computing the kernels? Can you comment on the size of landmark set vs. running times?
5. Also, in terms of running times, can you comment whether the reason for the slow performance of the embedding algorithms is the inherent computational complexity? Could it be an implementation issue?
6. The authors should attempt to provide results that are legible and easy to read. If there is lack of space, supplementary material can be provided with the rest of results.
Update
-------
The authors have answered several of my comments satisfactorily.


Pose Guided Person Image Generation
This paper proposes the novel Pose Guided Person Generation Network (PG 2 ) that allows to synthesize person images in arbitrary poses, based on an image of that person and a novel pose. Our generation framework PG 2 utilizes the pose information explicitly and consists of two key stages: pose integration and image refinement. In the first stage the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. The second stage then refines the initial and blurry result by training a U-Net-like generator in an adversarial way. Extensive experimental results on both 128×64 re-identification images and 256×256 fashion photos show that our model generates high-quality person images with convincing details.

The paper proposes a human image generator conditioned on appearance and human pose. The proposed generation is based on adversarial training architecture where two-step generative networks that produces high resolution image to feed into a discriminator. In the generator part, the first generator produce a coarse image using a U-shape network given appearance and pose map, then the second generator takes the coarse input with the original appearance to predict residual to refine the coarse image. The paper utilizes the DeepFashion dataset for evaluation.
The paper proposes a few important ideas.
* Task novelty: introducing the idea of conditioning on appearance and pose map for human image generation
* Techniques: stacked architecture that predicts difference map rather than direct upsampling, and loss design
The paper can improve in terms of the following points to stand out.
* Still needs quality improvement
* Significance: the paper could be seen one of yet-another GAN architecture
* Problem domain: good vision/graphics application, but difficult to generalize to other learning problems
The paper is well organized to convey the key aspects of the proposed architecture. Conditioned on appearance and pose information, the proposed generator stacks two networks to adopt a coarse-to-fine strategy. This paper effectively utilize the generation strategy in the dressing problem. The proposed approach looks appropriate to the concerned problem scenario. The difference map generation also looks a small but nice technique in generating higher resolution images.
Probably the major complaints to the paper is that the generated results contain visible artifacts and still requires a lot of improvement for application perspective. For example, patterns in ID346 of Fig 4 results in black dots in the final result. Even though the second generator mitigates the blurry image from the first generator, it seems the model is still insufficient to recover high-frequency components in the target appearance.
Another possible but not severe concern is that some might say the proposed approach is an application of conditional GANs. Conditioning or stacking of generators for adversarial training have been proposed in the past; e.g., below, though they are arXiv papers. The paper includes application-specific challenges, but this might not appeal to large number of audiences.
* Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, Dimitris Metaxas, "StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks", arXiv:1612.03242.
* Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, Serge Belongie, Stacked Generative Adversarial Networks, arXiv:1612.04357.
In overall, the paper successfully proposed a solution to the pose-conditioned image problem, and properly conducts evaluation. The proposed approach sufficiently presents technical novelty. The resulting images still needs quality improvement, but the proposed model at least generate something visually consistent images. My initial rating is accept.