Hierarchical Attentive Recurrent Tracking
Class-agnostic object tracking is particularly difficult in cluttered environments as target specific discriminative models cannot be learned a priori. Inspired by how the human visual cortex employs spatial attention and separate "where" and "what" processing pathways to actively suppress irrelevant visual features, this work develops a hierarchical attentive recurrent model for single object tracking in videos. The first layer of attention discards the majority of background by selecting a region containing the object of interest, while the subsequent layers tune in on visual features particular to the tracked object. This framework is fully differentiable and can be trained in a purely data driven fashion by gradient methods. To improve training convergence, we augment the loss function with terms for auxiliary tasks relevant for tracking. Evaluation of the proposed model is performed on two datasets: pedestrian tracking on the KTH activity recognition dataset and the more difficult KITTI object tracking dataset.

Summary: The authors present a framework for class-agnostic object tracking. In each time step, the presented model extracts a glimpse from the current frame using the DRAW-style attention mechanism also employed by Kahoue et al. Low level features are then extracted from this glimpse and fed into a two-stream architecture, which is inspired by the two-stream hypothesis. The dorsal stream constructs convolutional filters, which are used to extract a spatial Bernoulli distribution, each element expressing the probability of the object occupying the corresponding locations in the feature map. This location map is then combined with the appearance-based feature map computed by the ventral stream, removing the distracting features, which are not considered to be part of the object.
An LSTM is used for tracking the state of the object, with an MLP transforming the output of the LSTM into parameters for both attention mechanisms and an offset to the bounding box parameters in the previous time step.
The results on KTH and the more challenging KITTI dataset are significantly closer to the state of the art than previous attention based approaches to tracking.
Qualitative Assessment:
The paper reads well and is easy to follow. The description of the model is detailed enough for replication and experiments are performed providing evidence for some of the premises stated in the description of the model.
The results are impressive compared to previous work on attention based tracking. Since, you used a standard tracking benchmark, I think more performance numbers from the tracking community could have been included to show how close the presented method is to the state of the art. (example: Re3: Real-Time Recurrent Regression Networks for Object Tracking
D Gordon, A Farhadi, D Fox, arXiv preprint arXiv:1705.06368)


Attentional Pooling for Action Recognition
We introduce a simple yet surprisingly powerful model to incorporate attention in action recognition and human object interaction tasks. Our proposed attention module can be trained with or without extra supervision, and gives a sizable boost in accuracy while keeping the network size and computational cost nearly the same. It leads to significant improvements over state of the art base architecture on three standard action recognition benchmarks across still images and videos, and establishes new state of the art on MPII dataset with 12.5% relative improvement. We also perform an extensive analysis of our attention module both empirically and analytically. In terms of the latter, we introduce a novel derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods (typically used for fine-grained classification). From this perspective, our attention formulation suggests a novel characterization of action recognition as a fine-grained recognition problem.

The authors propose to use a low rank approximation of second order pooling features for action recognition. They show results on three compelling vision benchmarks for action recognition and show improvements over the baseline.
Pros:
(+) The proposed method is simple and can be applied on top of any network architecture
(+) Clear improvement over baseline
Cons:
(-) Unclear novelty over well known second order pooling approaches
(-) Fair comparison with other attention based models is missing
It is unclear to me how the proposed approach is novel compared to other low rank second order pooling methods, typically used for fine grained recognition (e.g. Lin et al, ICCV2015), semantic segmentation etc. While the authors do a decent job walking us through the linear algebra and interpreting the variables used, the final approach is merely a classifier on second order features.
In addition, the authors make design choices that are not clearly justified. For multi class recognition, the authors do not justify the choice for a class agnoistic b but for class-specific a. These experiments should be added in order to prove the effectiveness of the proposed design.
The authors claim that they get rid of the box detection step present at other approaches such as R*CNN or Mallya & Lazebnik. However, they do not discuss how their approach would generalize to instance-specific action recognition. An image can contain many people who perform different actions. In this case, the action labels are not assigned at the frame level, but at an instance level. R*CNN was designed for instance-level recognition, thus the necessity of the person box detection step. The authors should discuss how the proposed approach can handle this more generic setup of instance-level labels? This is a limitation of the current approach, rather than a benefit as implied in the Related Work section.
The authors fail to provide a fair comparison with competing approaches. The authors use a ResNet101 or an Inception-V2 architecture for their base network. They show improvement over their own non-attentive baseline, using the same architecture. However, the competing approaches, such as R*CNN or Mallya & Lazebnik all use a VGG16 network. This makes comparison with these approaches and the current approach unfair and inconclusive on MPII and HICO dataset. On HMDB, the improvement over the TSN BN-inception method is small while comparisons with other methods is unclear due to the varying underlying base architectures.


On the Optimization Landscape of Tensor Decompositions
Non-convex optimization with local search heuristics has been widely used in machine learning, achieving many state-of-art results. It becomes increasingly important to understand why they can work for these NP-hard problems on typical data. The landscape of many objective functions in learning has been conjectured to have the geometric property that "all local optima are (approximately) global optima", and thus they can be solved efficiently by local search algorithms. However, establishing such property can be very difficult.
In this paper, we analyze the optimization landscape of the random over-complete tensor decomposition problem, which has many applications in unsupervised leaning, especially in learning latent variable models. In practice, it can be efficiently solved by gradient ascent on a non-convex objective. We show that for any small constant ε > 0, among the set of points with function values (1 + ε)-factor larger than the expectation of the function, all the local maxima are approximate global maxima. Previously, the best-known result only characterizes the geometry in small neighborhoods around the true components. Our result implies that even with an initialization that is barely better than the random guess, the gradient ascent algorithm is guaranteed to solve this problem. Our main technique uses Kac-Rice formula and random matrix theory. To our best knowledge, this is the first time when Kac-Rice formula is successfully applied to counting the number of local optima of a highly-structured random polynomial with dependent coefficients.

### Summary
The paper analyzes the landscape of tensor decomposition problem. Specifically, it studies random over-complete tensors. The associated objective function is nonconvex, yet in practice simple methods based on gradient ascent are observed to solve this problem. This paper proves why we should expect such outcome by showing that there is almost no local maxima other than the global maxima of the problem when the optimization is initialized by any solution that is slightly better than random guess. Importantly, it is shown that these initial points do not have to be close to the true components of the tensor. This is an interesting result and well written paper. The analysis involves two steps: local (points close to true components) and global (point far from true components). The number of local maxima in each case is analyzed and shown to be exactly 2n for the former and almost nonexistent for the latter.
### Questions
1. Authors mention that in practice gradient ascent type algorithms succeed. Do these algorithms use any specific initialization that meets your criterion? More precisely, do they use a scheme to produce initialization which is slightly better than chance and thus the objective value is slightly greater than 3*n?
IF YES: Please elaborate briefly how this is done or provide a reference.
IF NO: Any idea how such initial points could be generated easily? Could it be a very hard problem on its own?
2. The analysis is performed on random tensors, where each true component is assumed to be drawn iid from Gaussian distribution N (0, I). It seems to be a strong assumption and not true for real world problems. While I am not disputing the importance of the result provided in this paper, I was wondering if the authors had thoughts about analysis scenarios that could potentially explain the success of local schemes in a broader setup? For example, performing a worst-case analysis (i.e. choosing the worst tensors) and then showing that these worst tensors form a negligible fraction of the space of tensors? This would go beyond the Gaussian setting and may be more pertinent to explaining why local search succeeds on real world problems.
### Minor Comments
1. Lemma 2.2. Eq. 2.1, the first E is missing a | at the begining right after [.
2. Lines 80, 84, 123 local minima should be local maxima?
3. You refer to hard optimization as nonconvex, but since your task in 1.1. is defined as a maximization, concave functions are actually ideal for that. To avoid confusion around minimization/maximization/nonconvexity, you might want to replace 1.1 from Max f(x) with Min -f(x) so that everything remains a minimization, and thus nonconvexity implies hardness. In that case, you can also ignore my comment 2.


Ensemble Sampling
Thompson sampling has emerged as an effective heuristic for a broad range of online decision problems. In its basic form, the algorithm requires computing and sampling from a posterior distribution over models, which is tractable only for simple special cases. This paper develops ensemble sampling, which aims to approximate Thompson sampling while maintaining tractability even in the face of complex models such as neural networks. Ensemble sampling dramatically expands on the range of applications for which Thompson sampling is viable. We establish a theoretical basis that supports the approach and present computational results that offer further insight.

This paper proposes ensemble sampling which essentially approximates Thompson Sampling for complex models in a tractable way. This will be useful in a variety of applications. There does not seem to be an obvious mistake in the proofs. However, there are a number of limitations of the proposed method. See below for detailed comments:
1. What is the underlying noise model for the rewards? Please clarify this.
2. The weights of the neural network are assumed to be Gaussian distributed. Why is this a valid assumption? Please explain.
3. Please explain under what assumptions can the proposed method be applied? Can it applied for other model classes such as generalized linear models, and SVMs?
4. A neural network offers more flexibility. But is this truly necessary for applications where the time horizon is not too large, and the number of samples are limited?
5. Missing references in lines 85 - 92. Please add citations wherever necessary.
6. In line 3 of Algorithm 1, the model is picked uniformly at random. Can the results be improved by using a more intelligent sampling scheme?
7. Using a sample from a randomly perturbed prior and measurements seems similar to the Perturb and MAP framework in [1]. Please cite this work and describe how your framework is different.
8. To update the models, it is necessary to store the entire history of observations. This is inefficient from both a memory and computational point of view.
9. In line 120, "some number of stochastic gradient descent iterations". Please clarify this. How many are necessary? Do you model the error which arises from not solving the minimization problem exactly?
10. In Theorem 3, the approximation error increases with T. The approximate regret is then in some sense linear in T. Please justify this. Also, what is the dimension dependence of the approximation error? Also, please give some intuition on the linear dependence of on the gap \delta.
11. In section 5, the experiments should show the benefit of using a more complex model class. This is missing. Please justify this.
12. A sampling based technique should be used as a baseline. This is also missing. Please justify.
13. In line 249, "we used a learning rate of.. ". How did you tune these hyper-parameters? How robust are your results to these choices?
[1]. Gaussian Sampling by Local Perturbations,


Kernel Feature Selection via Conditional Covariance Minimization
We propose a method for feature selection that employs kernel-based measures of independence to find a subset of covariates that is maximally predictive of the response. Building on past work in kernel dimension reduction, we show how to perform feature selection via a constrained optimization problem involving the trace of the conditional covariance operator. We prove various consistency results for this procedure, and also demonstrate that our method compares favorably with other state-of-the-art algorithms on a variety of synthetic and real data sets.

In this paper, authors propose a new nonlinear feature selection based on kernels. More specifically, the conditional covariance operator has been employed to measure the conditional independence between Y and X given the subset of X. Then, the feature selection can be done by searching a set of features that minimizing the conditional independence. This optimization problem results in minimizing over matrix inverse and it is hard to optimize it. Thus, a novel approach to deal with the matrix inverse problem is also proposed. Finally, the consistency of the feature selection algorithm is proved. Through experiments, the authors showed that the proposed algorithm compares favorably with existing algorithms.
The paper is clearly written and the proposed algorithm is technically sound. Overall, I like the algorithm.
Detailed comments:
1. In the proposed algorithm, some information about tuning parameters are missing. So, please include those information in the paper. For example, lambda_2 in Eq. (20). Also, how to initialize the original w?
2. It would be great to compare with other kernel based feature selection algorithms such as HSIC-LASSO and KNIFE. Also, related to the first comments, is it possible to further improve the proposed algorithm by properly initializing the w? For instance, HSIC-LASSO is a convex approach and can have reasonable features.
3. The results of TOX-171 is pretty interesting. It is nice to further investigate the reason why the proposed approach outperforms rest of it.
4. It is nice to have comparison of redundancy rate. In feature selection community tends to use this measure for evaluation.
Zhao, Z., Wang, L., and Li, H. (2010). Efficient spectral feature se lection with minimum redundancy. In AAAI Conference on Artificial Intelligence (AAAI) , pages 673–678.


Multi-Objective Non-parametric Sequential Prediction
Online-learning research has mainly been focusing on minimizing one objective function. In many real-world applications, however, several objective functions have to be considered simultaneously. Recently, an algorithm for dealing with several objective functions in the i.i.d. case has been presented. In this paper, we extend the multi-objective framework to the case of stationary and ergodic processes, thus allowing dependencies among observations. We first identify an asymptomatic lower bound for any prediction strategy and then present an algorithm whose predictions achieve the optimal solution while fulfilling any continuous and convex constraining criterion.

This paper presents an asymptotic analysis for nonparametric sequential prediction when there are multiple objectives. While the paper has some relevance to a certain subset of the machine learning community, I have some concerns about the paper's relevance to NIPS, and I also am unclear on some of the basic setup of the paper. I discuss these issues in the remainder of the review. Overall, I think the results in the paper are technically strong and, with the right motivation for $\gamma$-feasibility (see below) I would be weakly positive on the paper.
Relevance:
While I can see that the contents of the paper have some relevance to machine learning, I feel that this sort of paper would fit much better in a conference like COLT or ALT, given the technical level and given how much this paper can benefit from the additional space offered by those venues. I am not convinced that the problem addressed by this paper is interesting to the broader machine learning community and even many theoretical researchers within online learning. I can understand that an asymptotic analysis may be interesting as it allows one to obtain a first approach to a problem, but if the asymptotic nature of the analysis is fundamental, as it is in this case (the authors mention that positive non-asymptotic results simply are not possible without additional assumptions), then is there hope to finally achieve learning algorithms that can actually be used? I do not see any such hope from this paper, especially in light of the construction of experts that should be fed into the MHA algorithm.
Motivation for the objective:
This paper lacks sufficient motivation for the formal objective; while the high-level objective is clear, namely, to obtain a strategy that minimizes the average $u$-loss subject to the average $c$-loss being at most $\gamma$, the formal rendering of this objective is not sufficiently motivated. I understand why one would want to use strategies that are $\gamma$-bounded (as defined in Definition 1). However, the authors then further restrict to processes that are $\gamma$-feasible with no motivation for why we should restrict to these processes. How do we know that this restriction is the right one, and not too great of a restriction? If we restrict less, perhaps we would arrive at a different $\mathcal{V}^*$ that is larger than the one arising from $\gamma$-feasible processes. In particular, with additional space, I feel that the authors should be able to properly explain (or at least give some explanation, as none whatsoever is given in the current paper) why we should care about the conditional expectation of $u(y, X_0)$ (conditional on the infinite past). It is paramount to motivate the $\gamma$-feasibility restriction so that $\mathcal{V}^*$ can be understood as a quantity of fundamental interest.
Other comments on the paper:
While the MHA algorithm seems interesting, without an explanation of what experts are used in the main paper, it is difficult to get a sense of what the algorithm is really doing. The authors should devote some space in the main text explaining the construction of the experts. I wonder if the authors could have provided proof sketches in the main paper for the main results, Theorems 1 and 3, and bought themselves about 3 pages (each of those proofs is about 1.5-2 pages), as well as an exposition that would be clearer to the audience.
MINOR COMMENTS
I looked through part of the proof of Theorem 1 and it seems fine, but I am not an expert in this area so I am trusting the authors.
In the math display between lines 68 and 69, I think the a.s. should not be there. If it should be there, what does it mean? All the randomness has been removed by the outer expectation.
Line 131: ``an histogram'' --> ``a histogram''
This paper is at times sloppily written. Examples:
    Line 131, you write $H_{h,k}$ and then later write $k, l = 1, 2, \ldots$, and this swapping of $h$ and $l$ happens throughout the paper.
    Math display between lines 107 and 108: The minus sign in front of the last summation on the first line should be a plus sign.
  ``Optimallity'' -> ``Optimality''
UPDATE AFTER REBUTTAL
I am satisfied with the authors' response regarding the formal framing of the problem and the conditional expectation, provided that they provide details (similar to their rebuttal) in the paper.


Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition
Spectral decomposition of the Koopman operator is attracting attention as a tool for the analysis of nonlinear dynamical systems. Dynamic mode decomposition is a popular numerical algorithm for Koopman spectral analysis; however, we often need to prepare nonlinear observables manually according to the underlying dynamics, which is not always possible since we may not have any a priori knowledge about them. In this paper, we propose a fully data-driven method for Koopman spectral analysis based on the principle of learning Koopman invariant subspaces from observed data. To this end, we propose minimization of the residual sum of squares of linear least-squares regression to estimate a set of functions that transforms data into a form in which the linear regression fits well. We introduce an implementation with neural networks and evaluate performance empirically using nonlinear dynamical systems and applications.

This paper presents a method that takes advantage of MLPs to learning non-linear functions that support Koopman analysis of time-series data. The framework is based on Koopman operator theory, an observation that complex, nonlinear dynamics may be embedded with nonlinear functions g where a spectral properties of a linear operator K can be informative about both the dynamics of the system and predict future data. The paper proposes a neural network architecture and a set of loss functions suitable for learning this embedding in a Koopman invariant subspace, directly from data. The method is demonstrate both on numerical examples and on a few applications (Lorenz, Rossler, and unstable phenomena data).
The paper is exceptionally clearly written, and the use of a neural network for finding Koopman invariant subspaces, a challenging and widely applicable task, is well motivated. The figures nicely illustrate the results, and I find the examples convincing. There's a few questions I was curious about. Firstly, how well does LKIS tolerate larger magnitudes of noise? Does it converge on the right Koopman eigenfunctions, or is there a point past which it fails? Second, how do the g's found by LKIS compare with those found by EDMD and kernel DMD?
I read through the author’s rebuttal and my colleagues’ review, and decided to not change my assessment of the paper.


Beyond Parity: Fairness Objectives for Collaborative Filtering
We study fairness in collaborative-filtering recommender systems, which are sensitive to discrimination that exists in historical data. Biased data can lead collaborative-filtering methods to make unfair predictions for users from minority groups. We identify the insufficiency of existing fairness metrics and propose four new metrics that address different forms of unfairness. These fairness metrics can be optimized by adding fairness terms to the learning objective. Experiments on synthetic and real data show that our new metrics can better measure fairness than the baseline, and that the fairness objectives effectively help reduce unfairness.

In this paper the authors explore different metrics for measuring fairness in recommender systems.  In particular, they offer 4 different metrics that measure if a recommender system over-estimates or under-estimates how much a group of users will like a particular genre.  Further, they show that by regularizing the by the discrepancy across groups does not hurt the model accuracy while improving fairness on the metric used for the regularization (as well as undiscussed slight effects on the other fairness metrics).
In contrast to the work of Hardt et al, this paper focuses on a regression task (rating prediction) and explores the difference between absolute and relative errors.  This is an interesting point in recommender systems that could easily be overlooked by other metrics.  Further, the clear demonstration of bias on MovieLens data and the ability to improve it through regularization is a valuable contribution.  Additionally, the simplicity of the approach and clarity of the writing should be valued.
However, the metrics are also limited in my opinion.  In particular, all of the metrics focus on average ratings over different groups.  It is easy to imagine a model that has none of the biases discussed in this paper, but performs terribly for one group and great for the other.  A more direct extrapolation of equality of opportunity would seem to focus on model accuracy per-rating rather than aggregates.  I do not think this ruins the work here, as I mentioned above that the directionality of the errors is an important observation, but I do think it is a limitation that will need to be developed upon in the future.
In this vain, a significant amount of recommender systems research is not cited or compared against.  Data sparsity, missing not at random, learning across different subgroups, using side information, etc. have all been explored in the recommender systems literature.  At the least, comparing against other collaborative filtering models that attempt to improve the model accuracy by side information would provide better context (as well as reporting accuracy of the model by both gender and genre).  A few pieces of related work are included below.
"Collaborative Filtering and the Missing at Random Assumption" Marlin et al
"Beyond Globally Optimal: Focused Learning for Improved Recommendations" Beutel al
"It takes two to tango: An exploration of domain pairs for cross-domain collaborative filtering" Sahebi et al


Online Learning with a Hint
We study a variant of online linear optimization where the player receives a hint about the loss function at the beginning of each round. The hint is given in the form of a vector that is weakly correlated with the loss vector on that round. We show that the player can benefit from such a hint if the set of feasible actions is sufficiently round. Specifically, if the set is strongly convex, the hint can be used to guarantee a regret of O(log(T )), and if the set is q-uniformly convex for q ∈ (2, 3), the hint can be used to guarantee a regret of o( √ T ). In contrast, we establish Ω( √ T ) lower bounds on regret when the set of feasible actions is a polyhedron.

The paper concerns online linear optimization where at each trial, the player, prior to prediction, receives a hint about the loss function. The hint has a form of a unit vector which is weakly correlated with the loss vector (its angle's cosine with loss vector is at least alpha). The paper shows that:
- When the set of feasible actions is strongly convex, there exists an algorithm which gets logarithmic regret (in T). The algorithm is obtained by a reduction to the online learning problem with exp-concave losses. The bound is unimprovable in general, as shown in the Lower Bounds section.
- When the set of actions is (C,q)-uniformly convex (the modulus of uniform convexity scales as C eps^q), and the hint is constant among trials, a simple follow-the-leader algorithm (FTL) achieves a regret bound which improves upon O(sqrt(T)) when q is between 2 and 3. This is further extended to the case when the hint comes from a finite set of vectors.
- When the set of actions is a polyhedron, the worst case regret has the same rate as without hints, i.e. Omega(sqrt(T)), even when the hint is constant among trials.
The first result is obtained by a clever substitution of the original loss function by a virtual loss which exploits the hint vector and is shown to be exp-concave. Then, an algorithm is proposed which as subroutine uses any algorithm A_exp for exp-concave online optimization, feeding A_exp with virtual losses, and playing an action for which the original loss is equal to the virtual loss (by exploiting the hint) of A_exp. Since the virtual loss of the adversary is no larger then its original loss, the algorithm's regret is no more than the regret of A_exp.
Interestingly, the second result uses FTL, which does not exploit the hint at all (and thus does not even need to know the hint). So I think this result is more about the constraint on the losses -- there is a direction v, along which the loss is positive in each round. This result resembles results from [15] on FTL with curved constraint sets. I think the results from [15] are incomparable to those here, contrary to what discussion in the appendix say. On the one hand, here the authors extended the analysis to uniformly convex sets (while [15] only work with strongly-convex sets). On the other hand, [15] assume that the cumulative loss vector at each trial is separated from the origin, and this is sufficient to get logarithmic regret; whereas here logarithmic regret is achieved under a different assumption that each individual loss is positive along v (with constant alpha).
The extension of Theorem 4.2 to arbitrary hints (Appendix C) gives a bound which is exponential in dimension, while for q=2 Section 3 gives a much better algorithm which regret is only linear in the dimension. Do you think the exponential dependence on d for q > 2 is unimprovable?
The paper is very well written, and the exposition of the main results is clear. For each result, the authors give an extended discussion, including illustrative examples and figures, which builds up intuition on why the result holds. I checked all the proofs in the main part of the paper and did not find any technical problems. I found the contribution to online learning theory strong and significant. What the paper lacks, however, is a motivating example(s) of practical learning problem(s) in which the considered version of hint would be available.
---------------
I have read the rebuttal. The authors clarified comparison to [15], which I asked about in my review.


Controllable Invariance through Adversarial Feature Learning
Learning meaningful representations that maintain the content necessary for a particular task while filtering away detrimental variations is a problem of great interest in machine learning. In this paper, we tackle the problem of learning representations invariant to a specific factor or trait of data. The representation learning process is formulated as an adversarial minimax game. We analyze the optimal equilibrium of such a game and find that it amounts to maximizing the uncertainty of inferring the detrimental factor given the representation while maximizing the certainty of making task-specific predictions. On three benchmark tasks, namely fair and bias-free classification, language-independent generation, and lighting-independent image classification, we show that the proposed framework induces an invariant representation, and leads to better generalization evidenced by the improved performance.

The paper proposes to learn invariant features using adversarial training. Given s a nuisance factor (s attribute of x), a discriminator tries to predict the nuisance factors s  (an attribute of the input) given a encoder representation h=E(x,s)  , and an encode rE  tries to minimize  the prediction of nuisance factor and and to predict the desired output. The encoder is function of x and s.
Novelty:
The paper draws some  similarity with Ganian et al on unsupervised domain adaptation and their JMLR version. The applications to  Multilingual  machine translation and fairness applications  are to the best of the knowledge of the reviewer new in this context and are  interesting.
Comments:
- The application of the method to invariant features learning in object recognition is not realistic. In machine translation s being the one hot of the language at hand or in fairness s being an attribute of the input , is realistic. But if one wants invariance to rotation or lightning one can not supply the angle or the lightening to the encoder. E(x,s) is not realistic for geometric invariance i.e when s is indeed hidden and not given at test time. Have you tried in this case for instance E being just E(x)?
See for instance this work  https://arxiv.org/pdf/1612.01928.pdf  that uses actually a very similar training with just E(x)
- I think the paper should be more focused on the multiple sources / single target domain adaptation aspect. Geometric invariances are confusing in this setting, since the nuisance factor can not be assumed known at test time.
- How would you extend this to deal with continuous nuisances factors s?


Generalized Linear Model Regression under Distance-to-set Penalties
Estimation in generalized linear models (GLM) is complicated by the presence of constraints. One can handle constraints by maximizing a penalized log-likelihood. Penalties such as the lasso are effective in high dimensions, but often lead to unwanted shrinkage. This paper explores instead penalizing the squared distance to constraint sets. Distance penalties are more flexible than algebraic and regularization penalties, and avoid the drawback of shrinkage. To optimize distance penalized objectives, we make use of the majorization-minimization principle. Resulting algorithms constructed within this framework are amenable to acceleration and come with global convergence guarantees. Applications to shape constraints, sparse regression, and rank-restricted matrix regression on synthetic and real data showcase strong empirical performance, even under non-convex constraints.

I found much to like about this paper. It was very clear and well-written, and the proposed method is elegant, intuitive, novel (to the best of my knowledge), seemingly well motivated, and widely applicable. In addition, the main ideas in the paper are well-supported by appropriate numerical and real data examples.
I would like to see the authors compare their method to two other methods that I would see as major competitors: first, the relaxed LASSO (Meinshausen 2007) wherein we fit the lasso and then do an unpenalized fit using only the variables that the lasso has selected. As I understand the field, this is the most popular method and the one that Hastie, Tibshirani, and Wainwright (2015) recommend for users who want to avoid the bias of the lasso (but note that the shrinkage “bias” is sometimes desirable to counteract the selection bias or “winner’s curse” that the selected coefficients may suffer from). Second, best-subsets itself is now feasible using modern mixed-integer optimization methods (Bertsimas, Kind, and Mazumder, 2016) (if one of these competitors outperforms your method, it would not make me reconsider my recommendation to accept, since the method you propose applies to a much more general class of problems).
Bertsimas, Dimitris, Angela King, and Rahul Mazumder. "Best subset selection via a modern optimization lens." The Annals of Statistics 44.2 (2016): 813-852.
Hastie, Trevor, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity: the lasso and generalizations. CRC press, 2015.
Meinshausen, Nicolai. "Relaxed lasso." Computational Statistics & Data Analysis 52.1 (2007): 374-393.


f -GANs in an Information Geometric Nutshell
Nowozin et al showed last year how to extend the GAN principle to all fdivergences. The approach is elegant but falls short of a full description of the supervised game, and says little about the key player, the generator: for example, what does the generator actually converge to if solving the GAN game means convergence in some space of parameters? How does that provide hints on the generator's design and compare to the flourishing but almost exclusively experimental literature on the subject? In this paper, we unveil a broad class of distributions for which such convergence happens -namely, deformed exponential families, a wide superset of exponential families -. We show that current deep architectures are able to factorize a very large number of such densities using an especially compact design, hence displaying the power of deep architectures and their concinnity in the f -GAN game. This result holds given a sufficient condition on activation functions -which turns out to be satisfied by popular choices. The key to our results is a variational generalization of an old theorem that relates the KL divergence between regular exponential families and divergences between their natural parameters. We complete this picture with additional results and experimental insights on how these results may be used to ground further improvements of GAN architectures, via (i) a principled design of the activation functions in the generator and (ii) an explicit integration of proper composite losses' link function in the discriminator.

The authors identify several interesting GAN-related questions, such as to what extend solving the GAN problem implies convergence in parameter space, what the generator is actually fitting when convergence occurs and (perhaps most relevant from a network architectural point of view) how to choose the output activation function of the discriminator so as to ensure proper compositeness of the loss function.
The authors set out to address these questions within the (information theoretic) framework of deformed exponential distributions, from which they derive among other things the following theoretical results:
They present a variational generalization (amenable to f-GAN formulation) of a known theorem that relates an f-divergence between distributions to a corresponding Bregman divergence between the parameters of such distributions. As such, this theorem provides an interesting connection between the information-theoretic view point of measuring dissimilarity between probability distributions and the information-geometric perspective of measuring dissimilarity between the corresponding parameters. They show that under a reversibility assumption, deep generative networks factor as so called escorts of deformed exponential distributions. Their theoretical investigation furthermore suggests that a careful choice of the hidden activation functions of the generator as well as a proper selection of the output activation function of the discriminator could potentially help to further improve GANs. They also briefly mention an alternative interpretation of the GAN game in the context of expected utility theory.
Overall, I find that the authors present some interesting theoretical results, however, I do have several concerns regarding the practical relevance and usefulness of their results in the present form.
Questions & concerns:
To begin with, many of their theorems and resulting insights hold for the specific case that the data and model distributions P and Q are members of the deformed exponential family. Can the authors justify this assumption, i.e. elaborate on whether it is met in practice and explain whether similar results (such as Theorems 4 & 6) could be derived without this assumption?
One of the author’s main contributions, the information-geometric f-GAN identity in Eq.(7), relates the well-known variational f-divergence formulation over distributions to an information-geometric optimization problem over parameters. Can the authors explain what we gain from this parameter-based point of view? Is it possible to implement GANs in terms of this parameter-based optimization and what would be the benefits? I would really have liked to see experimental results comparing the two approaches to optimization of GANs. Somewhat surprising, the authors don’t seem to make use of the right hand side of this identity, other than to assert that solving the GAN game implies convergence in parameter space (provided the residual J(Q) is small). And why is this implication not obvious? Can the authors give a realistic scenario where convergence in the variational f-divergence formulation over distributions does not imply convergence in parameter space?
As an interesting practical implication of their theoretical investigation, the authors show that the choice of the output activation function g_f (see section 2.4 in Ref.[34]) matters in order for the GAN loss to have the desirable proper compositeness property. I found this result particularly interesting and would certainly have liked to see how their theoretically derived g_f (as the composition of f′ and the link function) compares experimentally against the heuristic choice provided in Ref. [34].
Consequences for deep learning and experimental results: Under the assumption that the generator network is reversible, the authors show that there exists an activation function v such that the generator’s hidden layers factor exactly as escorts for the deformed exponential family. In Table 2 (left), the authors compare several generator architectures against different choices of activation functions (that may or may not comply with this escort factorization). The plot for the GAN MLP, for instance, indicates that the “theoretically superior” μ-ReLU performs actually worse than the baseline ReLU (limiting case μ → 1). From their analysis, I would however expect the invertible μ-ReLU (satisfying reversibility assumption) to perform better than the non-invertible baseline (not satisfying reversibility assumption). Can the authors explain why the μ-ReLU performs worse? Can they comment on whether the DCGAN architecture satisfies the reversibility assumption and how these results compare against the Wasserstein GAN which is not based on f-divergences (and for which I therefore do not expect their theoretical analysis to hold)? Finally, as far as I can tell, their results on whether one activation (Table 2 center) or link function (Table 2 right) performs better than the others are all within error bars. I don’t think this provides sufficient support for their theoretical results to be relevant in practice.
Minor concerns:
Many of their results are derived within the framework of deformed exponential densities. As the general ML community might not yet be familiar with this, I would welcome it if the authors could provide more intuition as to what a deformation (or signature) and an escort is for instance.
In Theorem 8, the authors derive several upper bounds for J(Q). How do these bounds compare against each other and what are the implications for which activation functions to use? They also mention that this upper bound on J(Q) is decreasing with the normalization parameter of the escort Z. Can they elaborate a bit more on why this is a good thing and how we can leverage this?
In light of these comments, I believe the insights gained from the theoretical analysis are not substantial enough from the point of view of a GAN practitioner.


Neural system identification for large populations separating "what" and "where"
Neuroscientists classify neurons into different types that perform similar computations at different locations in the visual field. Traditional methods for neural system identification do not capitalize on this separation of "what" and "where". Learning deep convolutional feature spaces that are shared among many neurons provides an exciting path forward, but the architectural design needs to account for data limitations: While new experimental techniques enable recordings from thousands of neurons, experimental time is limited so that one can sample only a small fraction of each neuron's response space. Here, we show that a major bottleneck for fitting convolutional neural networks (CNNs) to neural data is the estimation of the individual receptive field locations -a problem that has been scratched only at the surface thus far. We propose a CNN architecture with a sparse readout layer factorizing the spatial (where) and feature (what) dimensions. Our network scales well to thousands of neurons and short recordings and can be trained end-to-end. We evaluate this architecture on ground-truth data to explore the challenges and limitations of CNN-based system identification. Moreover, we show that our network model outperforms current state-of-the art system identification models of mouse primary visual cortex.

I am no expert in this field, so right now I can not fully judge on the potential impact of the paper at hand in this field of science.
So first  I want to ask  a few clarifying questions to the authors to give me a clearer picture. I will also give a preliminary judgement at the end.
Introuction
34-47:  I think it should be mentioned that using more expressive models such like CNNs will come at a cost. Simple models such as GLMs are very interpretable, it also easier to argue for biological plausibility than with neural network trained by backpropagation.
I do agree with your point about:"Thus, we should be able to learn much more complex nonlinear functions by combining data from many neurons and learning a common feature space from which we can linearly predict the activity of each neuron."
Related Work:
The approaches of McIntosh et al.[17] and Batty et al. [18] seem to be the most similar to yours. What were the practical and theoretical reasons for not comparing to them in your experiments?
3 Learning a common feature space
This section describes a well-known insight from transfer learning in ML. Maybe establishing a link here is useful as you use a ML technique for a neuroscience application.
5 Learning non-linear feature spaces
Is this comparison to GLMs fair? Your proposed architecture seems to share a lot of properties with the ground truth (such as convolutional layers).
5.2 Application to data from primary visual cortex
It seems you needed to make a few modifications to your architecture. Can you respond to a possible çritique that says your approach only works better because you have more degrees of freedom to adjust hyperparameters to a given task?  Without these adjustments, were you still able to improve over previous state of the art?
--
The paper is well written and my initial impression is that this is a good contribution. The intuition to combining data from many neurons and learning a common feature space is intuitive and  appears straightforward. I can't fully judge if this is novel or incremental, compared to the works of  McIntosh et al.[17] and Batty et al. and I have some open questions with regards to the experiments.


Visual Interaction Networks: Learning a Physics Simulator from Video
From just a glance, humans can make rich predictions about the future of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains or require information about the underlying state. We introduce the Visual Interaction Network, a generalpurpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.

Summary
---
Interaction Networks (INs) [2] and the Neural Physics Engine [7] are two recent and successful instantiations of a physics engine. In each case a hand programed 2d physics engine was used to simulate dynamics of 2d balls that might bounce off each other like billiards, orbit around a made up gravitational well, or a number of other settings. Given ball states (positions and velocities), these neural approaches predict states at subsequent time steps and thus simulate simple entity interaction. This paper continues in the same vein of research but rather than taking states as input, the proposed architecture takes images directly. As such, the proposed Visual Interaction Networks (VINs) must learn to embed images into a space which captures state information and learn how to simulate 2d physics in that embedding space.
The VIN comes in three parts.
1) The visual encoder is a CNN which takes a sequence of frames which show the three most recent states of a simulation and embeds them into a different embedding for each object (the number of objects is fixed). Some tricks help the CNN focus on pairs of adjacent frames and do position computations more easily.
2) The dynamics predictor does the same job as an IN but takes state embeddings as input (from the visual encoder) and predicts state embeddings. It's implemented as a triplet of INs which operate at different time scales and are combined via an MLP.
3) The state decoder is a simple MLP which decodes state embeddings into hand coded state vectors (positions and velocities).
Hand coded states are be extracted from the ground truth 2d simulation and a loss is computed which forces decoded state embeddings to align to ground truth states during training, thus allowing the dynamics predictor to learn an appropriate 2d physics model.
This model is tested across a range of dynamics similar to that used in INs. Performance is reported both in terms of MSE (how close are state predictions to ground truth states) and relative to a baseline which only needs to extract states from images but not predict dynamics. Across different number of objects and differing dynamics VINs outperform all baselines except in one setting. In the drift setting, where objects do not interact, VINs are equivalent to an ablated version that removes the relational part from each IN used in the dynamics predictor. When compared to the simulation rolled out by the ground truth simulator using MSE all models show increased errors longer from the initial state (as expected), but VIN is smallest at every time step. A particularly interesting finding is that VINs even outperform INs which are given ground truth states and need not decode them from images. This is hypothesized to be due to noise from the visual component of VINs. Finally, videos of VIN rollouts are provided and appear qualitatively to convincingly simulate each intended 2d dynamics.
Strengths
---
This is a well written paper which presents the next step in the line of work on neural physics simulators, evaluates the proposed method somewhat thoroughly, and produces clean and convincing results. The presentation also appropriately orients the work relative to old and new neural physics simulators.
Weaknesses
---
This paper is very clean, so I mainly have nits to pick and suggestions for material that would be interesting to see. In roughly decreasing order of importance:
1. A seemingly important novel feature of the model is the use of multiple INs at different speeds in the dynamics predictor. This design choice is not
ablated. How important is the added complexity? Will one IN do?
2. Section 4.2: To what extent should long term rollouts be predictable? After a certain amount of time it seems MSE becomes meaningless because too many small errors have accumulated.  This is a subtle point that could mislead readers who see relatively large MSEs in figure 4, so perhaps a discussion should be added in section 4.2.
3. The images used in this paper sample have randomly sampled CIFAR images as backgrounds to make the task harder.
While more difficult tasks are more interesting modulo all other factors of interest, this choice is not well motivated.
Why is this particular dimension of difficulty interesting?
4. line 232: This hypothesis could be specified a bit more clearly. How do noisy rollouts contribute to lower rollout error?
5. Are the learned object state embeddings interpretable in any way before decoding?
6. It may be beneficial to spend more time discussing model limitations and other dimensions of generalization. Some suggestions:
    * The number of entities is fixed and it's not clear how to generalize a model to different numbers of entities (e.g., as shown in figure 3 of INs).
    * How many different kinds of physical interaction can be in one simulation?
    * How sensitive is the visual encoder to shorter/longer sequence lengths? Does the model deal well with different frame rates?
Preliminary Evaluation
---
Clear accept. The only thing which I feel is really missing is the first point in the weaknesses section, but its lack would not merit rejection.


Unsupervised learning of object frames by dense equivariant image labelling
One of the key challenges of visual perception is to extract abstract models of 3D objects and object categories from visual measurements, which are affected by complex nuisance factors such as viewpoint, occlusion, motion, and deformations. Starting from the recent idea of viewpoint factorization, we propose a new approach that, given a large number of images of an object and no other supervision, can extract a dense object-centric coordinate frame. This coordinate frame is invariant to deformations of the images and comes with a dense equivariant labelling neural network that can map image pixels to their corresponding object coordinates. We demonstrate the applicability of this method to simple articulated objects and deformable objects such as human faces, learning embeddings from random synthetic transformations or optical flow correspondences, all without any manual supervision.

Blue565
Unsupervised object learning from dense equivariant image labelling
An impressive paper, marred by flaws in exposition, all fixable.
The aim is to construct an object representation from multiple images, with dense labelling functions (from image to object), without supervision. Experiments seem to be very successful, though the paper would be improved by citing (somewhat) comparable numerical results on the MAFL dataset. The method is conceptually simple, which is a plus. The review of related methods seems good, though I admit to not knowing the field well enough to know what has been missed.
Unfortunately, insufficient details of the method are given. In particular, there is no discussion of the relationships between the loss functions used in practice (in (4) and (5)) and those in the theoretical discussion (in page 4, first unnumbered equation and (2)).
Also, I have some concerns about the authors’ use of the terms “equivariant” and “embedding” and their definition of “distinctive”. The standard definition of “equivariant” is: A function f is equivariant with respect to group of transformations if f(gx) = gf(x) for all x and all transformations g.
(We note that the usage of “equivariant” in [30] is correct.)
In contrast, f is invariant if f(gx) = f(x) for all x and all transformations g.
Your definition in (1) says that \Phi is invariant with respect to image deformations. So please change the manuscript title!
Note that both equivariance and invariance are a kind of “compatibility”.
The word “embedding” usually means an embedding of an object into a larger space (with technical conditions). This doesn’t seem to be how the word is used here (though, OK, the sphere is embedded in R^3). The manuscript describes the map \Phi as an embedding. Perhaps “labelling” would be better, as in the title.
The definition of “distinctive” on p4, line 124, is equivalent to: invariance of \Phi and injectivity of \Phi(x, \cdot) for every x.
On p4, line 27:  for a constant labelling, argmax is undefined, which is worrying.
Overall the quality of the English is good, however there are several typos that should be fixed, a few of which are noted below.
The paper is similar to [30], which is arXiv 1705.02193 (May 2017, unpublished as far as I know). They have significant overlap, however both the method and the datasets are different.
—Minor comments.
p1, line 17: typo: “form”->”from”
p2, Figure 1 caption:
- should be “Left: Given a large number of images”
- typos: “coordiante”, “waprs”
p4, first displayed equation: \alpha undefined. should be (x,g)
p5, line 142: \Phi(x,u) has become \Phi_u x without comment.
Fig 3 caption doesn’t explain clearly what the 2nd and 3rd rows are.


Welfare Guarantees from Data
Analysis of efficiency of outcomes in game theoretic settings has been a main item of study at the intersection of economics and computer science. The notion of the price of anarchy takes a worst-case stance to efficiency analysis, considering instance independent guarantees of efficiency. We propose a data-dependent analog of the price of anarchy that refines this worst-case assuming access to samples of strategic behavior. We focus on auction settings, where the latter is non-trivial due to the private information held by participants. Our approach to bounding the efficiency from data is robust to statistical errors and mis-specification. Unlike traditional econometrics, which seek to learn the private information of players from observed behavior and then analyze properties of the outcome, we directly quantify the inefficiency without going through the private information. We apply our approach to datasets from a sponsored search auction system and find empirical results that are a significant improvement over bounds from worst-case analysis.

This paper studies a non-worst-case notion of the price of anarchy (PoA). An auction A’s PoA is the worst-case ratio between the welfare of the optimal auction and the welfare in a Bayes-Nash equilibrium of A, taken over all value distributions and equilibria. The authors propose a non-worst case relaxation which they call the distributional PoA (DPoA), which is defined by a distribution D over bids. It measures the same ratio as the original PoA, but only over all value distributions and equilibria that could generate the bid distribution D. They bound the DPoA for single-dimensional settings and show that it can be estimated using samples from D.
In one example, the authors provide an upper bound on the DPoA for the single item first price auction (Lemma 1). There are a few things I couldn’t reproduce in the proof. In line 140, the authors write that 1/\mu*\int_0^{v_i(1 – e^{-\mu})} (1 – G_{-i}(z)) dz <= T_i. Isn’t it less than or equal to T_i/\mu? Also, it seems as though if I add 1/\mu*\int_0^{v_i(1 – e^{-\mu})} (1 – G_{-i}(z)) dz to the right-hand-side of Equation (8), I’ll get 1/\mu*\int_0^{v_i(1 – e^{-\mu})} (1 – G_{-i}(z) + G_{-i}(z)) dz = 1/\mu*\int_0^{v_i(1 – e^{-\mu})} dz = v_i(1 – e^{-\mu})/\mu. (The authors write that it’s just v_i(1 – e^{-\mu}).) Lastly, in line 145, the authors write that \sum_i T_i*x_i >= max_i T_i. Since this is a single-item auction, there’s exactly one x_i that equals 1 and the rest equal 0. Should the inequality be flipped? (\sum_i T_i*x_i <= max_i T_i.)
I would really like to see the proof of Lemma 5 written out. It’s not clear to me how it follows immediately from Lemma 9.9 in [5]. Rademacher complexity does have nice compositional properties. However, I’m not convinced, for example, that “multiplication of bid vectors with constants” is covered by Lemma 9.9 in [5]. Lemma 9.9 in [5] applies to function classes mapping some domain X to R. The function class defined by “multiplication of bid vectors with constants” maps from R^{n-1} to R^{n-1}. How exactly does Lemma 9.9 apply? Also, it seems like multiplication would have to be involved in the form of many payment functions. For example, in the first price auction, the payment function is b_i*x_i, where x_i can be written as {b_i > b_1}*…*{b_i > b_{i-1}}*{b_i > b_{i+1}}*…*{b_i > b_n}. Is there a way to write this without multiplication? The authors write that the conditions in Lemma 5 hold for sponsored search auctions. Is there no multiplication in that setting either?
I think this paper is currently better written for an economics community as opposed to a learning community. For example, it’s not clear to me what a pay-your-bid auction is. On line 187, the authors write that “For any auction, we can re-write the expected utility of a bid b: u_i(b, v_i) = x_i(b)(v_i – p_i(b)/x_i(b)).” What if x_i(b) = 0?
I think that overall, this is a really interesting contribution to beyond-worst-case analysis and the experiments seem to validate its practical relevance. However, I think the presentation could be improved to make it more understandable for an ICML audience. I’m also hesitant to recommend it for acceptance without a proof of Lemma 5.
======after rebuttal======
I'm willing to believe Lemma 5 if the authors are assuming that number of bidders is constant. I didn't realize that they were making this assumption. The authors should mention this explicitly in the preliminaries and/or the lemma statement.


Differentiable Learning of Logical Rules for Knowledge Base Reasoning
We study the problem of learning probabilistic first-order logical rules for knowledge base reasoning. This learning problem is difficult because it requires learning the parameters in a continuous space as well as the structure in a discrete space. We propose a framework, Neural Logic Programming, that combines the parameter and structure learning of first-order logical rules in an end-to-end differentiable model. This approach is inspired by a recently-developed differentiable logic called TensorLog [5], where inference tasks can be compiled into sequences of differentiable operations. We design a neural controller system that learns to compose these operations. Empirically, our method outperforms prior work on multiple knowledge base benchmark datasets, including Freebase and WikiMovies.
the structure is a discrete optimization problem, and one that involves search over a potentially large problem space. Many past learning systems have thus used optimization methods that interleave moves in a discrete structure space with moves in parameter space [12,13,14,27].
In this paper, we explore an alternative approach: a completely differentiable system for learning models defined by sets of first-order rules. This allows one to use modern gradient-based programming frameworks and optimization methods for the inductive logic programming task. Our approach is inspired by a differentiable probabilistic logic called TensorLog [5]. TensorLog establishes a connection between inference using first-order rules and sparse matrix multiplication, which enables certain types of logical inference tasks to be compiled into sequences of differentiable numerical operations on matrices. However, TensorLog is limited as a learning system because it only learns parameters, not rules. In order to learn parameters and structure simultaneously in a differentiable framework, we design a neural controller system with an attention mechanism and memory to learn to sequentially compose the primitive differentiable operations used by TensorLog. At each stage of the computation, the controller uses attention to "softly" choose a subset of TensorLog's operations, and then performs the operations with contents selected from the memory. We call our approach neural logic programming, or Neural LP.
Experimentally, we show that Neural LP performs well on a number of tasks. It improves the performance in knowledge base completion on several benchmark datasets, such as WordNet18 and Freebase15K [3]. And it obtains state-of-the-art performance on Freebase15KSelected [25], a recent and more challenging variant of Freebase15K. Neural LP also performs well on standard benchmark datasets for statistical relational learning, including datasets about biomedicine and kinship relationships [12]. Since good performance on many of these datasets can be obtained using short rules, we also evaluate Neural LP on a synthetic task which requires longer rules. Finally, we show that Neural LP can perform well in answering partially structured queries, where the query is posed partially in natural language. In particular, Neural LP also obtains state-of-the-art results on the KB version of the WIKIMOVIES dataset [16] for question-answering against a knowledge base. In addition, we show that logical rules can be recovered by executing the learned controller on examples and tracking the attention.
To summarize, the contributions of this paper include the following. First, we describe Neural LP, which is, to our knowledge, the first end-to-end differentiable approach to learning not only the parameters but also the structure of logical rules. Second, we experimentally evaluate Neural LP on several types of knowledge base reasoning tasks, illustrating that this new approach to inductive logic programming outperforms prior work. Third, we illustrate techniques for visualizing a Neural LP model as logical rules.

This paper develops a model for learning to answer queries in knowledge bases with
incomplete data about relations between entities. For example, the running example in the paper is answering
queries like HasOfficeInCountry(Uber, ?), when the relation is not directly present in the
knowledge base, but supporting relations like HasOfficeInCity(Uber, NYC) and CityInCountry(NYC, USA).
The aim in this work is to learn rules like HasOfficeInCountry(A, B) <= HasOfficeInCountry(A, C) && CityInCountry(C, B). Note that this is a bit different from learning embeddings for entities in a knowledge
base, because the rule to be learned is abstract, not depending on any specific entities.
The formulation in this paper is cast the problem as one of learning two components:
- a set of rules, represented as a sequence of relations (those that appear in the RHS of the rule)
- a real-valued confidence on the rule
The approach to learning follows ideas from Neural Turing Machines and differentiable program synthesis,
whereby the discrete problem is relaxed to a continuous problem by defining a model for executing the rules
where all rules are executed at each step and then averaged together with weights given by the confidences.
The state that is propagated from step to step is softmax distribution over entities
(though there are some additional details to handle the case where rules are of different lengths).
Given this interpretation, the paper then suggests that a recurrent neural network controller can
be used to generate the continuous parameters needed for the execution. This is analogous to the neural
network controller in Neural Turing Machines, Neural Random Access Machines, etc.
After the propagation step, the model is scored based upon how much probability ends up on the
ground truth answer to the query.
The key novelty is developing two representations of the confidence-weighted rules (one which is straightforward to optimize as a continuous optimization problem, and one as confidence weights on a set of discrete rules) along with an algorithm for converting from the continuous-optimization-friendly version to the rules version.
To my knowledge, this is a novel and interesting idea, which is a neat application of ideas from the
neural programs space.
Experimentally, the method performs very well relative to what looks to be a strong set of baselines.
The experiments are thorough and thoughtful with a nice mix of qualitative and quantitative results,
and the paper is clearly written.
One point that I think could use more clarification is what the interpretation is of what the
neural network is learning. It is not quite rules, because (if I understand correctly), the neural
controller is effectively deciding which part of the rules to apply next, during the course of execution.
This causes me a bit of confusion when looking at Table 3. Are these rules that were learned relative to
some query? Or is there a way of extracting query-independent rules from the learned model?


Adaptive Clustering through Semidefinite Programming
We analyze the clustering problem through a flexible probabilistic model that aims to identify an optimal partition on the sample X 1 , ..., X n . We perform exact clustering with high probability using a convex semidefinite estimator that interprets as a corrected, relaxed version of K-means. The estimator is analyzed through a non-asymptotic framework and showed to be optimal or near-optimal in recovering the partition. Furthermore, its performances are shown to be adaptive to the problem's effective dimension, as well as to K the unknown number of groups in this partition. We illustrate the method's performances in comparison to other classical clustering algorithms with numerical experiments on simulated high-dimensional data.
In this work, we build upon the work and context of [6], and transpose and adapt their ideas for point clustering: we introduce a semidefinite estimator for point clustering inspired by the findings of [17] with a correction component originally presented in [6]. We show that it produces a very strong contender for clustering recovery in terms of speed, adaptivity and robustness to model perturbations. In order to do so we produce a flexible probabilistic model inducing an optimal partition of the data that we aim to recover. Using the same structure of proof in a different context, we establish elements of stochastic control (see for instance Lemma A.1 on the concentration of random subgaussian Gram matrices in the supplementary material) to derive conditions of exact clustering recovery with high probability and show optimal performances -including in high dimensions, improving on [16], as well as adaptivity to the effective dimension of the problem. We also show that our results continue to hold without knowledge of the number of structures given one single positive tuning parameter. Lastly we provide evidence of our method's efficiency and further insight from simulated data.
Notation. Throughout this work we use the convention 0/0 := 0 and [n] = {1, ..., n}. We take a n b n to mean that a n is smaller than b n up to an absolute constant factor. Let S d−1 denote the unit sphere in R d . For q ∈ N * ∪ {+∞}, ν ∈ R d , |ν| q is the l q -norm and for M ∈ R d×d , |M | q , |M | F and |M | op are respectively the entry-wise l q -norm, the Frobenius norm associated with scalar product ., . and the operator norm. |D| V is the variation semi-norm for a diagonal matrix D, the difference between its maximum and minimum element. Let A B mean that A − B is symmetric, positive semidefinite.

The paper proposes a semidefinite relaxation for K-means clustering with isotropic and non-isotropic mixtures of sub-gaussian distributions in both high and low-dimensions.
The paper has a good technical analysis of the methods proposed in it. There are some supporting empirical experiments too. However, one thing would have been useful to see was how computation time scales with both p and n. The empirical experiments considered n=30, which is pretty small for many application situations. So, for a better picture, it would have been good to see some results on computation times.
The paper is nicely written and the flow of arguments in the paper is relatively clear. There are some minor typos like in line 111 in definition of b2, the subscript should be b in [n]\{a, b1}.
The paper extends an approach of semidefinite relaxation for K-means clustering to account for non-isotropic behavior of mixture components. The method proposed can work for high-dimensional data too. The method will be quite useful but possibly computationally heavy too. So, an idea about the computational resources necessary to implement this method would have been useful.


Diving into the shallows: a computational perspective on large-scale shallow learning
Remarkable recent success of deep neural networks has not been easy to analyze theoretically. It has been particularly hard to disentangle relative significance of architecture and optimization in achieving accurate classification on large datasets. On the flip side, shallow methods (such as kernel methods) have encountered obstacles in scaling to large data, despite excellent performance on smaller datasets, and extensive theoretical analysis. Practical methods, such as variants of gradient descent used so successfully in deep learning, seem to perform below par when applied to kernel methods. This difficulty has sometimes been attributed to the limitations of shallow architecture. In this paper we identify a basic limitation in gradient descent-based optimization methods when used in conjunctions with smooth kernels. Our analysis demonstrates that only a vanishingly small fraction of the function space is reachable after a polynomial number of gradient descent iterations. That drastically limits the approximating power of gradient descent leading to over-regularization. The issue is purely algorithmic, persisting even in the limit of infinite data. To address this shortcoming in practice, we introduce EigenPro iteration, a simple and direct preconditioning scheme using a small number of approximately computed eigenvectors. It can also be viewed as learning a kernel optimized for gradient descent. Injecting this small, computationally inexpensive and SGD-compatible, amount of approximate second-order information leads to major improvements in convergence. For large data, this leads to a significant performance boost over the state-of-the-art kernel methods. In particular, we are able to match or improve the results reported in the literature at a small fraction of their computational budget. For complete version of this paper see https://arxiv.org/abs/1703.10622.

This paper looks at analyzing the problem of long times required for convergence of kernel methods when optimized via gradient descent. They define a notion of computational reach of gradient descent after t iterations and the failure of gradient descent to reach the epsilon neighborhood of an optimum after t iterations. They give examples of simplistic function settings with binary labels where gradient descent takes a long time to converge to the optimum. They also point out that adding regularization improves the condition number of the eigenspectrum (resulting in possible better convergence) but also leads to overregularization at times.
They introduce the notion of EigenPro, where they pre-multiply the data using a preconditioner matrix, which can be pre-computed, improves the time for convergence by making the lower eigenvalues closer to the largest one as well as making cost per iteration efficient. They do a randomized SVD of the data matrix/kernel operator to get the eigenvalues and generate the pre-conditioning matrix using the ratio of the eigenvalues.They show that the per-iteration time is not much higher than kernel methods and demonstrate experiments to show that the errors are minimized better than standard kernel methods using less GPU time overall. The acceleration provided is significant for some of the Kernels.
The paper is dense in presentation but is well written and not very difficult to follow. However there would be a lot of details that can be provided to compare how the pre-conditioning matrix can influence gradient descent in general and whether it should always be applied to data matrices every time we try to train a linear model on data. The authors also provide the argument of how lowering the ratio of the smaller eigenvalues compared to the larger one makes the problem more amenable to convergence. It would be good to see some motivation/geometric description of how the method provides better convergence using gradient descent. It would also be interesting to explore if other faster algorithms including proximal methods as well as momentum based methods also can benefit from such pre-conditioning and can improve the rates of convergence for kernel methods.


Deliberation Networks: Sequence Generation Beyond One-Pass Decoding *
The encoder-decoder framework has achieved promising progress for many sequence generation tasks, including machine translation, text summarization, dialog system, image captioning, etc. Such a framework adopts an one-pass forward process while decoding and generating a sequence, but lacks the deliberation process: A generated sequence is directly used as final output without further polishing. However, deliberation is a common behavior in human's daily life like reading news and writing papers/articles/books. In this work, we introduce the deliberation process into the encoder-decoder framework and propose deliberation networks for sequence generation. A deliberation network has two levels of decoders, where the first-pass decoder generates a raw sequence and the second-pass decoder polishes and refines the raw sentence with deliberation. Since the second-pass deliberation decoder has global information about what the sequence to be generated might be, it has the potential to generate a better sequence by looking into future words in the raw sentence. Experiments on neural machine translation and text summarization demonstrate the effectiveness of the proposed deliberation networks. On the WMT 2014 English-to-French translation task, our model establishes a new state-of-the-art BLEU score of 41.5.

-----
After response: thank you for the thorough response. Two of my major concerns: the weakness of the baseline and the lack of comparison with automatic post-editing have been resolved by the response. I've raised my evaluation with the expectation that these results will be added to the final camera ready version.
With regards to the examples, the reason why I said "cherry-picked?" (with a question mark) was because there was no mention of how the examples were chosen. If they were chosen randomly or some other unbiased method that could be noted in the paper. It's OK to cherry-pick representative examples, of course, and it'd be more clear if this was mentioned as well. Also, if there was some quantitative analysis of which parts of the sentence were getting better this would be useful, as I noted before.
Finally, the explanation that beam search has trouble recovering from search errors at the beginning of the sentence is definitely true. However, the fact that this isn't taken into account at training and test time is less clear. NMT is maximizing the full-sentence likelihood, which treats mistakes at any part of the sentence equally. I think it's important to be accurate here, as this is a major premise of this work.
---------
This paper presents a method to train a network to revise results generated by a first-pass decoder in neural sequence-to-sequence models. The second-level model is trained using a sampling-based method where outputs are sampled from the first-pass model. I think idea in the paper is potentially interesting, but there are a number of concerns that I had about the validity and analysis of the results. It would also be better if the content could be put in better context of prior work.
First, it wasn't entirely clear to me that the reasoning behind the proposed method is sound. There is a nice example in the intro where a poet used the final word of the sentence to determine the first word, but at least according to probabilities, despite the fact that standard neural sequence-to-sequence models are *decoding* from left to right, they are still calculating the joint probability over words in the entire sentence, and indirectly consider the words in the end of the sentence in the earlier decisions because the conditional probability of latter words will be lower if the model makes bad decisions for earlier words. Of course this will not work if greedy decoding is performed, but with beam search it is possible to recover. I think this should be discussed carefully.
Second, regarding validity of the results: it seems that the baselines used in the paper are not very representative of the state of the art for the respective tasks. I don't expect that the every paper has to be improving SOTA numbers, but for MT if "RNNSearch" is the original Bahdanau et al. attentional model, it is likely missing most of the improvements that have been made to neural MT over the past several years, and thus it is hard to tell how these results will transfer to a stronger model. For example, the below paper achieves single-model BLEU scores on the NIST04-06 data sets that are 4-7 BLEU points higher than the proposed model:
> Deep Neural Machine Translation with Linear Associative Unit. Mingxuan Wang, Zhengdong Lu, Jie Zhou, Qun Liu (ACL 2017)
Third, the paper has a nominal amount of qualitative evaluation of (cherry-picked?) examples, but lacks any analysis of why the proposed method is doing better. It would be quite useful if analysis could be done to show whether the improvements are indeed due to the fact that the network has access to future target words in a quantitative fashion. For example, is the proposed method better at generating beginnings of sentences, where standard models are worse at generating words in this more information-impoverished setting.
Finally, there is quite a bit of previous work that either is similar methodologically or attempts to solve similar problems. The closest is automatic post-editing or pre-translation such as the following papers:
> Junczys-Dowmunt, Marcin, and Roman Grundkiewicz. "Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing." WMT2016.
> Niehues, Jan, et al. "Pre-Translation for Neural Machine Translation." COLING2016
This is not trained jointly with the first-past decoder, but even without joint training is quite useful. Also, the following paper attempts to perform globally consistent decoding, and introduces bilingual or bidirectional models that can solve a similar problem:
> Hoang, Cong Duy Vu, Gholamreza Haffari, and Trevor Cohn. "Decoding as Continuous Optimization in Neural Machine Translation." arXiv 2017.


Hierarchical Implicit Models and Likelihood-Free Variational Inference
Implicit probabilistic models are a flexible class of models defined by a simulation process for data. They form the basis for theories which encompass our understanding of the physical world. Despite this fundamental nature, the use of implicit models remains limited due to challenges in specifying complex latent structure in them, and in performing inferences in such models with large data sets. In this paper, we first introduce hierarchical implicit models (HIMs). HIMs combine the idea of implicit densities with hierarchical Bayesian modeling, thereby defining models via simulators of data with rich hidden structure. Next, we develop likelihood-free variational inference (LFVI), a scalable variational inference algorithm for HIMs. Key to LFVI is specifying a variational family that is also implicit. This matches the model's flexibility and allows for accurate approximation of the posterior. We demonstrate diverse applications: a large-scale physical simulator for predator-prey populations in ecology; a Bayesian generative adversarial network for discrete data; and a deep implicit model for text generation.

The paper defines a class of probability models -- hierarchical
implicit models -- consisting of observations with associated 'local'
latent variables that are conditionally independent given a set of
'global' latent variables, and in which the observation likelihood is
not assumed to be tractable. It describes an approach for KL-based
variational inference in such 'likelihood-free' models, using a
GAN-style discriminator to estimate the log ratio between a
'variational joint' q(x, z), constructed using the empirical
distribution on observations, and the true model joint density. This
approach has the side benefit of supporting implicit variational
models ('variational programs'). Proof-of-concept applications are
demonstrated to ecological simulation, a Bayesian GAN, and sequence
modeling with a stochastic RNN.
The exposition is very clear, well cited, and the technical machinery
is carefully explained. Although the the application of density ratio
estimation to variational inference seems to be an idea 'in the air'
and building blocks of this paper have appeared elsewhere (for example
the Adversarial VB paper), I found this synthesis to be cleaner,
easier to follow, and more general (supporting implicit models) than
any of the similar papers I've read so far.
The definition of hierarchical implicit models is a useful point in
theoretical space, and serves to introduce the setup for inference
in section 3. However the factorization (1), which assumes iid
observations, is quite restrictive -- I don't believe it technically
even includes the Lotka-Volterra or stochastic RNN models explored in
the paper itself! (since both have temporal dependence). It seems
worth acknowledging that the inference approach in this paper is more
general, and perhaps discussing how it could be adapted to problems
and models with more structured (time series, text, graph)
observations and/or latents.
Experiments are probably the weakest point of this paper. The 'Bayesian GAN'
is a toy and the classification setup is artificial; supervised learning is
not why people care about GANs. The symbol generation RNN is not
evaluated against any other methods and it's not clear it works
particularly well. The Lotka-Volterra simulation is the most
compelling; although the model has few parameters and no latent
variables, it nicely motivates the notion of implicit models and shows
clear improvement on the (ABC) state of the art.
Overall there are no groundbreaking results, and much of this
machinery could be quite tricky to get working in practice (as with
vanilla GANS). I wish the experiments were more compelling.
But the approach seems general and powerful, with the
potential to open up entire new classes of models to effective
Bayesian inference, and the formulation in this paper will likely be
useful to many reasearchers as they begin to flesh it out. For that
reason I think this paper is a valuable contribution.
Misc comments and questions:
Lotka-Volterra model: I'm not sure the given eqns (ln 103) are
correct. Shouldn't the Beta_3 be added, not subtracted, to model the
predator birth rate? As written, dx_2/dt is always negative in
expectation which seems wrong. Also Beta_2 is serving double duty as
the predator *and* prey death rate, is this intentional? Most sources
(including the cited Papamakarios and Murray paper) seem to use four
independent coefficients.
line 118: "We described two classes of implicit models" but I only see
one? (HIMs)
line 146: "log empirical log q(x_n)" is redundant
Suppose we have an implicit model, but want to use an explicit
variational approximation (for example the mean-field Gaussian in the
Lotka-Volterra experiment). Is there any natural way to exploit the
explicit variational density for faster inference?
Subtracting the constant log q(x) from the ELBO means the ratio
objective (4) no longer yields a lower bound to the true model
evidence; this should probably be noted somewhere. Is there an
principled interpretation of the quantity (4)? It is a lower bound on
log p(x)/q(x), which (waving hands) looks like an estimate of the
negative KL divergence between the model and empirical distribution --
maybe this is useful for model criticism?


Streaming Sparse Gaussian Process Approximations
Sparse pseudo-point approximations for Gaussian process (GP) models provide a suite of methods that support deployment of GPs in the large data regime and enable analytic intractabilities to be sidestepped. However, the field lacks a principled method to handle streaming data in which both the posterior distribution over function values and the hyperparameter estimates are updated in an online fashion. The small number of existing approaches either use suboptimal hand-crafted heuristics for hyperparameter learning, or suffer from catastrophic forgetting or slow updating when new data arrive. This paper develops a new principled framework for deploying Gaussian process probabilistic models in the streaming setting, providing methods for learning hyperparameters and optimising pseudo-input locations. The proposed framework is assessed using synthetic and real-world datasets.

This paper presents a variational and alpha-divergence approach specifically designed for the case of Gaussian processes with streaming inputs.
The motivation given in the introduction is convincing. The background section is very well presented, sets up the methodology for section 3 and at the same time explains why a batch based approach would be inadequate. Section 3 which is the most important, is however more difficult to follow. It is not a problem to understand how one equation leads to the other, but many important details are left out of the description making it very difficult to understand in a conceptual level. In particular, it'd help to explain the concept behind starting from eq. (5). Further, what is p(a|b) in eq. (6)? It seems to be an important element for our understanding of the method, as it somehow involves the update assumption. I was also wondering whether it'd be possible to arrive at the same bound by starting with the joint distribution and simply applying Jensen's inequality together with (4). That would probably be a clearer presentation.
The extension to alpha-divergences and related conversation is interesting, although I would also expect to see a deeper comparison with [9] in the methodological level. This is important since the proposed method converges to [9] for alpha zero, which is what the authors use in practice.
The experiments are structured nicely, starting from intuitive demonstration, moving to time-series and spatial data and finishing with classification data. The method does seem to perform well and run fast - it seems to be two time slower than SVI-GP, which still makes it a fast method. However, the experiments do not explore the hypothesis whether the SVI (batch-based) or the streaming based method performs better, simply because they use different approximations (we see that in time zero SVI is already much worse). Since the uncollapsed bound has been used for classification, I'd be keen to see it for the regression tasks too, to be able to compare with SVI-GP. Indeed, the SVI-GP seems to actually reduce the RMSE more than OSGP in Fig. 3.
Another useful property to explore / discuss, is what happens when the order of the data is changed. Further, the comparisons do not consider any other competitive method which is tailored to streaming data, such as [9] or [Gijsberts and Metta: Real-time model learning using incremental sparse spectrum GP Regression, 2013]. Finally, it'd be useful to either repeat the plots in Fig 2,3 with number of data in the x-axis (so all methods are aligned) or to simply put those and state the difference in running times.
Overall seems like a nice paper, which has good motivation and considers an important problem. However, the key points of the method are not demonstrated or explained adequately. I feel that re-working the presentation and experiments would significantly improve this paper.
EDIT after rebuttal period: The authors have clarified most of my concerns and I have raised my score to reflect that.


Spectral Mixture Kernels for Multi-Output Gaussian Processes
Early approaches to multiple-output Gaussian processes (MOGPs) relied on linear combinations of independent, latent, single-output Gaussian processes (GPs). This resulted in cross-covariance functions with limited parametric interpretation, thus conflicting with the ability of single-output GPs to understand lengthscales, frequencies and magnitudes to name a few. On the contrary, current approaches to MOGP are able to better interpret the relationship between different channels by directly modelling the cross-covariances as a spectral mixture kernel with a phase shift. We extend this rationale and propose a parametric family of complex-valued cross-spectral densities and then build on Cramér's Theorem (the multivariate version of Bochner's Theorem) to provide a principled approach to design multivariate covariance functions. The so-constructed kernels are able to model delays among channels in addition to phase differences and are thus more expressive than previous methods, while also providing full parametric interpretation of the relationship across channels. The proposed method is first validated on synthetic data and then compared to existing MOGP methods on two real-world examples.

The paper formulates a new type of covariance function for multi-output Gaussian processes. Instead of directly specifying the cross-covariance functions, the authors propose to specify the cross-power spectral densities and using the inverse Fourier transform to find the corresponding cross-covariances. The paper follows a similar idea to the one proposed by [6] for single output Gaussian processes, where the power spectral density was represented through a mixture of Gaussians and then back-transformed to obtaining a powerful kernel function with extrapolation abilities. The authors applied the newly established covariance for a synthetic data example, and for two real data examples, the weather dataset and the Swiss Jura dataset for metal concentrations.
Extending the idea from [6] sounds like a clever way to build new types of covariance functions for multiple-outputs. One of the motivations in [6] was to build a kernel function with the ability to perform extrapolation. It would have been interesting to provide this covariance function for multiple-outputs with the same ability. The experimental section only provides results with very common datasets, and the results obtained are similar to the ones obtained with other covariance functions for multiple-outputs. The impact of the paper could be increased by including experiments in datasets for which none of the other models are suitable.
Did you test for the statistical significance of the results in Tables 1 and 2. It seems that, within the standard deviation, almost all models provide a similar performance.