Adaptive Smoothed Online Multi-Task Learning
This paper addresses the challenge of jointly learning both the per-task model parameters and the inter-task relationships in a multi-task online learning setting. The proposed algorithm features probabilistic interpretation, efficient updating rules and flexible modulation on whether learners focus on their specific task or on jointly address all tasks. The paper also proves a sub-linear regret bound as compared to the best linear predictor in hindsight. Experiments over three multitask learning benchmark datasets show advantageous performance of the proposed approach over several state-of-the-art online multi-task learning baselines.

Authors proposed a method to do Online Multi-Task Learning. In their work, they: - Choose dataset from one domain (ex: spam detection, and 2 others), and split the data into multiple tasks (ex: each "user" is a task), and learn a per-user spam detector while leveraging the global data. - Formulate a loss objective (equations 1 and 2), that makes each parameter vector "w_k" do well on its own task loss "L_k" but at the same time, do well on other task losses "L_j" even for "j != k". - They model matrix \eta with entry \eta_{kj} being the "similarity" between tasks k and j. In particular, the total loss for selecting w_k is \sum_j \eta_{kj} L_j(w_k) -- in other words, w_k should incur low loss for other tasks where \eta_{kj} is high. They propose to define \eta as: \eta  = \alpha * I_K + (1 - \alpha) P; where \alpha is a scalar in [0, 1]; I_K is KxK identity matrix; and P is a row-stochastic matrix. - They discuss alternatives where \alpha and P are fixed (for example SHAMO [1]) but they also explain their algorithm, which trains P and varies (increments) \alpha over time, where the choice of \alpha is justified in section 2.5 (Regret Bound). - They propose an "Online Formulation" where the parameters evolve smoothly over time: They evolve to minimize the loss, but also to minimize the changes of parameters from previous iteration/example. Their "smoothness" is achieved by:    * minimizing the L2 norm of delta "w", this is similar [20], though they *dont* cite this similarity (maybe it is obvious??)    * minimizing the change in P, this is novel in online learning as far as my little knowledge goes: where they suggest to minimize the KL divergence: KL(P_k^(t+1) || P_k^(t)) - They show that their model performs better than others on 3 multi-task, binary datasets.
The paper is technically sound, with some minor flaws that are explained below. Overall, I am on the "edge" of deciding if this should be accepted or rejected as a Poster. The largest factor contributing to my "reject" thought is the incompleteness of the experiments, as explained below. Nonetheless, I will provide lots of things that confused me and can be improved:   Problem Setup: 1- L19, and in other places: you claim "online learning", yet you assume that all K tasks emit examples at the same time step, which inherently invalidates your motivation: now data from all tasks is equal in quantity. You mention on L153 that the assumption can be relaxed with "added complexity". I *think* you should mention how one can train without that assumption.  Language / math correctness: 2- You mention on L23, L156, and elsewhere, that your model only updates when it makes a mistake. This does not agree with the math you have. In particular, your hinge-loss emits updates *even* if your prediction is on the correct side of the hyper-plane but if it was within "1" (margin) away from the hyperplane.  3- You mention probability simplex and use \Delta^{K-1}. What does probability simplex mean? I tried googling it and failed to find a justifying answer. I know about the tetrahedron geometric simplex but I bit it is not it. Did you just mean a "distribution" [i.e. positive entries summing to 1]? And why set the dimensions to K-1? Is it because the diagonal entry in P is set to zero since there is an Identity matrix in equation 3 which fills the diagonal?  4- Line 139, equation (4), the notation got confusing! What is \eta_{kj}(p_k) ? I am assuming that it is the (k, j)-th entry with the given p_k. Regardless if my assumption is correct, please explain notation right after you present it the first time.  5- In equation 6, my immediate response was a shock that there is no \alpha in the update rule. I should calculate the derivative myself perhaps.  Unfair comparison:  6- For your argument on multi-task learning, You should experiment with OSMTL-ONE and OSMTL-ITL, to show that your data-sharing loss is effective. You only compare the -ONE and -ITL using the PA algorithm, which is not equivalent to OSMLT, and therefore it is not clear if your wins come from the "smooth" online updates (PA is 'aggressive') or if they come from the data sharing. You don't want to over-sell the multi-task (MT) aspect if the wins comes from the "smooth" (S). If somehow PA-ONE is equivalent to OSMLT-ONE, please make it clear in the paper.   Incompleteness: 7- The online experimental setting is not clear. In particular, do you go over the training data just once? What is the size of the test data? was it held-out during training? Please complete this information. If the datasets have Train : Test splits and you have used them, please mention so.


Operator Variational Inference
Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference ( ), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling-allowing inference to scale to massive data-as well as objectives that admit variational programs-a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of on a mixture model and a generative model of images.

The paper introduces Operator Variational Inference, that defines novel variational objectives by using operators and extends standard variational approaches based on the minimization of the KL divergence. Operator Variational Inference is efficient even without the standard mean field assumption, and it allows to approximate the posterior with a broad class of flexible distributions. When using the reparametrization gradient, the introduced variational objectives do not require the variational distribution q to be analytically tractable, and one can only learn how to sample from it with a Variational Program.
There has been a lot of interest recently in defining variational objectives that improve upon the standard variational bound, as minimizing KL[q||p] leads to variational distributions that severely underestimate the variance of the posterior. By using operators, this paper introduces a novel way to form variational objectives that allow to fit a more flexible class of variational approximations, either by optimizing the parameters of q or by learning how to sample from it (variational programs). I find this approach very interesting, and a promising research direction.  The presentation of the paper, especially in the theoretical part, is sometimes confusing and difficult to understand: - I found section G/H/I/J of the supplementary material necessary to understand more intuitively Operator Variational Inference. They should be put in the main paper, or at least referenced more explicitly and using section names. - In lines 144-161 you only introduce in detail the type of distributions obtained with the reparametrization, but in section 4 you also present the score function gradient. This is confusing, especially as it is only said in the supplementary material that in this case it is needed an analytically tractable variational distribution, as needed in equation 8.  The mixture of Gaussian experiment is useful to understand the behaviour of the different variational objectives, but it looks strange that when using the KL objective the variational approximation does not pick only one of the modes. The MNIST experiment is not enough to assess how the different methods perform. You may consider using more that 10 units (e.g. 100) and reporting how much time it takes for the different methods to converge (with "Stein" you also need to optimize the test function). More importantly, I would have liked to see results also on different data sets.   Finally, an interesting related work is "Variational Inference with Renyi Divergence", Y Li, RE Turner - arXiv preprint arXiv:1602.02311, 2016


Fast learning rates with heavy-tailed losses
We study fast learning rates when the losses are not necessarily bounded and may have a distribution with heavy tails. To enable such analyses, we introduce two new conditions: (i) the envelope function sup f ∈F | • f |, where is the loss function and F is the hypothesis class, exists and is L r -integrable, and (ii) satisfies the multi-scale Bernstein's condition on F. Under these assumptions, we prove that learning rate faster than O(n −1/2 ) can be obtained and, depending on r and the multi-scale Bernstein's powers, can be arbitrarily close to O(n −1 ). We then verify these assumptions and derive fast learning rates for the problem of vector quantization by k-means clustering with heavy-tailed distributions. The analyses enable us to obtain novel learning rates that extend and complement existing results in the literature from both theoretical and practical viewpoints.

This paper gives learning bounds with heavy-tailed losses in hypothesis classes with logarithmic entropy numbers. If the r-th moment of the supremum loss over all hypotheses in the class is integrable  and a  'multiscale Bernstein condition' holds then fast rates ( close to O(n^{-1}) ) can be obtained.
This paper provides some new results in an important area which is receiving more and more attention: fast rates when loss functions are unbounded and heavy-tailed. Existing results based on empirical process theory often rely on bounded or sub-Gaussian loss, and the heavy tails (hence non-sub-Gaussian) case is considerably harder.  The results presented seem sound and are definitely novel. They rely on results of Sara van de Geer and collaborators on concentration inequalities for unbounded empirical processes. The  material is very technical and I would suggest moving even some more material to the appendix. This would allow you to include more extensive discussion of some points:   The multiscale Bernstein condition is novel and crucial and I would like to see a simple example to explain the difference between the microscopic and macroscopic scales that the authors talk about (the k-means example is nice and really illustrates that the approach is useful,  but it didn't really make the intuition of/need for  multiscale Bernstein clear to me). In this respect it is of interest that Audibert (Fast Learning Rates..., Annals of Statistics, 2009) shows that a simple , standard Bernstein condition is enough to get fast rates, even for unbounded losses (and it appears even for heavy-tailed losses), if one uses a pseudo-Bayesian estimator with online-to-batch conversion. The authors of the present  paper essentially prove uniform convergence rates for the whole class {\cal F}, which implies that e.g. ERM will attain these rates, so it's different from Audibert who uses a very specific  randomized estimator and measures error by the expected error according to the predictor's randomization. Nevertheless, it suggests that the multi-scale may perhaps not be necessary, so I would like to some more (informal) explanation of why the authors need it. Do they think it may be dispensed with? (the authors should most definitely check out Audibert's paper. Note that with 'improper learners' that adopt a hypothesis outside the assumed class one can often get better rates, but Audibert 'merely' randomizes estimators).  There's also a very recent paper by Mehta and Grunwald (Fast Rates for Unbounded Losses, Arxiv, 2016) where results are proved that are similar in spirit, although the conditions are quite different (yet they seem 'analogous'): instead of the existence-of-moments-of-envelope condition, there is a 'witness condition',  and instead of a multiscale Bernstein condition, there is the v-PPC condition (for bounded losses, both multiscale Bernstein and v-PPC  conditions are equivalent to the standard Bernstein/central condition; for unbounded losses they seem VERY different). It would be good if the authors could provide some discussion on the relation between these papers (though I concede this does not seem easy; proof techniques are quite different).   I would also like to see an explanation of why the approach does not immediately generalize to polynomial entropy numbers. You use chaining in the proofs, so superficially it seems as if the approach would be amenable to larger entropy numbers. You very briefly say in the conclusion that you expect that you can extend it (BTW 'polynomial' there should be 'logarithmic'!)  but why isn't the generalization immediate?   It should also be pointed out explicitly that the in-probablity results are polynomial and not logarithmic in \delta. This is presumably unavoidable with heavy tails, but it  is different from what's normally proved in learning theory - see also the paper by Sabato and Hsu (2015).   But let me emphasize that the main results here are quite different from those by Audibert/Grunwald&Mehta/Sabato&Su, and given the challenging nature of this area I do recommend acceptance.    Minor issues:  The abstract is longer than necessary 'weak-tailed' (beneath Lemma 2.1) is not standard terminology Section 3.4: \gamma-stochastic mixability should be 1-\gamma stochastic mixability (at least that's what I get form Van Erven et al. 2015, I did not check the Mehta&Williamson paper).  Thm 4.2, fine -> finite, minimizer -> minimizesrs References: many et al's which should not be there


Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.

Inspired by batch-normalisation, the paper proposes to reparametrise neural networks by decoupling length and direction of the connection wight vectors. A theoretical analysis shows that the resulting gradients rescale the standard wight gradient and project it onto the compliment of the wight vector. It is agued that this leads to an improvement of the conditioning of the gradient.  Experiments show that the proposed reparametrisation leads to competitive results on various bench mark sets and improves the performance of different NN models.
The suggested reparametrisation and its theoretical analysis are very interesting and I enjoyed reading the paper.   However, some points in the theoretical analysis could be improved:   The paper argues that the new parametrisation improves the conditioning matrix of the gradient, but neither a strong theoretical argument nor a empirical demonstration for this are given. In line 127 it is said "Empirically, we find that w is often (close to) a dominant eigenvector of the covariance matrix C", but the correspond experiments are neither shown in the paper nor in the supplemental material.   In line 122/123 the authors claim "It has been observed that neural networks with batch normalization also have this property (to be relatively insensitive to different learning rates), which can be explained by this analysis.". However, it did not became clear to me, how the analysis of the previous sections can be directly transferred to batch normalisation.   Regarding the experimental part:  Except from the experiments on MNIST in section 5.2 each experiment is only run once (i.e. with one random initialisation). This makes an proper experimental evaluation of the different models challenging. Did the authors made sure that the observed differences were not only observes for this specific initialisation and was the same initialisation shared by all models?  Several reported runs would moreover allow for statements about statistical significant performance differences.   Why were all different methods (including the different versions of bath-normalisation) only compared in the first experiment, while the further compared only weight-normalisation against the standard parametrisation? It would be interesting to see how the other methods performed on this tasks as well.   Often for the standard parametrisation a smaller learning rate is chosen than for weigh-normalisation? Couldn't this result in the faster learning? And did larger learning rates lead to to large variance for the standard parametrisation?  Furthermore, the effective learning rates are probably different as well. It could be interesting to scale all gradients to the same length to analyse the pure effect of the changed direction of the gradients.   Where the results reported in Figure 6 gained after 200 epochs? Would have further training improved the results?   Missing references: When introducing mean-only batch normalisation the following paper could have been cited: Y. LeCun, L. Bottou, G. Orr, and K. R. Mu ̈ller. Efficient backprop. In Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, page 546. Springer Berlin/Heidelberg, 1998.  In line 268/269 it says "it is not clear how batch normalization could be applied to this model (LSTM)". Note, that recent research by Cooijmans et all (2016) on "Recurrent Batch Normalisation" tries to answer this question (https://arxiv.org/pdf/1603.09025)   minor comments:  - in equation 9:  w’ is not defined, in my opinion it took me a while to figure out that you mean the transposed vector (not the derivative), so that ww' is an outer product. -line 127: "gradient in v" -> gradient with respect to v -line 266: "enables" -> enable  - Figure2 and 6 are Tables


Hierarchical Question-Image Co-Attention for Visual Question Answering
A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.
1 .

This paper presents a new method for VQA. As a main novelty, the paper proposes a co-attention mechanism that jointly reasons about visual (image) and textual (question) attentions. Furthermore, the proposed method considers a hierarchical model by using co-attention maps at a word, phrase, and question levels. Results indicate a slightly improvement with respect to alternative state-of-the-art approaches.
The paper presents an incremental contribution with respect to previous methods for VQA that only exploit an image attention mechanism guided by question data. Here, they also consider a question attention mechanism guided by image information. In this sense, the main hypothesis of this work is that jointly considering visual and question attention mechanisms can improve the performance of current VQA systems. I agree that this hypothesis can be relevant for the case of long questions, but I believe there is also a risk that question based attention guided by image information can be misleading, in the sense that usually an image includes several information sources, while the question is more focused. In Figure 3, authors include a graph that shows the impact of question length in performance, while this figure seems to show a tendency, the effect is still weak, maybe a numerical analysis can help to support this point. I believe, an analysis of potential differences (not only question length) between most common errors of previous works (only image attention) and the proposed approach (image and question attention) can help to support the relevance of the proposed attention mechanism.  In terms of numerical results, in the abstract and in Section 4.3, the authors claim an increase in performance with respect to alternative state-of-the-art approaches of 60.4% to 62.1% in VQA dataset, and 61.6% to 65.4% in COCO-QA. However, these numbers are misleading, they consider different visual features. In particular, alternative methods use VGG, while the proposed method uses deep residual features which provide superior performance. Considering the same VGG features, the proposed method presents a smaller increase 60.3% to 60.5% in VQA and 61.6% to 63.3% in COCO-QA. While the authors also include results of their method using VGG features, it is important to clarify this part, for example, in the abstract.  Authors claim novelty in the CNN convolucional 1D method for text processing, but I believe the proposed method include just minor variations with respect to previous works, such as https://arxiv.org/pdf/1511.02274v2.pdf.   In some parts of the paper, the description of the method is very limited. I believe, it will be difficult to repeat the results of this paper based only on the information included. As an example, it is not clear to me, how do the method combine the hierarchical representations for the question embedding (word, phrase, question)?, is it the same approach used to encode the answers (Eq.7)?. Also, in Eq. 6, why do you use the same projection matrix for visual and question features?  The use of Eq.4 instead of Eq.3 to obtain the attention maps needs further explanation. It will be good to include comparative results. Also, it will be good to include more intuitions about the different terms in Eq. 4.   In Figure 4, the color attention code for words is not clear. In this sense, the visualization in Figure 4 does not support the claim: "our model is capable of localizing the main part of the question". I recommend to normalize the values to highlight with red the most relevant parts of the question.   Minor issues: - Line 107, what the "c" stands for? - Line 88 hierarchical - Line 124, c_{i,j} should be C.  - Eq. 5, i=1..N instead of n=1..N. - Line 142, equal to 1.  - Line 150, missing a "to".  - Line 136 Breifly. - Line 237 "we can find some have"


Sample Complexity of Automated Mechanism Design
The design of revenue-maximizing combinatorial auctions, i.e. multi-item auctions over bundles of goods, is one of the most fundamental problems in computational economics, unsolved even for two bidders and two items for sale. In the traditional economic models, it is assumed that the bidders' valuations are drawn from an underlying distribution and that the auction designer has perfect knowledge of this distribution. Despite this strong and oftentimes unrealistic assumption, it is remarkable that the revenue-maximizing combinatorial auction remains unknown. In recent years, automated mechanism design has emerged as one of the most practical and promising approaches to designing high-revenue combinatorial auctions. The most scalable automated mechanism design algorithms take as input samples from the bidders' valuation distribution and then search for a high-revenue auction in a rich auction class. In this work, we provide the first sample complexity analysis for the standard hierarchy of deterministic combinatorial auction classes used in automated mechanism design. In particular, we provide tight sample complexity bounds on the number of samples needed to guarantee that the empirical revenue of the designed mechanism on the samples is close to its expected revenue on the underlying, unknown distribution over bidder valuations, for each of the auction classes in the hierarchy. In addition to helping set automated mechanism design on firm foundations, our results also push the boundaries of learning theory. In particular, the hypothesis functions used in our contexts are defined through multi-stage combinatorial optimization procedures, rather than simple decision boundaries, as are common in machine learning.

This paper deals with the sample complexity of automated mechanism design for the problem of maximizing the revenue in a combinatorial auction (CA). Given a class of auction mechanisms, the automated mechanism design takes as input samples from the bidders’ valuation distribution (which, in practice, may be the history records from the previous auctions), and output the choice of auction mechanism with high revenue. This work presents several upper bounds on the sample complexities for various auction classes.  Although Morgenstern and Roughgarden (reference [19] in this paper) studied the same problem of bounding the sample complexities of CA, their work only deals with “simple auctions” which can be reduced to the single-bidder setting. In contrast, this paper studies the hierarchy of deterministic CA families consists of VCG-based mechanisms. As opposed to sequential auctions, these auction classes have more complex structure, and provide the auctioneer more degrees of freedom in designing mechanism.   In particular, the authors study the affine maximizer auction (AMA), the virtual valuation CA (VVCA), the lambda-auction, the mixed bundling auction with reserve prices (MBARPs) and the mixed bundling auction (MBA). For AMA, VVCA and lambda-auction, the authors provide certain negative results. Firstly, an exponential upper bound on the Rademacher complexity of the revenue function is given for the class AMA. (Theorem 2) Based on this, an exponential upper bound on sample complexity for all these three classes is provided, while two exponential lower bounds are provided for lambda-auction and VVCA respectively as complementary results (Theorem 1).   For the less general models MBARPs and MBA, on the other hand, many positive results are given. For MBA with only 2 bidders, the authors first show that the pseudo-dimension is 2 (theorem 3), and provide a polynomial upper bound on sample complexity (Theorem 4). Then the authors generalize these results for the general case with n bidders (Theorem 5, 6). Finally, the authors provide polynomial upper bounds on the pseudo-dimension and sample complexity of MBARPs.
This paper appears to be mostly a followup on the work of [19], "learning simple auctions" where techniques used to prove generalization error bounds were used to help reason about the design of mechanisms. In the present submission, the authors focus a much broader classes of auctions (affine maximize auction, virtual valuation CA, lambda-auction, mixed bundling auction with reserve price, and mixed bundling auction), and provide much more general results.  The techniques and results are interesting. But in the context of the previous work already introducing the underlying idea that you can apply learning theory to mechanism design problems of this form, I do wonder whether the results are suitably exciting to deserve publication in NIPS. (The paper is clearly above the bar for other conference venues)


"Short-Dot": Computing Large Linear Transforms Distributedly Using Coded Short Dot Products
Faced with saturation of Moore's law and increasing size and dimension of data, system designers have increasingly resorted to parallel and distributed computing to reduce computation time of machine-learning algorithms. However, distributed computing is often bottle necked by a small fraction of slow processors called "stragglers" that reduce the speed of computation because the fusion node has to wait for all processors to complete their processing. To combat the effect of stragglers, recent literature proposes introducing redundancy in computations across processors, e.g., using repetition-based strategies or erasure codes. The fusion node can exploit this redundancy by completing the computation using outputs from only a subset of the processors, ignoring the stragglers. In this paper, we propose a novel technique -that we call "Short-Dot" -to introduce redundant computations in a coding theory inspired fashion, for computing linear transforms of long vectors. Instead of computing long dot products as required in the original linear transform, we construct a larger number of redundant and short dot products that can be computed more efficiently at individual processors. Further, only a subset of these short dot products are required at the fusion node to finish the computation successfully. We demonstrate through probabilistic analysis as well as experiments on computing clusters that Short-Dot offers significant speed-up compared to existing techniques. We also derive trade-offs between the length of the dot-products and the resilience to stragglers (number of processors required to finish), for any such strategy and compare it to that achieved by our strategy.

In this paper, the authors introduce a method to compute a matrix-vector product in parallel. The method relies on a key assumption that the number of available processors is larger than the number of columns in the matrix. Under this assumption, they propose a method that distributes the columns of a modified matrix, computes (sparse) vector dot products in parallel and only has to wait for a small number of the processors to finish computing the dot products. This way, they can avoid the problem of waiting for straggler processors to finish.
The paper is easy to understand, and the concept seems clear. Detailed comments follow (both major and minor, in no particular order):  - My biggest concern about the practicality of the method is the (pretty restrictive) assumption that the number of available processors is larger than the number of columns. Almost all modern applications have a large number of variables to fit, and the number of available processors is almost always smaller (at most in the order of hundreds). So while the math is correct and the results look impressive, i’d like to see this aspect discussed and addressed in detail.   - line 10: a only —> only a  - Figure 2: at this point, with K > M, short dot is not necessarily faster. it depends on the sparsity (s) as well. You address this later, but this has to be made clear before directing one’s attention to fig 2.   - line 142: chosen —> choosing  - The flow of ideas to prove theorem 1 is confusing. It is better to separate lemmas from the main theorem proof.   - Figure 5 does not paint a full picture. For example, what happens when \mu is varied? Why is MDS worse than no coding at all? Are there any values of \mu for which the repetition scheme is good? Why is the expected time  = 1 even when the number of dot products is nearly 0 for MDS and repetition?   - line 233: Could you include a plot akin to fig 5 but taking the encoding and decoding complexity also into account? Although it is a one-time computation, it cannot be ignored.   - How do the performances of the various algorithms compare on real datasets? You mention linear regression, but experiments on real data would be nice to have.


Nearly Isometric Embedding by Relaxation
Many manifold learning algorithms aim to create embeddings with low or no distortion (isometric). If the data has intrinsic dimension d, it is often impossible to obtain an isometric embedding in d dimensions, but possible in s > d dimensions. Yet, most geometry preserving algorithms cannot do the latter. This paper proposes an embedding algorithm to overcome this. The algorithm accepts as input, besides the dimension d, an embedding dimension s ≥ d. For any data embedding Y, we compute a Loss(Y), based on the push-forward Riemannian metric associated with Y, which measures deviation of Y from from isometry. Riemannian Relaxation iteratively updates Y in order to decrease Loss(Y). The experiments confirm the superiority of our algorithm in obtaining low distortion embeddings.

This paper presents a new type of data embedding that preserves the geometry of the data as defined by a Reimannian metric (inner product on the tensor space of every point). Then they provide a method for gradient descent to solve this problem as the objective is not convex.
This paper claims that contrary to the plethora of embeddings it utilizes [3, 4,12,19,20] that the Reimannian metric is best for geometry preservation without motivating this thought or providing the intuition for this claim. The authors claim that unlike previous methods they do not provide a one-shot mapping, but they ignore the existence of methods like tsne which do employ gradient descent in deriving the embedding. Finally they do not test this on sufficiently many datasets of significant size show how their embeddings are better than previously existing datasets or are practical in terms of runtime.   Figure 1: Their initialization is already very similar to the true shape; how does the algorithm perform without so much prior information? Also, the plot of MSE/loss vs sigma is poorly drawn. I'm not sure if they're constrained by runtime, but to connect the dots between their data points I would expect to see A) intervals for MSE & loss drawn from several randomized initializations and B) more points tested for sigma  Figure 2: Poorly formatted; it's not immediately obvious that they have two data sets with the same embeddings. Also, top-left/top-right etc is difficult to follow; there should be sections of the figure labeled A-G with corresponding sections in the legend. Again, the line plots are not publication quality. There need to be more points sampled on the X axis or replicates of each condition to provide an interval; I assume the trends going up then down then up are an artifact of individual runs?  Figure 3: Their analysis here is only half-complete. They present a real dataset, show superficial improvements in its analysis as a result of their method, but provide no physics-based analysis that would suggest their method actually improves the ability of scientists to analyze this kind of data.


Consistent Kernel Mean Estimation for Functions of Random Variables
We provide a theoretical foundation for non-parametric estimation of functions of random variables using kernel mean embeddings. We show that for any continuous function f , consistent estimators of the mean embedding of a random variable X lead to consistent estimators of the mean embedding of f (X). For Matérn kernels and sufficiently smooth functions we also provide rates of convergence. Our results extend to functions of multiple random variables. If the variables are dependent, we require an estimator of the mean embedding of their joint distribution as a starting point; if they are independent, it is sufficient to have separate estimators of the mean embeddings of their marginal distributions. In either case, our results cover both mean embeddings based on i.i.d. samples as well as "reduced set" expansions in terms of dependent expansion points. The latter serves as a justification for using such expansions to limit memory resources when applying the approach as a basis for probabilistic programming.

This paper quantifies the price of estimating the distribution of $f(X)$, when {\em dependent} Kernel mean embedding is used. As shown in Eq. (2), O(1/n) of parametric rate is achievable with KME based on i.i.d. samples. However, when "expansion-points" are dependent, the rate is not known and possibly worse. The authors quantify this gap precisely in Theorem 2. The take-home message is that when f is smooth, there is no loss, but for non-smooth f, you pay the price in sample complexity. Hence, for practitioners, this gives a guideline for deciding when to use techniques such as reduced set expansions.   The paper is very well written, and I really appreciate the fact that the authors took the effort in Sections 1.1, 1.2, 1.3, and 1.4 to motivate when dependency naturally arises in KME and how memory trades off with accuracy. I learned something non-trivial that I did not know before I read this paper.
Quantifying the price of dependent expansion points is an interesting mathematical question. However, this paper could improve in motivating the general readers from machine learning community to get interested in a broader topic of Kernel Mean Embedding. The authors attempt to do this in the last paragraph in Section 4, relating it to Probabilistic Programming Systems, which seems to be a weak connection. AS a reader, I was curious as to where such techniques of KME and reduced set expansions can be potentially used, or are used currently in solving some application specific problems.   Another disappointing aspect of the paper is that the authors did not delve deeper into the question Multiple arguments in Section 3. What is currently provided is a direct corollary of the Theorem 2, and the paper assigns too much space for something that has little information over what is already said. However, it is an interesting question to ask: given i.i.d. samples from joint distribution $(X,Y)$, what is the sample-efficient way to construct KME? (although this is outside the scope of this paper)  Overall, the questions addressed in this paper is theoretically very interesting, but mainly theoretical since there is no algorithmic innovation. Further, the techniques necessary to prove the main results seem to be largely available from existing literature e.g. [Kanagawa and Fukumizu 2014] as the authors point out, which I appreciate since I could not have made the connection if the authors did not state it so clearly. Hence, this paper addresses an interesting question and gives a clean answer, but the solution ended up being simple that it lost in Novelty/originality.   Typos:  - On page 2, $\{(x'_j,w'_j)\}_{i=1}^n$ should be $\{(x'_j,w'_j)\}_{j=1}^N$. - On page 2, $\{(f(x'_j),w'_j)\}_{i=1}^n$ should be $\{(f(x'_j),w'_j)\}_{j=1}^N$.


Unsupervised Learning for Physical Interaction through Video Prediction
A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment. Many existing methods for learning the dynamics of physical interactions require labeled object information. However, to scale real-world interaction learning to a variety of scenes and objects, acquiring labeled data becomes increasingly impractical. To learn about physical object motion without labels, we develop an action-conditioned video prediction model that explicitly models pixel motion, by predicting a distribution over pixel motion from previous frames. Because our model explicitly predicts motion, it is partially invariant to object appearance, enabling it to generalize to previously unseen objects. To explore video prediction for real-world interactive agents, we also introduce a dataset of 59,000 robot interactions involving pushing motions, including a test set with novel objects. In this dataset, accurate prediction of videos conditioned on the robot's future actions amounts to learning a "visual imagination" of different futures based on different courses of action. Our experiments show that our proposed method produces more accurate video predictions both quantitatively and qualitatively, when compared to prior methods.

The paper presents three deep learning methods for predicting future images of a scene. The authors learn a distribution of the pixels in the previous scene for each pixel. The first method predicts pixel movements and the other two are designed to implicitly estimate object movements. The network includes multiple masks including one for the background. The methods were evaluated on pushing data collected using multiple robots and the Human3.6M dataset.
The paper mentions appendices and a link to videos, but the link does not work and the appendices are not included in the main file nor the supplementary material. I am reviewing the paper as it is.  The descriptions of the three methods need to be expanded. The method descriptions rely heavily on Fig. 1 and assume that most of the details are clear from the network structure. I did not find this to be the case. What is the structure of the m/M? Do the they adapt to the local image patches or are they constant over the entire image and rely on the masks? The authors should consider including a visualization of an m/M. Can they be interpreted?  The amount of data collected using the robot setup is impressive. The methods also outperform the state of the art. That being said, the paper is missing a discussion of the methods’ weaknesses and limitations. For example, why is the image resolution 64x64 pixels? Is it possible to use higher resolutions? The pushing task mainly involves planar movements parallel to the image plane. Can the method be applied to more 3D movements like a can tipping over? The authors should consider showing an example of a larger turning motion from the Human3.6M dataset for figure 7.  The qualitative results also leave room for improvement. There is a considerable amount of blurring in the images.  The task of predicting videos is obviously very challenging and I would not expect perfect results. However, the authors need to discuss the sources of these errors and how they may be addressed in the future.   The authors could consider exploiting the robot setup more for the evaluations. Can the robot grasp a moved object using only the predicted image? Is the quality of the prediction sufficient?  The MSE criterion is based on the optical flow computed for the ground truth data and the predicted data. It would be helpful to include an example image showing the optical flow computed for both images. How well does the optical flow handle the blurring?  Are there situations where the authors would recommend one of their proposed methods over the others?  The authors could merge some of the plots to make additional space for discussions.   How does the performance change when the background masks are removed?  Overall, the paper presents an interesting method for a very challenging task. The authors also present a large new dataset for video prediction. The network structure is impressive and seems well thought through. I therefore think that the authors have done great work. This work is unfortunately not reflected in the paper. Adding more detailed descriptions and thorough discussions of the methods’ limitations would make the paper more accessible and significantly increase the paper’s long term impact.


Optimistic Bandit Convex Optimization
We introduce the general and powerful scheme of predicting information re-use in optimization algorithms. This allows us to devise a computationally efficient algorithm for bandit convex optimization with new state-of-the-art guarantees for both Lipschitz loss functions and loss functions with Lipschitz gradients. This is the first algorithm admitting both a polynomial time complexity and a regret that is polynomial in the dimension of the action space that improves upon the original regret bound for Lipschitz loss functions, achieving a regret of e O T 11/16 d 3/8 . Our algorithm further improves upon the best existing polynomial-in-dimension bound (both computationally and in terms of regret) for loss functions with Lipschitz gradients, achieving a regret of e O T 8/13 d 5/3 .

The present paper provides a new algorithm---optimistic BCO---for bandit convex optimization. It leads to improved regret bounds among computationally efficient algorithms, for both Lipschitz loss functions and loss functions with Lipschitz gradients.
The presentation is globally clear, the paper seems technically solid, but a careful proofreading is necessary before publication. For instance there are too many notational inconsistencies throughout the paper.  As far as originality is concerned, the improvement in the regret bounds is new, but they may seem incremental, especially because there are known to be suboptimal (putting algorithmic issues aside). Indeed Bubeck and Eldan (2015) proved through a non-constructive approach that the optimal rate in T is sqrt(T) up to log factors. The authors also mention the recent manuscript by Hazan and Li (2016) who claim to get the optimal sqrt(T) regret (up to log factors) but whose algorithm has an exponential dependency w.r.t. the dimension of the decision set, both in terms of its regret bound and its computational complexity. On the contrary, the algorithm provided here has a polynomial dependency w.r.t. the ambient dimension.  I cannot figure out whether the prediction of information re- use is just a technical trick or if it can be used to get optimal bounds. Could the authors provide some insight about that? Besides, why just predict \bar{g}_{t+1} and not \bar{g}_{t+2},...,\bar{g}_{t+k}?   Some specific comments: - Is it an oblivious adversary? (cf. definition of regret line 54) - Line 126: please specify \widehat{g}_{t-i} when t-i \leq 0. - Previous work : please also mention existing lower bounds - Figure 1: k is also a parameter? - \eta in the definition of x_{t+1} is not at the right place? See also line 183 and 330. - In theorem 1 and 2, must k be an integer? - In lemma 5, x is not defined, x^star= min ... - Line 218 missing square bracket - Line 317, please say somewhere that \widehat{f}_t(x) \geq f_t(x), - Line 330 I do not understand the conditional expectation, what is F_t? - Page 14: inversion between y_t and z_t and T and t - End of lemma 7 where do you check that \lambda(x_t,F_t)\leq 1/2? - Line 347 Since ... it is an x_t - Line 352 what is E_{t-i}? - Same line, I do not understand the upper bound of the third term by L. - Line 357 in Lemma 2 there is a factor C in the upper bound of the local norm of \widehat{g}_s.


How Deep is the Feature Analysis underlying Rapid Visual Categorization?
Rapid categorization paradigms have a long history in experimental psychology: Characterized by short presentation times and speeded behavioral responses, these tasks highlight the efficiency with which our visual system processes natural object categories. Previous studies have shown that feed-forward hierarchical models of the visual cortex provide a good fit to human visual decisions. At the same time, recent work in computer vision has demonstrated significant gains in object recognition accuracy with increasingly deep hierarchical architectures. But it is unclear how well these models account for human visual decisions and what they may reveal about the underlying brain processes. We have conducted a large-scale psychophysics study to assess the correlation between computational models and human behavioral responses on a rapid animal vs. non-animal categorization task. We considered visual representations of varying complexity by analyzing the output of different stages of processing in three stateof-the-art deep networks. We found that recognition accuracy increases with higher stages of visual processing (higher level stages indeed outperforming human participants on the same task) but that human decisions agree best with predictions from intermediate stages.
Overall, these results suggest that human participants may rely on visual features of intermediate complexity and that the complexity of visual representations afforded by modern deep network models may exceed the complexity of those used by human participants during rapid categorization.

The paper compares performance of deep networks to psychophysical performance of humans performing classification on the same images.  From the correlation in performance it is claimed that human performance is best modeled based on intermediate levels of a deep net.
Overall this is an interesting study, and I do appreciate the method of using Mechanical Turk to collect data. As far as I know the result is novel, that humans correlate most strongly with intermediate layers of a variety of different deep nets. I think the Discussion section is a bit to grandiose in claiming that layer 10 of their fine-tuned VGG-16 model presents some sort of supporting evidence of the Serre et al claim about how many processing stages humans have. I also would have liked to see an investigation of what the representations at 70% total depth were for the different networks. It would have supported their argument better if they could have asserted that the features at 70% depth in a smaller network, such as AlexNet corresponded strongly with the features found in a larger net, such as VGG-19.   Specific  comments:  As far as I could tell they did not use a mask for the human subjects.  Why?   If we assume that the mask serves the purpose it is intended to serve, then it should stop contributions of recurrent and top-down processing from influencing the human's response. I think using a mask would increase support for their argument that error correlations can be used to assess the "depth" of the human recognition system.  The authors compared the human results to results from three staple deep nets: AlexNet, VGG16, and VGG19. For convolutional layers they grab a random sample of features from each layer. They fix the number of features they use, which I find a little concerning. Given that deeper layers in these networks have fewer features, if they fix the sample size then they are going to get a more complete set of features as they ascend the hierarchy, and thus a more complete description of the layer's representation. Given the features from a certain Deep Net layer, they use a SVM to perform the 2AFC task. This surprises me, I would think that they would have just used a couple of fully connected layers as is done in the ImageNet challenge.   They find that the humans and deep nets have the highest response correlation when using features from layers at 70% of the model's depth. It seems like this is consistent across models, which is a bit confusing. The conclusion they draw from the result is that humans use intermediate-level features. However, it is based on the assumption that the features at whatever layer corresponds to 70% of the model's depth are uniform across models. They do not discuss this directly, and I would be very interested to see if it is true at all. Does modifying the total model depth have any influence on the features learned at a relative depth?  They further note that the maximum correlation observed with one of their models (fined tuned VGG16) occurs with the 10th layer of processing, which is about how many "processing stages" Serre, Oliva, & Poggio claim are in the ventral stream. I find this whole argument misleading, given that the specific layer depth that correlates with humans is going to vary monotonically with the total depth of the network. If you consider their result that the highest correlation is always at around 70% of the total depth, it seems reasonable to just choose a network that has the correct number of total layers to match the Serre et al.'s findings. Also, they claim that they use VGG-16 because it is the best performing network, but that is not true according to Figure 3a. VGG-19 performs best out of the box, only after fine-tuning VGG-16 do they do the best. They do not comment on how well a fine-tuned VGG-19 net would do.


Probing the Compositionality of Intuitive Functions
How do people learn about complex functional structure? Taking inspiration from other areas of cognitive science, we propose that this is accomplished by harnessing compositionality: complex structure is decomposed into simpler building blocks. We formalize this idea within the framework of Bayesian regression using a grammar over Gaussian process kernels. We show that participants prefer compositional over non-compositional function extrapolations, that samples from the human prior over functions are best described by a compositional model, and that people perceive compositional functions as more predictable than their non-compositional but otherwise similar counterparts. We argue that the compositional nature of intuitive functions is consistent with broad principles of human cognition.

This paper tests the hypothesis that people's hypothesis space for learning functions is compositional. Function learning is formalized as Bayesian regression with functional forms given by compositional Gaussian process kernels (summing and multiplying radial basis, linear, and periodic kernels). A spectral mixture kernel provides an alternative hypothesis. A series of experiments demonstrates that people prefer to extrapolate functions (given points on the function) using compositional structure and find compositional functions to be more predictable.
Technical quality The number of experiments reported in the paper is impressive. Moreover, the use of the MCMCP method to probe people's posterior belief distribution is interesting. However, because Experiments 1a, 2a, and 3 use compositional ground truth, the conclusions that can be drawn from these experiments seem limited. It is not clear if people prefer compositional structures in these experiments because the data are generated from them or because they always prefer compositional structures. Furthermore, why are the compositional kernels not compared to the spectral mixture kernel in Experiments 2a and 2b? Perhaps the most convincing demonstration that people prefer compositional structure is Experiment 1b, in which compositional extrapolations are chosen more often even when the ground truth is the spectral mixture.  I wonder if it would be possible to incorporate a prior over the compositional structures that expresses people's preference for linear and periodic functions and (presumably) simpler combinations, and look at the full posterior belief distribution rather than just the likelihood?  A small comment on Experiment 3: I don't think many participants would find drawing a function using a Bezier curve very intuitive. What did the MTurk participants think of this? Maybe this experiment could be conducted on touchscreen devices instead.  Novelty/originality This work introduces the compositional kernel and includes several novel and interesting behavioral experiments.  Potential impact This paper will probably be of interest to many cognitive scientists and some computer scientists as well. However, it seems somewhat unsurprising from the outset that people prefer to think of functions as consisting of compositional structure rather than as a spectral density (unless there is strong evidence to believe the latter). The paper would be improved if some reasons to think otherwise were presented at the beginning.  Clarity and presentation The paper is well-written and clear. However, it would be helpful to include in the Introduction some examples of why function learning is important for an organism. I did not fully understand the math in Sections 2, 3.1, and 3.2, but I appreciated that the authors tried to provide intuitive explanations for the mathematical formalisms at every step. One small comment is that in lines 43-45, there are two symbols, x_asterisk and x_star, and it's not clear what they each denote (or if they're actually the same symbol).


Ladder Variational Autoencoders
Variational autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch-normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.

The authors propose a modification of the variational auto-encoder that consists in correcting the model's generative distribution with a data dependent likelihood factors, in a process that resembles the one used by the ladder network: a non-probabilistic auto-encoder that tries to match the values of the hidden layers in the encoder and decoder networks.  The proposed modification is based on using a deterministic inference network in the variational auto-encoder that samples the top latent variables from a Gaussian distribution, as in the variational auto-encoder, after that the latent variables at intermediate layers are sampled from a product of Gaussian factors, where one of the factors come from the original generative model, and the others come from the inference network.  The experiments show that the proposed modification results in better estimates of the test-log-likelihood. The proposed model learns richer latent distributions. The experiments performed also show that using batch normalization and a deterministic warm-up approach during training are important for obtaining improved results.
Quality: the paper seems to be technically sound. However, I do not like that the authors report results on table 1 that are not directly comparable because of differences in the training procedure.  Clarity: the paper is clearly written. However, the authors could have done a much better job at explaining how the inference network generates values for the latent variables. I took me too long to fully understand the whole process. It is not clear how the authors apply the batch normalization procedure during training. They should describe this in the text. I assume they only use batch-normalization in the inference network since using it in the generative network would change the generative model. What are the discontinuous lines in Figure 2? They should be described in the caption of the figure.  Originality: the contribution is original. The authors proposed the first probabilistic equivalent of the ladder network, a model that has been very successful. They also provide significant insights on how to improve the VAE by using sensible modifications for the training process by using warm-ups and batch normalization.  Significance: the experimental results clearly show the advantages of the proposed approach. The modifications are therefore significant. However, it still not clear why this approach should be preferred with respect to other possible alternatives.


Learning shape correspondence with anisotropic convolutional neural networks
Convolutional neural networks have achieved extraordinary results in many computer vision and pattern recognition applications; however, their adoption in the computer graphics and geometry processing communities is limited due to the non-Euclidean structure of their data. In this paper, we propose Anisotropic Convolutional Neural Network (ACNN), a generalization of classical CNNs to nonEuclidean domains, where classical convolutions are replaced by projections over a set of oriented anisotropic diffusion kernels. We use ACNNs to effectively learn intrinsic dense correspondences between deformable shapes, a fundamental problem in geometry processing, arising in a wide variety of applications. We tested ACNNs performance in challenging settings, achieving state-of-the-art results on recent correspondence benchmarks.

This paper proposes a novel model called Anisotropic Convolutional Neural Network (ACNN), which generalizes classical convolutional neural networks to non-Euclidean domains. This work builds on two methods, namely, the Geodesic Convolutional Neural Network ("Geodesic Convolutional Neural Networks on Riemannian Manifolds") and the concept of Anisotropic Diffusion Descriptors. While sharing many foundational ideas, it describes a new "fused" approach resembling the idea of the Geodesic Convolutional Neural Network, but using kernels of the anisotropic Laplacian operator and avoiding a construction of geodesic local patches (which, if I understand everything correctly, appears to be an artificial and inconvenient construct). The suggested approach is demonstrated to outperform other approaches when applied to the problems of full and partial mesh correspondence.
> Technical quality.  I believe the paper to be of a high technical quality. The calculations are sound and appear to not have any major flaws or mistakes (other than small inaccuracies outlined below). The experimental methods used to evaluate the proposed method seem to be appropriate.  > Novelty/originality.  This work builds on the publication "Geodesic Convolutional Neural Networks on Riemannian Manifolds" and the idea of Anisotropic Diffusion Descriptors. It shares the foundational concepts and notation with these articles, but proposes a novel "fused" approach combining the strengths of these two prior publications. While not entirely novel in its details, the combination of two approaches seems to be a successful idea and as such a promising "step in the right direction".  > Potential impact or usefulness.  This work proposes a model which shows a significantly improved performance compared to prior approaches and seems to show a potential of having a large impact in its specialized sub-field.  > Clarity and presentation.  The paper is well written. However, the clarity of the presentation is lacking, seemingly due to the fact that the authors expect the reader to be familiar with prior publications on Geodesic Convolutional Neural Networks and Anisotropic Diffusion Descriptors. Some of the minor inaccuracies/problems include: (a) the conductivity tensor defined in Eq. (2) should have an additional x-dependent multiplier; (b) in Eq. (7), D should have subscripts; (c) the subsection about the discretization of the anisotropic Laplacian operator should include at least some references (Anisotropic Diffusion Descriptors, ...?); (d) instead of "the solution of heat equation (1) at time t is obtained by applying the anisotropic heat operator ...", the sentence should read "the solution of heat equation (1) with D=D_{\alpha \theta} at time t is obtained by applying the anisotropic heat operator ..."; (e) although the meaning of the notation < a,b >_H should be clear for the majority of the readers, including a definition or a reference would improve the clarity of the article.


SPALS: Fast Alternating Least Squares via Implicit Leverage Scores Sampling
Tensor CANDECOMP/PARAFAC (CP) decomposition is a powerful but computationally challenging tool in modern data analytics. In this paper, we show ways of sampling intermediate steps of alternating minimization algorithms for computing low rank tensor CP decompositions, leading to the sparse alternating least squares (SPALS) method. Specifically, we sample the Khatri-Rao product, which arises as an intermediate object during the iterations of alternating least squares. This product captures the interactions between different tensor modes, and form the main computational bottleneck for solving many tensor related tasks. By exploiting the spectral structures of the matrix Khatri-Rao product, we provide efficient access to its statistical leverage scores. When applied to the tensor CP decomposition, our method leads to the first algorithm that runs in sublinear time per-iteration and approximates the output of deterministic alternating least squares algorithms. Empirical evaluations of this approach show significant speedups over existing randomized and deterministic routines for performing CP decomposition. On a tensor of the size 2.4m ⇥ 6.6m ⇥ 92k with over 2 billion nonzeros formed by Amazon product reviews, our routine converges in two minutes to the same error as deterministic ALS.

This submission is concerned with approximate low-rank tensor approximation. Given a, say, 3-dimensional tensor T and a target rank R, the goal is to find vectors families of vectors a_r, b_r, and c_r for r=1,...,R such that T is well approximated by sum_{r=1}^R a_r * b_r * c_r, where here * denotes outer product. Henceforth A is the matrix with a_r as columns, etc. Error in the approximation is measured using the Frobenius norm. The submission focuses on one algorithm to solve this problem, namely alternating least squares (ALS), and discusses how to speed it up using leverage score sampling. Specifically, ALS in this context fixes 2 out of 3 of A, B, C, then tries to minimize the Frobenius norm error by optimizing the third. It then loops over these 3 families as the variable to optimize. The main observation of the submission is that once 2/3 of these matrices are fixed, the optimization problem is equivalent to a traditional least squares problem in which the matrix X is n^2 x n, and now standard results in the randomized linear algebra literature thus imply that it suffices to sample only O(n log n) rows of X via nonuniform sampling of the rows of X via its statistical leverage scores. Unfortunately computing the statistical leverage scores of X is expensive, since it has n^2 rows. The main observation of this paper is that for ALS, X has special structure (it is the "Khatri-Rao product" of the two matrices that were fixed), and a lemma is then proven in the submission showing that one can upper bound the leverage scores of X by simple functions of the leverage scores of the two matrices that were fixed. This overall leads to a faster algorithm for computing upper bounds on the leverage scores of X, which then translates into a faster ALS variant at the end of the day.
As the "LS" of ALS suggests, each iteration of ALS for tensor decomposition amounts to solving a least squares regression problem. The main contribution of this submission is then to observe that good upper bounds on the leverage scores of the underlying matrix can be quickly approximated due to special structure of the matrix, namely Theorem 3.2 of the submission. This is the only, albeit important, novel observation of this paper. Once Theorem 3.2 is obtained, filling in the other details is standard. From this one observation, they are able to compare quite favorably with [37] (see Figure (a) on page 8).  The plus side is clear: the ability to compete well with the state-of-the-art based on quite a simple observation. The two main downsides I see are: (1) they seem to be more comparable with [37] in empirical performance as opposed to hands down beating it, and (2) in figure (b) I didn't see any explanation of why again [37] wasn't compared with.  I was also confused as to why SPALS(alpha) figure numbers weren't monotonic with alpha in figure (a) on page 8.  Minor comments: * Some typos: "nearly optimality" --> "near optimality"; "while approximates the" --> "while approximating the"; "for large tensor" --> "for large tensors"; "each rank-1 components are" --> "each rank-1 component is"; "It is a challenging tasks" --> "It is a challenging task"; --> "one of the most powerful tool" --> "one of the most powerful tools"; "we provide the efficient" --> "we provide an efficient"; "In the remaining of this section" --> "In the remainder of this section"; "toolset for estimating the statistical leverage score" --> "toolset for estimating the statistical leverage scores"; "the statistical leverage scores of the i-th row" --> "the statistical leverage score of the i-th row"; "score of certain row" --> "score of a certain row"; "first inequality is because of" --> "first inequality is because"; "equals to $R$" --> "equals $R$"; "an rank-1" --> "a rank-1"; "In the remaining of this section" --> "In the remainder of this section"; "separate the calculation to two parts" --> "separate the calculation into two parts"; "evaluating former expression" --> "evaluating the former expression"; "second term is spare" --> "second term is sparse"; "by the leverage score of the design" --> "by the leverage scores of the design"; "not directly utilize" --> "not directly utilizing"; "requires an one-time" --> "requires a one-time"; "routines fairly recent" --> "routines was fairly recent"; "moves the non-zeros around" --> "moves the non-zeroes around"; "considered as the data" --> "considered as data"; "piecemeal invocation" --> "piecemeal invocations"; "each rank-1 components" --> "each rank-1 component" * On page 2, line 80 "(i,j) also represents the index i+Ij between 1 and IJ"; I found this confusing. What does it mean? * On page 4, lines 128 and 129: "Its optimality in solving ... of linear regression." It cannot really be explained by this; it's not so trivial. The upper bound requires using some matrix concentration, like matrix Bernstein or matrix Chernoff, or the non-commutative Khintchine inequality. * On page 4, line 133: "O(r log n)"; why did n enter the picture? I don't know any results of this form where n enters the picture.  * I found Figure (a) on page 8 to be confusing. First, how is "error" measured? Also, why is the SPALS(alpha) error not monotonic as a function of alpha? What are the units on the time and error? * On page 8, does n = 1000 mean the tensor is 1000 x 1000 x 1000? (All 3 dimensions are n, right?)


Testing for Differences in Gaussian Graphical Models: Applications to Brain Connectivity
Functional brain networks are well described and estimated from data with Gaussian Graphical Models (GGMs), e.g. using sparse inverse covariance estimators. Comparing functional connectivity of subjects in two populations calls for comparing these estimated GGMs. Our goal is to identify differences in GGMs known to have similar structure. We characterize the uncertainty of differences with confidence intervals obtained using a parametric distribution on parameters of a sparse estimator. Sparse penalties enable statistical guarantees and interpretable models even in high-dimensional and low-sample settings. Characterizing the distributions of sparse models is inherently challenging as the penalties produce a biased estimator. Recent work invokes the sparsity assumptions to effectively remove the bias from a sparse estimator such as the lasso. These distributions can be used to give confidence intervals on edges in GGMs, and by extension their differences. However, in the case of comparing GGMs, these estimators do not make use of any assumed joint structure among the GGMs. Inspired by priors from brain functional connectivity we derive the distribution of parameter differences under a joint penalty when parameters are known to be sparse in the difference. This leads us to introduce the debiased multi-task fused lasso, whose distribution can be characterized in an efficient manner. We then show how the debiased lasso and multi-task fused lasso can be used to obtain confidence intervals on edge differences in GGMs. We validate the techniques proposed on a set of synthetic examples as well as neuro-imaging dataset created for the study of autism.

After the rebuttal: I thank the authors for answering my many questions. There are important technical/experimental details left to the rebuttal that should have been in the submitted manuscript (and will hopefully be included in the final appendix). Also, I am not as optimistic as the authors about the ease of analysis if one replaces the individual sparsity assumption with another norm, but I don't think it is a major issue. I suggest that the authors modify the final text appropriately to clarify this detail. Overall, I still think this is a technically solid paper with some potential to impact the field. I have updated my scores slightly to reflect this. ===================================== The authors propose to improve a parametric (asymptotic) test for differences between precision matrices by replacing the de-biased lasso with a "multitask fused lasso". The proposed regularizer is constructed specifically for the case where the differences between the two precision matrices are sparse, in addition to separate sparsity of each precision matrices. The primary innovation is the (asymptotic) de-biasing of the estimator, resulting in a parametric (asymptotically normal) test. The technique is motivated by applications to testing differences in brain connectivity between two populations. Results on simulated data suggest that the proposed de-biasing improves statistical power to detect differences. Result on real data are focused on comparison between the proposed parametric test and the permutation based standard, suggesting similar performance without the expense of permutation.
The goal of improving testing for differences between graphs is clearly relevant to neuroimaging and other application domains. While the specifics are somewhat incremental, I think this is a great idea, and reasonably well executed.  Major issues: * Please explain specifically which gradients are used to get from (4) to (5). This derivation seems incorrect if one takes separate derivatives with respect to \beta_1 and \beta_2. How do you end up with a sum of terms (and not two separate terms)? Or is this the sum of the two gradient terms? If accepted, I strongly suggest more a detailed derivation in the appendix.  * While the authors spend significant effort motivating the need to test sparse differences between "dense" graphs (e.g. repeatedly referring to [19] to suggest that brain networks are not sparse), the resulting techniques are in-fact designed to test sparse differences between "sparse" graphs. I think this is unfortunate, because the idea of improving tests for edge differences is sufficiently interesting and potentially useful without the mismatch between motivation and algorithm. I strongly suggest that the authors modify the motivation to better reflect the proposed test.  * One of the issues with the regression approach to precision matrix estimation is that one gives up finite sample guarantees of both symmetry and positive semi-definiteness. Please add some details on how you handle this e.g. do you symmetrize before testing for differences? How do you do this? Do you implement cross-validation using the log likelihood? If so, how do you implement the determinant? I think these details are important for reproducibility and practical utility.  * I am not convinced of the motivation to (significantly) over-sample one of the groups in the two group test e.g. for simulations n1=800, n2=20 - 150. The authors suggest that this reflects of practice, but I am not aware of many situations where this occurs. Instead, most datasets collected for two sample testing tend to be balanced. Could the authors add more detail or a reference?  * On a related note, Could the authors comment on how the results change if the two groups are balanced? Is there still a gain from using the fused Lasso? How much?  * The algorithms are designed for individual sample level data, while the presented experiments are at the group level. As the fMRI data is collected for each individual, how do the authors construct samples used in the presented experiments? Are the data concatenated for all subjects in each group? Please clarify.  Minor issues: * Please note the number of brain regions used for the fMRI experiments. This is an important detail that is easy to add.   Suggestions: * It would be quite informative to compare to "multitask fused lasso" results with no debasing at all, to clarify what is gained.


Spectral Learning of Dynamic Systems from Nonequilibrium Data
Observable operator models (OOMs) and related models are one of the most important and powerful tools for modeling and analyzing stochastic systems. They exactly describe dynamics of finite-rank systems and can be efficiently and consistently estimated through spectral learning under the assumption of identically distributed data. In this paper, we investigate the properties of spectral learning without this assumption due to the requirements of analyzing large-time scale systems, and show that the equilibrium dynamics of a system can be extracted from nonequilibrium observation data by imposing an equilibrium constraint. In addition, we propose a binless extension of spectral learning for continuous data. In comparison with the other continuous-valued spectral algorithms, the binless algorithm can achieve consistent estimation of equilibrium dynamics with only linear complexity.

This paper concerns the modeling of discrete- and continuous-valued stochastic processes with observable operator models (OOMs), which (together with a number of closely related models like predictive state representations and multiplicity automata) offer fast constructive and asymptotically correct learning algorithms for a model class that properly include HMMs. Specifically, this article focusses on nonstationary processes. Technical contributions: 1. proofs that the standard model estimation techniques generalize immediately from stationary to nonstationary data (Section 3). 2. An algorithm to reconstruct (an estimate of) the asymptotic stationary process dynamics from initial nonstationary data sequences (Section 4). 3. A method, called "binless learning" or "binless OOMs" to estimate from continuous-valued initial nonstationary training sequences a novel kind of descriptive model of asymptotic characteristics of the process' distribution (Section 5). 4. Brief demos of the methods from Section 5 on synthetic data and data derived from simulations of an analytical model of a protein state switching dynamics. In these demos the objective is to estimate asymptotic process properties from training data that start from (very) out-of-equilibrium conditions. These demos reveal a much better compensation for the initial nonstationarity of training data than more traditional HMM models.
Novelty and relevance  The analytical results presented in Section 3 are not new (see my more detailed comments below), though they probably have not yet been presented in exactly this form. I believe that the authors, who apparently come from a modeling community in the biosciences, simply are not aware that it is commonplace wisdom in the OOM field that OOM models of nonstationary processes can be estimated in the same way as stationary ones (as long as the nonstationarity can be described as the effect of a nonequilibrium starting state; stronger forms of nonstationarity where the very observable operators vary with time are not considered in this paper, and have only superficially been investigated by OOM researchers elsewhere). The core contribution of Section 4, an algorithm to recover the stationary state from an OOM estimated from nonstationary data, is novel (but I think it might be unnecessarily complicated, see my detailed comment below). The "binless learning" of "binless OOMs" in Section 5 is entirely novel and quite ingeneous in my view - for me the absolute bummer part in this paper. Judging from the first of the three demos in Section 6, truly amazing inferences can be drawn about asymptotic properties of a stochastic process from highly nonstationary initial-condition-plagued training data (while the remaining two demos I couldn't understand because they require previous knowledge about techniques and terminology from statistical chemistry (?) that is not explained and that I can't summon). Although I am not familiar with the kind of bio/chemistry microphysics applications that the authors use for display, it would think that these binless OOMs constitute an truly impressive and important step of progress.  Technical correctness  Apart from some minor lapses, mostly notational typos (pointed out below) - fine.   Presentation  Very good English, crisp command of mathematical formalism. I find that the Introduction over-emphasizes the challenges arising from nonequilibrium data -- these are really solved and easy issues as long as the nonstationarity can be captured by non-equilibrium starting state (the case considered in this article). In contrast, the Intro under-sells in my view the novel modeling opportunities afforded by binless OOM learning.   Summary assessment and recommendation  A very interesting and innovative contribution -- namely, in Section 5, concerning "binless OOMs". The novelty of Sections 3-4 is marginal, the authors are probably not aware of the existing common understanding of the matters discussed in these Sections (the novel algorithm in Appendix A.4 excepted). If this were a journal paper submission I would recommend a substantial revision with a much deflated treatment of the Section 3-4 material and a much more prominent role of the binless idea. The last two demos are useless because they are unintelligible due to compression - in a revision they would need to be expanded.      ********** more detailed comments **********   Lines 28 ff: the article presents the estimation of OOMs from nonstationary data as something that has been rarely done before ("a major challenge … In most literature the observation data are assumed to be equilibrium…"). That is not true. Even in the earliest OOM publications (H. Jaeger(1998): Observable operator models of stochastic processes: a tutorial. GMD Report 42, German National Research Center for Information Technology 1998) it was explicitly recognized that the standard OOM estimation procedure works for stationary and nonstationary data alike. I dare say that this is and always has been obvious to researchers active in this field. In the recent review paper of Thon (Thon, M., Jaeger, H. (2015): Links Between Multiplicity Automata, Observable Operator Models and Predictive State Representations -- a Unified Learning Framework. Journal of Machine Learning Research 16, 103-147) the assumption of stationarity is not even introduced. In other words, Theorem 1 cannot be considered new. Only the generalization 1 (line 129) adds a new aspect to the known picture.    Section 4: the initial text paragraph confuses me. It starts by saying that "the only remaining problem for learning OOMs from nonequilibrium data is how to estimate initial state vectors". But then the text proceeds to state that this problem is uninteresting and instead one should try to find the equilibrium state. I am further confused by the fact that in line 149 it is not explained what \hat{omega} is intended to be - is it intended to be the equilibrium state? then it should be written \hat{\bar{\omega}}. -- On a side note, it is confusing that the index _{eq} is used to denote an 'equivalent' OOM, and the \bar to denote an equilibrium. Both words start with 'eq' and in fact I lost one hour getting myself un-trapped from this duplicity. Please consider changing the notation.   Lines 148 ff: I don't understand why the equilibrium state \hat{\omega} (or maybe in better notation \hat{\bar{\omega}}) isn't simply calculated as an eigenvector of \hat{Xi}(\mathcal{O}) of largest absolute eigenvalue. Furthermore, some thought should be given to processes that have more than one ergodic component. Then there is no uniquely defined equilibrium state.   Section 5: the choice of terminology "binless learning" or "binless OOM" should be explained/motivated. I never heard of this concept before and it is fun to see what Google queries try to retrieve (zero hits on these key phrase, alternative suggestions: "boneless ham", "boneless BBQ"…).   The cover story for motivation the material from Section 5 should be phrased more carefully. Text in the Introduction suggests an alternative to parametric / Hilbert space embedding OOMs, but Section 5 only offers a model of asymptotic characteristics of a process, not a model of the process, so the juxtaposition to those continuous-valued OOM techniques is misleading. Introduction should be rephrased carefully.    It took me a little effort to understand the mathematical type of the object defined in Eqn (19). It would help the reader to explicitly state that this is a (point) probability measure on \mathcal{O} \times \mathbb{R}^D_1 \times \mathbb{R}^D_2.    Line 173f: "where feature functions can be selected as splines, radial basis functions or other …": this needs more explanation. How can e.g.  a spline function, applied to L points, give a real number as a feature value?  Simulation studies reported in Figs 3 and 4: impossible to understand in this super-condensed presentation. Readers are required to know what TIC analysis is and what w_{TICA} is. The NIPS readership will be lost. Please invest an explanatory paragraph.  ********* minor comments / edits *********  line 90, first formal expression, counter index should range over n not s  Figure 1: indices in circles should read t-L etc, not t-s  Assumption 2 page 4: please specify the range of the f_t functions. I assume it's \mathbb{R}? (or re-use notation \phi from section 3.1)  It would be helpful for the reader to state at some early point that nonstationarity of a rank-m process represented by an m-dimensional OOM is equivalent to \omega \neq  \omega_{eq} \Xi(\mathcal{O}).   line 149 typo "quadrantic"  line 192: principle -> principal  line 198: "for general g with" incomplete or "with" superfluous


Maximization of Approximately Submodular Functions
We study the problem of maximizing a function that is approximately submodular under a cardinality constraint. Approximate submodularity implicitly appears in a wide range of applications as in many cases errors in evaluation of a submodular function break submodularity. Say that F is ε-approximately submodular if there exists a submodular function f such that (1−ε)f (S) ≤ F (S) ≤ (1+ε)f (S) for all subsets S. We are interested in characterizing the query-complexity of maximizing F subject to a cardinality constraint k as a function of the error level ε > 0. We provide both lower and upper bounds: for ε > n −1/2 we show an exponential query-complexity lower bound. In contrast, when ε < 1/k or under a stronger bounded curvature assumption, we give constant approximation algorithms.

This paper considers the problem of maximizing a set function, f, that is an eps approximation to a monotone submodular function, that is there exists a monotone submodular function F such that (1 – eps) f(S) \leq F(S) \leq (1 + eps) f(S) for all sets S, under a cardinality constraint given access to an oracle for evaluating f. While it has been known that for any constant eps and exponential number of calls to the oracle are needed to obtain a constant approximation when F is submodular, this paper furthers the study of this question by providing several new and interesting facts.  • An exponential number of queries are needed to obtain better than O(1/n^beta) when eps \geq 1/(n^(1/2 – beta).  • An exponential number of queries are needed to obtain better than O(1/n^beta) when eps \geq 1/(n^(1/3 – beta) in the special case where F is a coverage function.  • Greedy achieves 1 – 1/eps – O(delta) approx. when eps = delta/ k and there is a cardinality constraint of k. Moreover, they show this is essentially tight characterization of greedy.  • A (1-c)(1-eps)^2/(1+eps)^2 algorithm when F has c curvature.
This paper furthers the study of a natural question in optimization and machine learning, namely that of a maximizing a function that is approximately monotone submodular under a cardinality constraint. They strengthen previous lower bounds providing dimension dependent bounds on the amount of approximation until which an exponential number of queries. Moreover, they provide improved upper bounds on the quality of approximation achieved as dependent upon k and the curvature of F.  While the question studied by this paper is natural and possibly important and the writing is clear, it is less clear to what degree the results are novel and required new insight over previous work.  There is little discussion of how hard it what to achieve the lower bound or extend previous lower bound work to achieve their bounds. In other words, the paper would benefit from more clearly illustrating whether they provide a novel analysis or lower bound technique, versus simply applying previous machinery. Moreover, the upper bound they achieve for greedy feels somewhat natural and ultimately it is unclear if the paper actually proposes a new algorithm for performing approximate submodular maximization. Finally, it appears that there is a gap between the lower bounds and the upper bounds they provide again, making it difficult to ascertain whether or not the paper simply follows the natural analysis of previous work or provides fundamental new insights.  That said, the question of closing this gap is tantalizing, ultimately resulting in this borderline review.  Detailed Comments:  - Lines 8-10: are the inequalities on epsilon in the wrong direction? - Line 60  - 63: although it was said that F is assumed to be monotone, here I would be clear about whether there result applied to monotone submodular functions. - Lines 90-96: a little more discussion of best known running times under the assumption would be beneficial for comparison. - Line 145: say a little more about how this lower bound compares to previous lower bounds and the techniques they use. - Line 150: In the line after “an general” should be “a general” - Lines 273 – 281: say more about what the algorithm actually is and how compares to other algorithms. - General: discuss the gaps in upper bounds and lower bounds left open by this work.


Sampling for Bayesian Program Learning
Towards learning programs from data, we introduce the problem of sampling programs from posterior distributions conditioned on that data. Within this setting, we propose an algorithm that uses a symbolic solver to efficiently sample programs. The proposal combines constraint-based program synthesis with sampling via random parity constraints. We give theoretical guarantees on how well the samples approximate the true posterior, and have empirical results showing the algorithm is efficient in practice, evaluating our approach on 22 program learning problems in the domains of text editing and computer-aided programming.

This paper describes an algorithm for sampling from the set of programs consistent with a given set of input-output training examples.  The programs come from a finite set of possible programs defined by a "sketch" (which seems to be essentially a grammar), with a description length prior.  The paper builds on the existing approach of program synthesis by constraint solving, where the possible programs are encoded as assignments to a set of Boolean variables, and a SAT solver is used to find programs consistent with the examples.  The advance in this paper is that rather than just finding a single most likely solution, the algorithm samples from the posterior.  The paper provides an error bound on the sampling probabilities that allows trading off time versus accuracy.
I found this paper interesting and well-written, but I have some significant questions and comments about the approach.   * Figure 5 (page 6) shows that the existing MDL approach of finding a single most likely program does slightly better than the proposed sampling approach in terms of average accuracy.  The paper argues that sampling is useful because we can find the C most frequently sampled programs and show them to a user.  As shown in Figure 6, there is more likely to be a correct program in the top 3 programs than in the top 1.  But if we want to show the top C programs, do we really need to perform sampling, which the paper says is complicated by the existence of many long and unlikely programs that match the training examples?  Why can't we simply find the MDL program and then run the solver again with length restrictions to find other consistent programs of the same length, or slightly longer lengths?  * Page 5, line 179 says that the paper "implicitly defined a description-length prior, where |x| is the number of bits required to specify x in the SAT encoding".  Looking at the constraints in Figure 2(c), it appears that if bits x_1, ..., x_k fully define a program, then bits x_{k+1}, ..., x_n end up unconstrained.  So is a program of length k represented by 2^{n-k} distinct assignments to the bits?  Is that what's meant by an implicitly defined prior?  If so, then it seems that the prior is uniform over bit assignments (although not over programs), and I don't see the point of mapping from a uniform distribution on a separate embedding space as discussed in Section 2.  * For string manipulation, the paper uses data sets of size 5, with 1-3 examples used for training.  This leaves only 2-4 examples for testing, which doesn't seem like enough to yield reliable results.  For list manipulation, I couldn't find a statement about the size of the test sets.  I assume correct programs for these example problems are known, so it should be straightforward to generate tests sets of at least size 100.  Using larger test sets would reduce the chance that some of the returned programs yield high test accuracy by accident.  == Update After Author Feedback == I appreciate the authors' responses to my comments.  I think I did misinterpret Figure 6; I understand now that it is referring to the top C most frequent combinations of program outputs, not the top C most frequent programs.  Still, it would be informative to compare explicitly to a K-best MDL approach, to demonstrate that integrating out the program really helps.  I'm still wondering about how the prior distribution over programs ends up being defined.  I asked in my review if a program of length k ended up being represented by 2^{n-k} distinct assignments to the bits in the SAT encoding, and the response seemed to agree that this was true.  But then why do we need to add auxiliary variables in the embedding space as described on page 4?  Isn't it enough to sampling from a uniform distribution over bit assignments?  If we start with a space where shorter programs already account for more assignments, and then we also add the auxiliary variables, don't we end up with a *squared* MDL prior?  I hope this can be clarified in the final version.