Human-in-the-Loop Interpretability Prior
We often desire our models to be interpretable as well as accurate. Prior work on optimizing models for interpretability has relied on easy-to-quantify proxies for interpretability, such as sparsity or the number of operations required. In this work, we optimize for interpretability by directly including humans in the optimization loop. We develop an algorithm that minimizes the number of user studies to find models that are both predictive and interpretable and demonstrate our approach on several data sets. Our human subjects results show trends towards different proxy notions of interpretability on different datasets, which suggests that different proxies are preferred on different tasks.

Overview: This paper introduces a bayesian framework for incorporating human feedback into the construction of interpretable models.  The basic idea is to define a prior distribution over models that reflects the interpretability of a model M.  In this paper, the (expected) time for a user to predict the label of points assigned by model M is used.  Evaluating the prior is expensive; each call to p(M) requires a user study measuring HIS(x, M) for a sample of x values.  To reduce expense, the authors propose to only query models that have a sufficiently high likelihood (high accuracy) given the observed data.  The accompanying evaluation study shows that common proxies for interpretability (e.g., things like tree size) disagree among one another.  Furthermore, which proxy is best aligned with the human-assessed interpretability prior differs across use cases (data sets).  Comments:   Overall I found this to be a very interesting paper that’s largely very well written.  The idea of using user studies to elicit the relevant notion of “interpretability” is an interesting one.  Furthermore, it opens the door to follow-up studies that are specialized to specific downstream tasks.  (This sort of future direction could be mentioned in the discussion).  I did get a bit lost in sections of the experimental results.  I ask that the authors clarify the evaluation presented in Section 6.  More specific comments are provided below.    - In the introduction you write that interpretability depends on the “subjective experience of users.”  More than that, it depends on the purpose for which an explanation is being desired.  Assessing whether a model is fit-for-purpose would entail defining a specific task, which as you state is not something you do in this paper.  Nevertheless I think it’s an important part of framing the general problem.   - 3.3 prior for arbitrary models:  I am not convinced by the idea that the interpretability of local proxies reflect the interpretability of the underlying arbitrary non-interpretable model.  Have you considered framing this part of the analysis by saying there are two components: the base model (M) and the explanation model (e.g., local-proxy(M, x)).  In this setup one could search over both explanation models and base models.  Plausibly, an explanation model that works well for some M might work poorly for others.    - 4.3, Appendix B: Similarity kernel.  It seems like the similarity kernel is being defined in a way that two models that differ significantly in terms of their interpretability may nevertheless appear similar.  Consider the setting where our model class is regression models.  Correct me if I am wrong here, but I believe the proposed strategy for this class would amount to an RBF kernel between the estimated coefficient vectors.  This would say that a sparse model with coefficient vector (w_1, …, w_k, 0, …, 0) is similar to (w_1, …, w_k, e_1, …, e_m) where the e_j are small coefficients.  The same model could be deemed less similar to (w1, …, w_{k-1}, 0, w_k, 0, …, 0), depending on the choice of w_k.    Yet from a structural and interpretability standpoint, the latter model, being sparse and just weighting a different coefficient, seems more similar than the first (which might have a large number of small non-zero entries).  I suppose you can argue that if the e_j are all really small then they don’t really affect the model predictions much.  But I’m still not sure that the similarity kernels are capturing quite the right type of similarity.    - Section 6: I’m not sure I correctly understand lines 222 - 229.  What is meant by “parameter setting”?  Are these the parameters of your similarity kernel?  Is the following interpretation correct: You have 500 models, each of which is being assessed by one of k proxies (proxy measures of interpretability, such as sparsity).  To get an x for Figure 2(a) you look at the best model for each of the k proxies, and look at where it’s ranked by each of the remaining (applicable) k-1 proxies.  In the worst case, the best model for proxy A could be the worst (rank 500) model for proxy B.  Is this right?  - Section 6: Computing risk proxy on small sample.  Again, I’m not sure I correctly understand lines 233 - 241.  How are you defining “the right proxy” here?  Could you please further clarify the setup for this evaluation?  - Appendix Figure 5: The y-axis scale is too large.  No values exceed 20, but the axis goes up to 40.  - There are a handful of typos throughout the paper.  (E.g., line 254-255 “features non-zero features”)  Unfortunately I didn’t note all of them down as I was reading.  Please give the manuscript a couple of editing passes to weed these out.


Regret Bounds for Online Portfolio Selection with a Cardinality Constraint
Online portfolio selection is a sequential decision-making problem in which a learner repetitively selects a portfolio over a set of assets, aiming to maximize long-term return. In this paper, we study the problem with the cardinality constraint that the number of assets in a portfolio is restricted to be at most k, and consider two scenarios: (i) in the full-feedback setting, the learner can observe price relatives (rates of return to cost) for all assets, and (ii) in the bandit-feedback setting, the learner can observe price relatives only for invested assets. We propose efficient algorithms for these scenarios, which achieve sublinear regrets. We also provide regret (statistical) lower bounds for both scenarios which nearly match the upper bounds when k is a constant. In addition, we give a computational lower bound, which implies that no algorithm maintains both computational efficiency, as well as a small regret upper bound.

Summary  The paper studies the online portfolio selection problem under cardinality constraints and provides two algorithms that achieve sublinear regret. One algorithm handles the full information setting and the other algorithm handles the bandit feedback setting. Furthermore, the paper provides lower bounds for both the full information and bandit feedback settings. The approach that both algorithms take is to split the problem into two learning problems. One problem is to learn the optimal combination of assets and the other problem is to learn the optimal portfolio. To learn the optimal combination of assets a version of either the multiplicative weights algorithm (full information) or exp3 (bandit feedback) is used. To learn the optimal portfolio the Follow The Approximation Leader (FTAL) algorithm is used. Finally, the papers provides three experiments in which the two new algorithms and three baseline algorithms are evaluated on one synthetic data set and two real world data sets.   Strengths and weaknesses  The problem that is solved in this paper has nice practical applications. The analysis of the algorithm was elegant and surprisingly simple after splitting the problem into two parts. After the splitting in two parts the analysis almost directly follows from applying standard algorithms with their standard regret bounds. On the downside, since for each combination of assets a version of FTAL is run the proposed algorithm for the full information has exponential runtime in the dimension. However, the authors show that the runtime cannot be improved while preserving the regret bound.   Since the FTAL  is run locally in the sense that each combination of assets gets its own instance of FTAL it would be interesting to see if the dependency on the constants C1, C2, C3, C4, and C5 in the regret bound can also depend on the optimal combination of assets.   I do not understand the reason for choosing these particular baseline algorithms. It seems very restrictive to only consider x_{t,i} \in {0, 1}, as FTAL can consider x_{t,i} \in [0, 1]. Therefore, the conclusion that algorithms 1 and 2 outperform the baseline algorithms is not very surprising. It appears a more informative baseline algorithm would be to use MWU/exp3 with gradient descent or exponential weights with a continuous prior.   The presentation of the material was reasonably clear, although the section on computational complexity is quite dense.    Minor comments - Figures 1-3: the colors of MWU_disc and Algorithm 2 are difficult to discern, perhaps use black instead of purple. - the statement “our algorithms are superior to others” seems a bit strong.  - line 349: the low of large number -> the law of large numbers.   The authors response cleared up some of the confusion regarding the simulations.


Towards Understanding Acceleration Tradeoff between Momentum and Asynchrony in Nonconvex Stochastic Optimization
Asynchronous momentum stochastic gradient descent algorithms (Async-MSGD) have been widely used in distributed machine learning, e.g., training large collaborative filtering systems and deep neural networks. Due to current technical limit, however, establishing convergence properties of Async-MSGD for these highly complicated nonoconvex problems is generally infeasible. Therefore, we propose to analyze the algorithm through a simpler but nontrivial nonconvex problems -streaming PCA. This allows us to make progress toward understanding Aync-MSGD and gaining new insights for more general problems. Specifically, by exploiting the diffusion approximation of stochastic optimization, we establish the asymptotic rate of convergence of Async-MSGD for streaming PCA. Our results indicate a fundamental tradeoff between asynchrony and momentum: To ensure convergence and acceleration through asynchrony, we have to reduce the momentum (compared with Sync-MSGD). To the best of our knowledge, this is the first theoretical attempt on understanding Async-MSGD for distributed nonconvex stochastic optimization. Numerical experiments on both streaming PCA and training deep neural networks are provided to support our findings for Async-MSGD.

This manuscript analyzes the asymptotic  convergence of asynchronous momentum SGD (AMSGD) for streaming PCA. The fundamental claim [line 101 & 239] is that asymptotically, for streaming PCA, the delay tau is allowed to scale as     (1 - mu)^2 / sqrt(eta),  where mu is the step size and mu the momentum parameter.  Major Comments ============== Before we discuss the proof, I think the introduction is somewhat misleading. In line 76, the authors point out previous work all focus on analyzing convergence to a first order optimal solution. The readers can be confused that this paper improved the results of previous work. However, the problems studies in those paper and streaming PCA are different. As the authors also pointed out, streaming PCA is a very special case that all first order optimal solution is either global optimal or strict saddle point, but the previous work you mentioned studied more general or harder problems.  I would suggest you clarity this part clearly.   The proof has a few steps. In high level, the structure of the theoretical analysis should be improved.  Quite a few places of the presentation have inconsistency or undefined symbols.  I will mention them when discussing the proof as below.  1) Show the norm of iterates and the distance between any consecutive iterates are bounded.  2) Rewrite the ascent direction as a sum of three terms: one only involves the expectation of data matrix Sigma (m_k), and other two (beta_k and epsilon_k) contains the sample deviation (Simga_k - Simga).    The notation Simga_k was not defined, though I can infer it's X_k X_k^T. Besides, a small mistake: in line 162, equation (6), eta is a factor for all three terms m, beta and epsilon, but the momentum parameter mu wasn't rescaled.  One question I have is how to preserve the norm of v_k, as no projection step is used. Though we can see it's upper bounded, but it is not straightforward for me to see why it's also lower bounded.  3) Ignore beta and epsilon, proof that {v_k} converges to the principal eigenvector of Sigma. To show that, the authors exploits the rotation under eigenvalue decomposition and reparameterized V to H.  4) Consider the stochastic version of ODE. I'm lost in this part, as many notations are not defined. For example, in Thm 5, line 204, what is Y? In Proposition 6, what is phi? I don't have a good assessment for this part due to limited understanding.  Overall, I think this paper is solid work, but the current version might be preliminary for a publishment.  The theoretical part needs to be clarified to bring a clear and convincing analysis.    Minor Comments =============== Line 24, 30, 63: This manuscript is notation heavy. I understand you want to first state general AMSGD and then focus on streaming PCA. But if you can replace theta by v in the introduction, it will save you one parameter.   Line 32: equation (3) should be (2). (2) is MSGD and (3) is AMSGD Line 171: taht --> that Line 385 (Appendix) The hyperef to one section didn't compile.


Genetic-Gated Networks for Deep Reinforcement Learning
We introduce the Genetic-Gated Networks (G2Ns), simple neural networks that combine a gate vector composed of binary genetic genes in the hidden layer(s) of networks. Our method can take both advantages of gradient-free optimization and gradient-based optimization methods, of which the former is effective for problems with multiple local minima, while the latter can quickly find local minima. In addition, multiple chromosomes can define different models, making it easy to construct multiple models and can be effectively applied to problems that require multiple models. We show that this G2N can be applied to typical reinforcement learning algorithms to achieve a large improvement in sample efficiency and performance.

The authors proposed the Genetic-Gated Networks (G2Ns) which combine conventional gradient based policy optimization and genetic algorithms. The gates are binary masks injected to the conventional policy neural networks, and their values are adapted via genetic algorithms while learning the policies. Specifically, a population of differently configured gene vectors are initially generated, and the elite gene configuration which has the highest fitness score is selected in order to update the conventional neural network parameters via gradient based policy optimization. The gate values are updated via genetic algorithms periodically as well. The G2Ns are applied to the actor policy of Advantage Actor-Critic (G2AC) on Atari 2600 games and Proximal Policy Optimization (G2PPO) on eight MuJoCo environments. The empirical results show that G2AC achieved improved results on 39 out of 50 Atari games and G2PPO achieved improvement on some MuJoCo environments as well.   The G2Ns are general neural network framework which could be applied to either supervised learning or reinforcement learning. The reinforcement learning related aspect of G2N is in the Two-way Synchronous Optimization part. In the second phase of the two-way synchronous optimization, multiple policies with different gene configuration are leveraged to collect training trajectories. The empirical ablation studies on the two-way synchronous optimization in section 4.3 are very helpful in justifying the positive impact of such approach. The multiple policy-based exploration of G2Ns is related to the ideas of exploration via bootstrapped DQN or noisy networks. An in-depth analysis would also be helpful to highlight the uniqueness of G2Ns on exploration in general.   The G2Ns achieve significant improvement on discrete action problems as it is shown that the final performance on 39 out of 50 Atari games are improved. Some sample plots of learning curves on the Atari games would also be helpful in checking that they do not suffer from the poor initial performance issue as reported in the continuous domains. Given the plots in Figure 5 on the four Atari games, it seems that G2AC is outperformed by A2C in the first 10M training, which suggests the poor initial performance issue may be an inherent issue of G2Ns, not specific to the continuous reinforcement learning domains. Any additional material on this matter would be helpful.   The improvement of G2PPO in continuous domains is less significant compared to the G2AC in discrete domains as most of the curves are mixed in Figure 4.   It is curious to know the hyper-parameter sensitivity of G2Ns to the initial activation probability, the crossover probability and the mutation probability.    The paper is not related to multiagent RL at all. The term “multi-agent” is misused.   ----  I have read the reviews and the rebuttal. The rebuttal did not change my assessment of the paper. I would like to keep my score.


Learning latent variable structured prediction models with Gaussian perturbations
The standard margin-based structured prediction commonly uses a maximum loss over all possible structured outputs [26,1,5,25]. The large-margin formulation including latent variables [30,21] not only results in a non-convex formulation but also increases the search space by a factor of the size of the latent space. Recent work [11] has proposed the use of the maximum loss over random structured outputs sampled independently from some proposal distribution, with theoretical guarantees. We extend this work by including latent variables. We study a new family of loss functions under Gaussian perturbations and analyze the effect of the latent space on the generalization bounds. We show that the non-convexity of learning with latent variables originates naturally, as it relates to a tight upper bound of the Gibbs decoder distortion with respect to the latent space. Finally, we provide a formulation using random samples and relaxations that produces a tighter upper bound of the Gibbs decoder distortion up to a statistical accuracy, which enables a polynomial time evaluation of the objective function. We illustrate the method with synthetic experiments and a computer vision application.

UPDATE:  Upon reading the author response, I have decided to leave my review unchanged.  The topic of this work is of interest to the community, and it can make a decent publication. I am still concerned about novelty. While the author response argued that similar increments have been done in past works, I feel that there are some significant differences between the current case and the mentioned works. Accepting the paper is still a reasonable decision in my humble opinion.  Original Review:  The paper develops PAC-Bayesian bounds for margin-based structured prediction, when hidden variables are included in the problem. It mainly relies on the analysis of [1], adapted to the formulation of structured SVM with latent variables given in [2].  The point I found most interesting is the remark about attempts to convexify structured losses studied in the paper. The necessity for a non-convex formulation stems directly from the PAC-Bayesian bound, which is a nice and novel insight.  My main concern regarding this paper is whether it is sufficiently novel. The majority of technical work is an almost direct adaptation of [1], and the paper does not seem to offer a significantly novel approach for the analysis of structured prediction problems. With that said, the problem of latent variables in structured prediction is interesting. This is enough to shift my overall score to 6.  The paper could also gain from a more thorough experimental study. In all the proposed experiments, the latent space is very small and marginalization or enumeration of the hidden states is easy. I think that a problem with a large latent space, where the randomization of this space gives a significant advantage, could strengthen the point of the paper. This would also distinguish the algorithmic contribution of this paper from the algorithms analyzed in [1] and demonstrated in papers on natural language processing.  Overall, I think that while this paper makes nice contributions, it is not clear if they amount to a significantly novel submission.  [1] Honorio, J. and Jaakkola, T. Structured prediction: from Gaussian perturbations to linear-time principled algorithms. UAI 2016 [2] Yu, C. and Joachims, T. Learning structural SVMs with latent variables. ICML 2009


Deep State Space Models for Unconditional Word Generation
Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuances. We show that a non-autoregressive deep state space model with a clear separation of global and local uncertainty can be built from only two ingredients: An independent noise source and a deterministic transition function. Recent advances on flowbased variational inference can be used to train an evidence lower-bound without resorting to annealing, auxiliary losses or similar measures. The result is a highly interpretable generative model on par with comparable auto-regressive models on the task of word generation.

This paper introduces a probabilistic model for unconditional word generation that uses state space models whose distributions are parameterized with deep neural networks. Normalizing flows are used to define flexible distributions both in the generative model and in the inference network. To improve inference the inference networks uses samples from the prior SSM transitions borrowing ideas from importance-weighted autoencoders.  I enjoyed reading this paper, as it gives many useful insights on deep state space models and more in general on probabilistic models for sequential data. Also, it introduces novel ways of parameterizing the inference network by constructing a variational approximation over the noise term rather than the state.  I appreciated the usage of unconditional word generation as a simple example whose results are easier to interpret, but the validation of these results to more complex sequential tasks would make this a much stronger submission:   - I agree on the discussions on the importance of not having the autoregressive component as most of the models in the literature. In this paper however you only focus on the "simple" sequential task of word generation, while in many cases (e.g. audio or video modelling) you have very long sequences and possibly very high dimensional observations. The natural question is then how do these results generalize to more complex tasks? Would you for example still be able to train a model with a higher dimensional state space without the autoregressive component, performing annealing or having auxiliary losses?  - In more complex tasks you often need some memory in the model (e.g. LSTM units) which is why previous works on deep state space models combine RNNs with SSMs. How would you add memory to your model? Would your conclusions still hold in this case?  Also, more in depth analysis of the novel components of your model would have been useful:  - When defining the importance weighted ELBO, you use many different samples from the generative model as an input to the inference network. Previous work used for example just the prior mean in the inference network, instead of taking all the samples which is computationally intensive.  How does the model perform if for example you only pass the prior mean instead of all the samples?  - What happens if you only use normalizing flows in generative model/inference networks, and not in both? This is more commonly done in previous papers.   Minor comments - The results on the set-based metrics and the ones in table 2 should be discussed more in depth - Is your citation style in line with the nips guidelines? Usually nips paper have numbered citations  Typos: Line 6: built Line 281: samples


Community Exploration: From Offline Optimization to Online Learning
We introduce the community exploration problem that has many real-world applications such as online advertising. In the problem, an explorer allocates limited budget to explore communities so as to maximize the number of members he could meet. We provide a systematic study of the community exploration problem, from offline optimization to online learning. For the offline setting where the sizes of communities are known, we prove that the greedy methods for both of non-adaptive exploration and adaptive exploration are optimal. For the online setting where the sizes of communities are not known and need to be learned from the multi-round explorations, we propose an "upper confidence" like algorithm that achieves the logarithmic regret bounds. By combining the feedback from different rounds, we can achieve a constant regret bound.

Summary  In the submission, authors explore a new “community exploration problem”, both in an offline and online setting: An agent choose at each round t \in [K] one community among C_1,…,C_m. Then, a member is uniformly sampled (with replacement) from the chosen community. The goal for the agent is to maximize the overall number of distinct members sampled.  In the offline setting, the agent knows each community size. If the allocation strategy k_1+...+k_m=K has to be given before the beginning of the game (scenario 1), then a greedy non-adaptive strategy is shown to be optimal. If the agent can adapt its allocation on the fly (scenario 2), then a greedy adaptive strategy is shown to be optimal.  In the online setting, community sizes are unknown. Perhaps because tracking overall sets S_1,…,S_m of met members is costly (space & time complexity), authors consider the restricted case where only few parameters can be maintained across rounds. More precisely, they redefined the notion of round: a round is composed of K sub-rounds in which the agent performs a strategy k_1+...+k_m=K (restricted to scenario 1 or 2, depending on the setting), and updates parameters at the end of these K sub-rounds, to then go to the next round, with in memory only these parameters.  Authors provide CUCB-based algorithms, that call offline greedy algorithms, and prove a O(log(T)) bound on regret in the setting described above.  Quality  The submission appears to be technically correct. The offline part is interesting as an operational research problem, and proofs are well-written. I think this part represent the major contribution of the paper.  Clarity  For the online part, I regret that the setting was not clearly explained. We do not know what is allowed for the learning algorithm. I think this is an important step (particularly if one wants to compete with provided solutions) that has been largely neglected. Even if the setting can be considered as “implicitly defined” from the regret definition, no justification are given for the (unnatural) restriction considered (maybe the “complexity” argument I gave above could be a reason, but I don’t find it realistic).  The more natural online setting where collision with previous rounds can be used is only considered in the future work section. It would have been better to see it before, in order to contrast with the restriction considered.  Originality  There is a significant lack of related work. For instance, the setting described in [1] is clearly related and should appear in the paper. The online setting analysis is pretty classic.  Significance  Offline results can be considered as important, the optimality of greedy algorithm being not trivial. Provided application are also relevant.  I am not convinced for the online part. I don’t see a clear and well justified setting definition that support the notion of regret used in the paper.  [1] S. Bubeck, D. Ernst, and A. Garivier. Optimal discovery with probabilistic expert advice: finite time  analysis and macroscopic optimality. Journal of Machine Learning Research, 14:601–623, 2013.


On Markov Chain Gradient Descent *
Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the trajectory of a Markov chain. Existing results of this method assume convex objectives and a reversible Markov chain and thus have their limitations. We establish new non-ergodic convergence under wider step sizes, for nonconvex problems, and for non-reversible finite-state Markov chains. Nonconvexity makes our method applicable to broader problem classes. Non-reversible finite-state Markov chains, on the other hand, can mix substatially faster. To obtain these results, we introduce a new technique that varies the mixing levels of the Markov chains. The reported numerical results validate our contributions.

I'm happy with the authors' feedback, and will hence keep my "accept" decision.  ==============================================================  [Summary] This paper studies Markov chain gradient descent (MCGD) algorithm(s), an extension of stochastic gradient descent (SGD) in which the random samples are taken following a Markov chain. Built upon existing analysis of MCGD in [1,5,6], the paper generalizes the convergence result in various ways, answering a few open questions. In particular, the paper extends the ergodic result to non-ergodic ones, removes necessity for reversibility of the underlying Markov chain, and is able to deal with non-convex settings. The paper also provides a novel analysis based on keeping track of varying levels of mixing time corresponding to progressively better mixing, which may also shed light on future work. The numerical part of this paper is only for illustrative purposes (as the algorithm is not new), but the comparison between MCGD and SGD-T is informative and provides a good start and motivation for a more thorough study of the theoretical aspects of the algorithm.  [Quality, originality and significance] The paper deals with a few challenging open problems in the analysis of MCGD. Previously, all analysis are done for convex objectives and reversible Markov chains. But non-reversible Markov chains can have much better mixing/convergence behavior both in theory and practice [14], and non-convex settings are of great practical importance in e.g. deep learning. The paper elegantly deals with these issues by introducing a new varying mixing time level analysis, without adding artificial assumptions.   [Clarity] The paper is generally very well-written. The authors make notable efforts in motivating the necessity and challenges of the analysis for non-reversible and non-convex settings. In particular, the explanation of the assumptions between lines 117-128 is rather clear and frank. The example between lines 22-28 is also very good and showcase the advantage of MCGD over SGD in such a scenario. Nevertheless, a few things (mostly typos and wording) can still be slightly improved. Firstly, it might be slightly better to also mention the general MCMC in the example between lines 22-28. It may also help the readers to understand if a more explicit explanation of how the examples between lines 29-33 is related to the stochastic programming setting (1) is given. There are a few slight typos and wording issues, the removing of which may improve readability: 1. line 7: wider of -> wider range of;  2. line 19: expansive -> expensive;  3. line 25: If -> Even if; 4. line 29: natural -> naturally; 5. line 91: SG4 -> SGD4; 6. line 123: slower -> lower; 7. line 125: non-differential -> non-differentiable; 8. line 230: for the -> due to. Another suggestion is that the authors can explicitly state which results correspond to ergodic ones and which correspond to non-ergodic ones by e.g. directly stating that “Then we have the non-ergodic convergence” instead of just “Then we have” in line 193. This may help some readers who are not familiar with the definition of ergodicity in the optimization scenario to understand the results  [Some suggestions for improvement] Here I list a few things that I think can be further improved: 1. The example between lines 61-68 does not seem to guarantee a uniform stationary distribution for the Markov chain over the graph, and hence may not be completely consistent with (4). The authors may want to pay some attention to this potential issue. 2. It may help a lot if the authors can provide a non-reversible Markov chain for the numerical experiments (or just prove and state, if the example on page 3 is), and provide non-ergodic convergence curves. This will make the motivating example on page 3 even more inspiring, which will then illustrate the good performance of MCGD beyond ergodic convergence, reversible Markov chains, and convexity, which naturally ask for the new analysis proposed later in this paper. 3. The authors may want to either mention at the end of Section 1 or the beginning of Section 2 that the optimization problem in study for the finite state cases is always in the form of (4).   4. Give at least one sentence of explanation for the significance of obtaining non-ergodic convergence results, either in theory or in practice. Also explain the major hurdle of restricting to reversible Markov chains and convex objectives in the previous works. In particular, the authors may want to highlight the main idea of how varying mixing levels help solve the non-convexity and non-reversibility issue, and how it leads to non-ergodic results. 5. It looks a bit weird to add the reversibility assumption back in Section 5 when continuous state space is involved. The authors may want to make more explanations for this.  In general, this paper is well-written with insights and sufficient motivation, and it addresses a few challenging and open problems.


A Bayesian Approach to Generative Adversarial Imitation Learning
Generative adversarial training for imitation learning has shown promising results on high-dimensional and continuous control tasks. This paradigm is based on reducing the imitation learning problem to the density matching problem, where the agent iteratively refines the policy to match the empirical state-action visitation frequency of the expert demonstration. Although this approach can robustly learn to imitate even with scarce demonstration, one must still address the inherent challenge that collecting trajectory samples in each iteration is a costly operation. To address this issue, we first propose a Bayesian formulation of generative adversarial imitation learning (GAIL), where the imitation policy and the cost function are represented as stochastic neural networks. Then, we show that we can significantly enhance the sample efficiency of GAIL leveraging the predictive density of the cost, on an extensive set of imitation learning tasks with high-dimensional states and actions.

SUMMARY The paper proposes an improvement to the Generative Adversarial Imitation Learning (GAIL) approach. In GAIL, a discriminator (critic) function is updated to provide feedback on how close the distribution induced by the policy is to the expert distribution. The generator (policy) is then updated to make these distributions more similar. These updates happen interatively. The paper proposes to make this procedure more robust by defining a *distribution* over discriminator parameters, from which multiple samples are drawn. The generator then tries to improve wrt all of these samples, rather than a point estimate.   TECHNICAL QUALITY The paper uses mostly experiments (rather than, say, theoretical proofs) to support the claimed contributions. The new method is compared to two relevant baselines - published GAIL and a 'tuned GAIL' that uses the proposed innovations except the use of multiple posterior samples. Experiments are over five different common benchmarks for imitation and reinforcement learning, using as much as possible standardised settings from earlier papers. It is not clear how many trials were performed for each setting - and thus how reliable the averages are. It seems single trials were used in each environment. (but at least the findings were relatively consistent over the five tasks).   The development of the method seems mostly correct, I found the following (relatively minor) issues:   - paper states ML is a surrogate objective for a bayesian approach. It could be regarded as surrogate objective for a MAP estimate, but in a full Bayesian approach p(\phi|...) or p(\theta|...) aren't objectives, but the sought quantity (i.e., one doesn't maximise the posterior). - The main reason for the efficiency of BGAIL seems to be that using multiple samples of the posterior over discriminators (cost functions) seems more robust than using a point estimate. It would be interesting to describe this point in more detail: why is using a point estimate so brittle? Is there an experiment that could be done to get more insight in this point? - The paper implies (23) as another estimate for the quantity in (17), but in fact it is a different objective altogether (I.e. doesn't share LHS with (17)). That could be clearer.  - It seems that BGAIL would be more costly computationally because of the SVGD procedure. (as a tradeoff for a higher sample efficiency). Could you quantify the difference in computation time? - It wasn't clear to me what the +- indicates in table 1. Standard deviation over independent trials (how many)? Or st. dev within a trial? Or standard error, or multiple deviations, etc. Interpreting them as, say, 95 % confidence interval, most differences do not seem statistically significant between tuned GAIL and BGAIL in table 1 (but besides this asymptotic performance, learning seems to be a lot quicker for BGAIL) in the shown run. That makes it hard to judge the impact on asymptotic performance.   RELEVANCE, SIGNIFICANCE, NOVELTY The imitation learning problem is a relevant problem to the machine learning community, as a problem in itself and as a promising way to initialise reinforcement learning policies. The approach seems to be more sample efficient then tuned GAIL (the most competitive baseline) - although the effect depends a lot on the environment, with relatively modest improvements on some of the environments. Asymptotic performance is roughly the same. I would thus judge the work as somewhat significant. (I think the impact could increase if a bit more focused was put on the problem with brittleness and how that is solved by the BGAIL approach). The approach is somewhat novel, although it is still quite similar to the conventional GAIL approach. The derivation it self is interesting, although in some part it isn't clear whether a full Bayesian approach or using MAP estimates / posterior means are meant.   CLARITY The writing is mostly clear and well-structured. At times, there are small grammar issues (e.g. a missing article in "related to variational approach").  At two points, I felt the clarity of the text could be improved: -In 2.1 RL machinery is introduced - maybe it would be better to explain this step: we want to learn a cost function such that the gibbs policy follows the expert distribution as well as possible. For that, we need, Q, V, etc Derivation around the introduction of the 'discrimination events' is a bit meandering. - The discussion in paragraph 3.1 feels a bit convoluted with the introduction of the variables o_A, o_E, values 0_A, 1_E etcetera. It seems that this could perhaps be expressed more concisely using the output of the discriminator (and the true label) as functions, rather than introducing new random variables.    Further, it seems the algorithm is described in sufficient detail to be re-implemented. The experiments are missing some detail to be reproduced or interpreted (e.g. how can the +- in table 1 be interpreted, how many independent training runs were performed for each method). Without knowing the number of independent training runs, it is hard to judge the confidence in obtaining the same result in a reproduction study.   MINOR COMMENTS - Line 127: In general, we can't obtain MLE from posterior - The notation with C as wildcard for {A or E} is a bit confusing. It makes it seems like C is a variable and A and E are values.  - Line 162: "Imitation learning requires experience samples"-> not always, for example in the case of behaviour cloning where the learned policy is never run - There are many types of RBF kernel, please specify which one you used. (line 218)  I have read the rebuttal, and appreciate the extra results as well as the clarification regarding the experimental procedure and the number of policy runs performed with different random seeds. I have updated my score accordingly.


Interactive Structure Learning with Structural Query-by-Committee
In this work, we introduce interactive structure learning, a framework that unifies many different interactive learning tasks. We present a generalization of the queryby-committee active learning algorithm for this setting, and we study its consistency and rate of convergence, both theoretically and empirically, with and without noise.

This paper formulates a framework which unifies several interactive learning problems with a structure such as interactive clustering. Next, the authors show that QBC can be generalized and kernelized to solve the problems in the framework. The consistency and rate of convergence are analyzed.   I do not think I could judge the novelty of the theorems and proofs well. Thus, my comments focus on the practicality of the proposed algorithm, which I believe is relevant to its significance and the impact to the field.   Strength:  I think the direction of applying QBC to interactive learning settings other than active learning is promising. QBC might not be that practical in active learning problems because state-of-the-art models are usually neural network and modern neural networks have huge version space and there are lots of very different and equally good local minimums. It is also not easy to sample from neural networks efficiently (dropout only define the uncertainty on a small number of weights). However, the state-of-the-art clustering methods like GMM are usually very simple and easy to sample.  The writing is also very clear. Reading the paper is like reading a chapter of a textbook.  Weakness (or more like suggestions for future improvements):  Although paper tries to address the practical concerns (handling the noisy cases and only need to sample two parameters at a time), the method still has some practical issues. For example, QBC-based methods are vulnerable to noise. If we could quantify the level of noise at the beginning (e.g., knows the beta in eq. (1)), we can modify QBC to have nice guarantees. However, we usually do not know this information before collecting the data. In fact, most of the time, we do not even know which class of models with what kinds of hyper-parameters will perform very well before collecting enough data. Furthermore, clustering tasks are usually more ambiguous. There might also be many equally good local minimums in complicated clustering problems and models. Different annotators often have their own solutions from different local minimums. This means the noise level is usually high and hard to be predicted. Without knowing beta, I do not know how robust the structure QBC would be. Even if we know the noise level, it is possible that we spend lots of effort searching g* by eliminating lots of equally good local minimums, but we find that g* is not good enough eventually because the model assumption has some flaws or the original model becomes too simple as more samples are collected.  The experiments are only done using toy data and do not compare with other strategies such as uncertainty sampling. Is the algorithm scalable? Is it possible to know the time complexity of the proposed structural QBC when choosing the next example with respect to all the hyper-parameters? Can this study really motivate more people to use QBC in practice?   I understand that this is a theoretical paper, and my educated guess is that the theoretical part of the paper is very solid even though the practicality of the method has not been fully demonstrated. That's the main reason why I vote a clear accept.   Minor suggestions: 1. I can only find the explanation of notation \nu in Algorithm 1. Suggest mentioning the meaning of \nu in the text as well. 2. In Algorithm 2, it does not say how to determine n_t. What does the "appropriate number" mean in line 225? It is hard to find the answer in [30]. 3. Line 201, it should be Gilad-Bachrach et al. [16] investigated ... 4. In the related work section, I recommend the authors to cite some other efforts, especially the ones which try to make QBC based methods more practical and test its performances on multiple real-world datasets (e.g., Huang et al.)  Huang, Tzu-Kuo, et al. "Efficient and parsimonious agnostic active learning." Advances in Neural Information Processing Systems. 2015.  After rebuttal I think all reviewers agree that this is a solid theoretical work, so my judgment remains the same. The authors do not address the practicality issues in the rebuttal which prevents me to increase the overall score to an even higher level.


Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance
Applications of optimal transport have recently gained remarkable attention as a result of the computational advantages of entropic regularization. However, in most situations the Sinkhorn approximation to the Wasserstein distance is replaced by a regularized version that is less accurate but easy to differentiate. In this work we characterize the differential properties of the original Sinkhorn approximation, proving that it enjoys the same smoothness of its regularized version and we explicitly provide an efficient algorithm to compute its gradient. We show that this result benefits both theory and applications: on one hand, high order smoothness confers statistical guarantees to learning with Wasserstein approximations. On the other hand, the gradient formula is used to efficiently solve learning and optimization problems in practice. Promising preliminary experiments complement our analysis.

In this paper the authors advocate the use of the Sinkhorn distance over the "regularized" sinkhorn distance for computing divergence between discrete distributions. They show that the gradient of the former is better and leads to sharper results especially on barycenters. They also provide a close form expression for the gradient of the Sinkhorn distance using the implicit function theorem. Another contribution is a new generalization bound for structured prediction with Wasserstein distance. Numerical experiments are very short be show a better barycenter with the Sinkhorn distance and better reconstruction for images.   The paper is nice and some important results are presented. Still it lacks a few references and the numerical experiments are very limited as discussed more in details below.  + The Sinkhorn distance in equation (7) has already been used (and optimized) in the literature over the regularized version for a few years but it needs to be properly differentiated from the regularized version and the authors did a good job there.   + In [1], a reference cited in the paper, the authors did the gradient computation of the Sinkhorn distance using autodiff tools (gradient propagation along the Sinkhorn iterations). Those approaches are very efficient and come with a small overhead but are not discussed at all in the paper. The Sinkhorn distance has been also derived for discriminant subspace estimation [2]. Even though autodiff was used, the implicit function theorem is used in the supplementary material of [2] to compute the gradient of a bilevel optimization problem with Sinkhorn distance wrt to a linear operator on the samples.   + The discussion about the complexity of the proposed approach is a bit misleading (with vague assumtion about teh necessary number of iteration of Sinkhorn). The complexity of the proposed SInkhorn+gradient computation  is O(n^3) to get a sharp gradient. In other words the authors propose an approach to compute a better gradient that regularized sinkhorn but it comes at the cost of the same complexity of the unregularized OT (that is O(n^3) for solving, the gradoients are the dual potentials also returned by a dual solver). Since one of the important property is that the proposed gradient recover the true unregularized Wasserstein barycenter, why not use the true gradeint with same complexity?  Also  Autodiff is O(Ln^2) where L is the number of sinkhorn iterations so unless there is a true numerical (as in precision) gain to the propose gradient autodiff will be more efficient and needs to be discussed.   + The numerical experiments are interesting but a bit short. Figure 2 is nice  but it should be nice to show the impact of the regularization (in computational time if not on the final result).  Also what is the result when using Autodiff as in [1]?   + The generalization bound for structures learning are very nice. But the numerical experiments are very limited and hard to interpret even as a proof of concept. For instance the accuracy of a classifier on the reconstructed image is used to see if  the reconstruction was accurate. The authors have to be commended for using a measure of performance that is not tied to any of the divergences used . But those performances cannot really be compared or evaluated without the performance of the classifier on the original images. please add this columns to the Table in Fig. 3 so that we have an idea of the loss incurred by the reconstruction.  [1] Genevay, A., Peyré, G. and Cuturi, M., 2017. Learning generative models with sinkhorn divergences. arXiv preprint arXiv:1706.00292.  [2] Flamary, R., Cuturi, M., Courty, N. and Rakotomamonjy, A., 2016. Wasserstein Discriminant Analysis. arXiv preprint arXiv:1608.08063.   Author feedback ============  I want to commend the authors for their detailed and impressive feedback and strongly recommend them to include this material in the final version of the paper if accepted.  The new comparison and discussion of autograd make the paper much stronger. I'm still very surprised by the time for true Wasserstein, it has been known that a modern CPU bound network simplex can solve the OT problem (and give gradient estimates) for n=100 in the order of the ms so a time of 1000s for barycenter estimation  suggests bad implementation.


Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
The design of neural network architectures is an important component for achieving state-of-the-art performance with machine learning systems across a broad array of tasks. Much work has endeavored to design and build architectures automatically through clever construction of a search space paired with simple learning algorithms. Recent progress has demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification tasks. An open question is the degree to which such methods may generalize to new domains. In this work we explore the construction of meta-learning techniques for dense image prediction focused on the tasks of scene parsing, person-part segmentation, and semantic image segmentation. Constructing viable search spaces in this domain is challenging because of the multi-scale representation of visual information and the necessity to operate on high resolution imagery. Based on a survey of techniques in dense image prediction, we construct a recursive search space and demonstrate that even with efficient random search, we can identify architectures that outperform human-invented architectures and achieve state-of-the-art performance on three dense prediction tasks including 82.7% on Cityscapes (street scene parsing), 71.3% on PASCAL-Person-Part (person-part segmentation), and 87.9% on PASCAL VOC 2012 (semantic image segmentation). Additionally, the resulting architecture is more computationally efficient, requiring half the parameters and half the computational cost as previous state of the art systems.

This paper introduced a method to search multi-scale architectures for dense prediction. The authors defined a search space with 81 elementary operations. As training network for dense prediction is slow, they introduced a proxy task that is fast to evaluate and predict the performance in a large-scale training setting. The new architecture is evaluated on three dense prediction tasks: scene parsing, person-part segmentation, and semantic image segmentation.  I have several concerns about this paper: -- The authors did not analyze the correlation between training architecture with 30k iterations and training architecture with 90k iterations. This result is important to confirm the approach. -- As the architecture search is based on a random search, the authors should report the standard deviation in their results. In Figure 3 and 4, the variance of the top 50 architectures seems important. I think it could be interesting to give the results every 2k architectures to show the improvement.  -- The authors did not cite Neural Fabrics [101] and GridNet [102] that learned architecture for dense prediction. -- The authors showed that it is possible to find good architectures for dense prediction but they did not analyze or give insights why the learned architecture is better than hand-crafted architecture. What is missing in the existing architectures?  What I like: + The definition of the proxy task to speed-up the architecture search + The good results on three dense prediction tasks with a more computationally efficient architecture that requires half the parameters and half the computational cost.  Other comments: - The final output of the DPC bloc is the concatenation of the of all branch outputs. Did you explore others strategies? How did you choose the number of filters? - What is the speed-up to cache the feature? - The authors should indicate the batch size.  [101] Saxena S., Verbeek J. Convolutional Neural Fabrics. In Advances in Neural Information Processing Systems, 2016. [102] Fourure D., Emonet R., Fromont E., Muselet D., Tremeau A., Wolf C. Residual Conv-Deconv Grid Network for Semantic Segmentation. In BMVC, 2017.   I have read the author response and I am globally satisfied. I think an analysis to explain what is wrong in hand crafted architectures could significantly improve the quality and the impact of this paper.


HOUDINI: Lifelong Learning as Program Synthesis
We present a neurosymbolic framework for the lifelong learning of algorithmic tasks that mix perception and procedural reasoning. Reusing high-level concepts across domains and learning complex procedures are key challenges in lifelong learning. We show that a program synthesis approach that combines gradient descent with combinatorial search over programs can be a more effective response to these challenges than purely neural methods. Our framework, called HOUDINI, represents neural networks as strongly typed, differentiable functional programs that use symbolic higher-order combinators to compose a library of neural functions. Our learning algorithm consists of: (1) a symbolic program synthesizer that performs a type-directed search over parameterized programs, and decides on the library functions to reuse, and the architectures to combine them, while learning a sequence of tasks; and (2) a neural module that trains these programs using stochastic gradient descent. We evaluate HOUDINI on three benchmarks that combine perception with the algorithmic tasks of counting, summing, and shortest-path computation. Our experiments show that HOUDINI transfers high-level concepts more effectively than traditional transfer learning and progressive neural networks, and that the typed representation of networks significantly accelerates the search.

The authors present an algorithm for transfer learning using a symbolic program synthesizer for finding the most adequate neural network architecture and selecting relevant neural network modules from previous tasks for transfer. The approach is heavily based on concepts from programming languages, but also studies the relevant concept of high-level transfer that is crucial for true lifelong learning. Results show how the algorithm is capable of selectively transferring (high- and low-level) knowledge in a meaningful way, and numerical results validate the significance of the approach.  The authors claim that their method targets the lifelong learning problem, but theirs is really a transfer learning approach. Solving catastrophic forgetting by completely freezing the network parameters precludes the method from being true lifelong learning, in which the learning of subsequent tasks affects the performance of earlier tasks. As a transfer learning method, it is quite interesting in that it selects which parts of the network of a previous task are useful to transfer over to a new task and with which combinator, allowing it to create the relevant transfer architecture via search.  This is apparently the first study of program synthesis for differentiable programs. The idea of using program search for transfer learning is novel, and provides an interesting avenue for future work, although the search methods themselves don't seem novel. The usage of type-based pruning dramatically reduces the search-space, which is the only way for this approach to be applicable in a realistic learning scenario.  The presentation is very complete, with a good amount of details on the proposed language and clear explanations on how these details are useful (and necessary) for learning. However, other details like the specific heuristic used for sorting the program search queue should be provided.  As for the learning algorithm, it seems unjustified to simply compare validation performance as a selection criterion for the network architecture, especially since in a transfer learning context it is many times infeasible to set aside a validation set. Since the approach is presented as solving an optimization problem, using the cost function as the selection criterion seems better.  The experimental setting was good, and discussing (and showing in the appendix) the actual programs learned by the approach was helpful to see the actual capabilities of the algorithm of transferring knowledge. I would have liked to see some randomized experiments where the sequences of tasks were not chosen specifically to highlight the algorithm's strengths, although it was good that many different sequences were used. It is unclear to me what dataset size was used for the actual programs learned by the method showed in the appendix; this should be specified.  I notice a couple of presentation issues. First, Figure 3 is actually a table, and should be presented as such. Second, all the appendices are referenced as Appendix, but it would be far more helpful if the actual appendix number were used. Figure 4 is presented almost 2 pages after it is first discussed, which makes it awkward to look for.  Typos: - Line 101: "Figure 1 gives shows..." - Line 274: "Leanrning a effective..."


Learning Confidence Sets using Support Vector Machines
The goal of confidence-set learning in the binary classification setting [14] is to construct two sets, each with a specific probability guarantee to cover a class. An observation outside the overlap of the two sets is deemed to be from one of the two classes, while the overlap is an ambiguity region which could belong to either class. Instead of plug-in approaches, we propose a support vector classifier to construct confidence sets in a flexible manner. Theoretically, we show that the proposed learner can control the non-coverage rates and minimize the ambiguity with high probability. Efficient algorithms are developed and numerical studies illustrate the effectiveness of the proposed method.

Summary   The paper proposes an SVM-like classification method for estimating  sets containing a pre-specified amount of samples for each class. The overlap of these two sets is a region with ambiguity and should thus be small. The key results are: problem formulation and reformulation using  a convex surrogate loss function. Fisher consistency of this loss,  and an inequality comparing excess risks. Finite sample bounds, a brief description of an algorithm, and experiments on both artificial and real-world data.     Impression  The problem formulation is very interesting and the combination of theoretical and experimental results is above standard. In addition, the paper is easy to follow. My main concerns are:  - What is the conceptional difference between the proposed approach    and classification with reject option as in [2]. It seems that    both are equivalent by differently parametrized. Also, where is    the difference to [13]? - The theoretical analysis in Section 4.2 is certainly not based on   state-of-the-art techniques for analyzing SVMs.  - The description of the algorithm is too short and its computational   requirements are not discussed. - The experimental section should be improved: the type of SVM they   use as a comparison is not explained at all. I guess, it is one    with the hinge loss? If so, this cannot reliably estimate eta, or    at least a monotone transform of parts of it. As a result a hinge    loss SVM is simply not suitable for the considered problem and the    brief hint to [13] in the middle of page 6 is not giving me any   more information how to address this issue. As an alternative one    could have either used a least-squares-like SVM or one using a    convex loss with reject option as in [2]. Similarly, it is not clear,    whether and how the other methods considered in the experiments have    been adapted to the new problem formulation.          here are some thoughts to the authors' reply: 1. there is a difference between the motivation of an approach the the resulting approach.  I do agree that [2] has indeed a different motivation, but I actually would have loved to see a discussion on the resulting approach. For example, what effect have the subtle differences in the Bayes decision function in [2] and the present paper? I admit, however, that my concern/question was not too precise in this direction. 2. Again, I was a bit imprecise, but here is what I meant: analyzing (5) instead of (3) is a  cheap trick, which comes with a price: either the translation requires an overly conservative s (if it works at all), or a data-dependent s. Modern methods (for oracle inequalities instead of generalization error bounds, though) can actually work implicitly with a data-dependent s, although they usually analyze (3) directly. But again, I should have been more precise. 3. Of course deriving the Langrangian is boring, but here is an aspect thats need to be discussed in more detail:  line 171: The above optimization may be solved by any efficient quadratic  programming routine. What are these EFFICIENT routines? Certainly not the ones used by the authors as their timing  indicate. This hand-waiving argument really goes back to the nineties.  4 a) naive SVM is first mentioned on page 7 without any explanation there  or later. This should be updated on page 6 b) "overall message" is a bit too much considering the few experiments. c) see 1, though I do agree that adjusting d is a problem Overall, I happy to raise my score by one based on the feedback


Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior
Bayesian optimization usually assumes that a Bayesian prior is given. However, the strong theoretical guarantees in Bayesian optimization are often regrettably compromised in practice because of unknown parameters in the prior. In this paper, we adopt a variant of empirical Bayes and show that, by estimating the Gaussian process prior from offline data sampled from the same prior and constructing unbiased estimators of the posterior, variants of both GP-UCB and probability of improvement achieve a near-zero regret bound, which decreases to a constant proportional to the observational noise as the number of offline data and the number of online evaluations increase. Empirically, we have verified our approach on challenging simulated robotic problems featuring task and motion planning.

Detailed Comments:  Summary of the main ideas: The authors propose a method based in a variant of Empirical Bayes for GP-based-BO to provide point estimates for the mean and kernel of the GPs. This method, which is essentially a sort of transfer learning method as the authors state, has the restriction that it needs a dataset of points to work. Authors provide a theoretical analysis of a regret bound by using their method and the GP-UCB and the PI acquisition functions. The paper is more formally written that other papers in the field which is a relief and it is, in my humble opinion, very necessary. It also provides a theoretical analysis, which is also rare in the field, and also necessary.  Related to:  Kim, Beomjoon, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. "Learning to guide task and motion planning using score-space representation." Robotics and Automation (ICRA), 2017 IEEE International Conference on. IEEE, 2017. BOX algorithm that is the base of the methods that this paper suggest to provide estimates of mean and covariance matrices over a discrete domain. (This paper generalizes this for a continuous domain).  Robbins, Herbert. "An empirical Bayes approach to statistics." Herbert Robbins Selected Papers. Springer, New York, NY, 1985. 41-47. The empirical Bayes methodology that suggest fixed values instead of a distribution to sample hyperparameters. They adopt a variant of empirical Bayes in this paper.  GP-UCB and PI acquisition functions, along with an easy to read tutorial of BO, that this paper analyze are described in Brochu, Eric, Vlad M. Cora, and Nando De Freitas. "A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning." arXiv preprint arXiv:1012.2599 (2010).  Strengths:  Theoretical and empirical analysis of the method. Bayesian statistics applied in a way that it has not been applied before for BO priors. Very formally written. Very well organized.  Weaknesses:  Difficult for non BO experts. (This weakness can be overcome by the suggestions that I provide in this review).   Does this submission add value to the NIPS community? : Yes it does. I think that I have only read one or two papers in the BO field that provide such an excellent theoretical and empirical analysis. Moreover, theoretical results are critical in BO, and it does not only study the GP-UCB AF, which has been theoretically analyzed before, but also PI. The method is sound and based in solid bayesian statistics. It frees the BO user for having to select the kernel, which is a heavy burden sometimes and something ironical in a sense. If BO is used for AutoML it is ironical that it has to be auto-tuned as well. Personally, I think that these applications of bayesian statistics and theoretical analysis of BO should be more studied. But the key fact of the paper is that it is not only theoretical but it also provide a good empirical analysis. For all these reasons, I strongly recommend this paper and state that it does add value to the NIPS community.  Quality: Is this submission technically sound?: Yes, this way of estimating the mean and kernel has not been done before. Are claims well supported by theoretical analysis or experimental results?: Yes, it is the main strength of the paper. Is this a complete piece of work or work in progress?: Complete piece of work. Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?:  Clarity: Is the submission clearly written?: It has some issues for non experts. The paper assumes strong knowledge of BO and bayesian statistics. This may be a burder for non experts. I suggest adding some images, for example. An image of an estimation of the mean in 1D given some data, or a flow diagram with the offline and online phases. This would add some friendly style that lots of readers of this conference would appreciate. Also, all the variables must be explained, for example, in line 120, the different indexes i and j are not explained, this may be confusing for some readers. Special cases of the PI and GP-UCB are not described and their equations are not given numbers. These expressions are so easy to explain and so necessary for not experts that if they are not explained they may abandon to read the paper further. It would be a great pity.  On the other hand, proofs of the papers add clarity to some settings as the hyperparameter setting of the GP-UCB that are great and are not also common in BO papers.  Algorithm for Meta-BO is very clear and helpful.  Is it well organized?: Yes it is. It is very well organized. All sections introduce the content of posterior ones. Algorithms are brillianty referenced. One thing that needs to be added is a conclusions section. Does it adequately inform the reader?: If clarity issues that I have stated in Question 1 of Clarity are solved. Then, it does adequately inform the reader.  Originality: Are the tasks or methods new?: There are extensions of well-known techniques (BOX and Empirical Bayes). Is the work a novel combination of well-known techniques?: Yes it also is. Because it combines the extensions. Is it clear how this work differs from previous contributions?: Yes it is. It is different to transfer learning BO techniques and usual BO initialization of the mean and variance. Is related work adequately cited?: Yes it is. Lots of references are provided.  Significance: Are the results important?: Yes they are. This methodology is different from previous ones and seeing theoretical and empirical results of it is a guarantee of the success. Are others likely to use the ideas or build on them?: The paper is a bit hard to read, if the style is frendlier, I am sure that it is going to be sused. Does the submission address a difficult task in a better way than previous work?: Yes, in a sense. Using empirical Bayes and leaving free the kernel is a good idea. Does it advance the state of the art in a demonstrable way?: Yes it does, it demonstrates from an empirical and theoretical way. Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?: Yes, proofs in the supplementary material complement the exposition of the manuscript and provide an unique theoretical approach.  Arguments for acceptance:  The theoretical and empirical analysis for a BO method that this paper performs is really unusual in BO. Moreover, the paper has coherence, the methodology is sound and solid and it advances the state of the art. I strongly suggest this paper for acceptance.  This paper is also written with an excellent style, rarely found in computer science papers. I had a hard time to find typos, everything is written very carefully. Authors have a really strong knowledge of the field.   Arguments against acceptance:  If clarity issues are not solved, the paper is difficult to read. The style is great but, again, the paper is difficult to read and understand. Variables and equations are not always explained. I suggest to provide more intuitive explanations.  Typos:  Line 17, provide here the Rasmussen reference. [37]. Line 82, instead of writing k, write $k$.  More detailed comments and suggestions:  Congratulations to the authors, please make it easier to read for newcomers and this paper will be a success.


Communication Efficient Parallel Algorithms for Optimization on Manifolds
The last decade has witnessed an explosion in the development of models, theory and computational algorithms for "big data" analysis. In particular, distributed computing has served as a natural and dominating paradigm for statistical inference. However, the existing literature on parallel inference almost exclusively focuses on Euclidean data and parameters. While this assumption is valid for many applications, it is increasingly more common to encounter problems where the data or the parameters lie on a non-Euclidean space, like a manifold for example. Our work aims to fill a critical gap in the literature by generalizing parallel inference algorithms to optimization on manifolds. We show that our proposed algorithm is both communication efficient and carries theoretical convergence guarantees. In addition, we demonstrate the performance of our algorithm to the estimation of Fréchet means on simulated spherical data and the low-rank matrix completion problem over Grassmann manifolds applied to the Netflix prize data set.

The rebuttal letter of the authors carefully considered my remarks, committed to fix some of the issues, while deferred some more substantial work to the future. I am pleased to accept the manuscript after having read the perceptive rebuttal letter, which addressed most of my considerations. To me, the most fundamental issue is that some of the assumptions made in the paper don't hold in contemporary distributed computing systems, but I am open to accept the paper in terms of its theoretical contribution.  This paper builds upon and extends the iterative local estimation algorithm (ILEA) in two ways. Firstly, this paper allows optimization to take place on a different machine between iterations. Secondly, the more general so-called retraction map is used in place of the exponential map. Ultimately, the goal of the paper is to scale global optimization in a parallel fashion while exploiting the geometry of the parameter space.  The paper aims at addressing the important problem of scalable parallel optimization and takes reasonable steps towards this aim. At the same time, there are several points to consider attentively towards the proclaimed aim: 1.1) It is not entirely clear to me for which parallel architecture the algorithm is designed. Is the goal to use the algorithm for high performance computing or for distributed or cloud computing? On the basis of lines 31-33, it seems to me that the authors have designed the algorithm for low latency distributed frameworks, such as Hadoop or Spark. If this is the case, low latency is not the only design principle for a scalable distributed algorithm. It is also important that the algorithm is designed keeping in mind that it is meant to work on data management layers, such as HDFS and YARN. Thus, the assumption made in lines 63-65 about IID observations across different machines is not aligned with the reality of batch and real-time data workloads provided by YARN. In more statistical terms, online learning problems can't be based on IID distributed data across nodes, and in general the users, developers or engineers don't manage cloud computing at so low-level so as to align data allocation according to such an IID assumption through time. 1.2) Along the lines of the above remark, it is not mentioned in the examples which parallel framework was used for running the proposed algorithm. Did you use an implementation of the message passing interface (MPI) or the actor model in Akka for instance? You may provide more information about these matters in section 5 (simulation study). 1.3) The first example is based on a sphere. Given that the Betti number of a sphere is zero, this example does not provide a geometrically challenging parameter space. It would be useful to use a more challenging manifold. 1.4) The algorithm is also ran on Netflix data. The issue here is that there is no absolute RMSE to compare the proposed algorithm to other established optimization methods. The landmark paper on optimization 'Lipschitzian optimization without the Lipschitz constant' by Jones, Perttunen and Stuckman, journal of optimization theory and application, 1993, provides nine test functions for benchmarking optimization algorithms, see Table 1 and equation (12) in Jones' paper. To get an idea of how well your proposed algorithm works, I suggest you run it on these benchmark functions and you tabulate the results along with the analogous results for other standard optimization methods. 1.5) You mention in lines 35-36 that your goal is to address challenges arising from inference on big non-Euclidean data or data with non-Euclidean parameters. Since you define the manifold M in the parameter space in lines 104-105, my understanding is that the paper addresses only the latter case (data with non-Euclidean parameters), so I would suggest you drop the reference to the former case (big non-Euclidean data) from lines 35-36.  Some minor fixes are proposed below: 2.1) Line 18: Change 'include diffusions matrices' to 'include diffusion matrices'. 2.2) Line 30: A useful paper missing from the list of relevant references you provide in line 30 is the following, 'Merging MCMC subposteriors through Gaussian-process approximations' by Nemeth and Sherlock. 2.3) Line 66: It is mentioned that calligraphic D refers to the parameter dimensionality. If not mistaken, calligraphic D refers to data rather than parameters in your paper? 2.4) Lines 71-72: You may want to rephrase this sentence, there seems to be unintended repetition of the point about communication. 2.5) Lines 104-105: The symbols d and M appear in lines 104-105 for the first time if not mistaken. State more explicitly here that d is the dimension of the parameter space and that the manifold M lives in the parameter space. It can be deduced by reading carefully, yet it would help the reader to state this more directly. 2.6) Line 117: The notation d_{R} has not been introduced. It can be guessed that d_{R} is a metric given that you refer to the triangular inequality, yet it would be helpful to introduce notation more explicitly. 2.7) Line 126: Change 'the addictive constant' to 'the additive constant'. 2.8) Lines 134-135: It is not clear to me what 'population risk' means. Do you mean 'global risk'? 2.9) From line 150 onwards, you use three vertical lines instead of two; do the three vertical lines refer to a norm different than the norm represented by two vertical lines? If this is not a typo, introduce the relevant notation (unless of course you have already explain this in the paper and it eluded me). 2.10) In equation (3), you introduce the symbol 'W'. Does W_{ij} denote a binary variable indicating the presence or absence of an edge between the i-th movie and j-th user? Introduce the notation W_{ij} and its possible relation with the symbol w_k appearing right before line 203.


The Sample Complexity of Semi-Supervised Learning with Nonparametric Mixture Models
We study the sample complexity of semi-supervised learning (SSL) and introduce new assumptions based on the mismatch between a mixture model learned from unlabeled data and the true mixture model induced by the (unknown) class conditional distributions. Under these assumptions, we establish an Ω(K log K) labeled sample complexity bound without imposing parametric assumptions, where K is the number of classes. Our results suggest that even in nonparametric settings it is possible to learn a near-optimal classifier using only a few labeled samples. Unlike previous theoretical work which focuses on binary classification, we consider general multiclass classification (K > 2), which requires solving a difficult permutation learning problem. This permutation defines a classifier whose classification error is controlled by the Wasserstein distance between mixing measures, and we provide finite-sample results characterizing the behaviour of the excess risk of this classifier. Finally, we describe three algorithms for computing these estimators based on a connection to bipartite graph matching, and perform experiments to illustrate the superiority of the MLE over the majority vote estimator.

# Summary The authors try to derive the sample complexity of semi-supervised learning with mixture models, when the unsupervised mixture model may not (exactly) match the mixture model implied by the class conditional distributions of the underlying true distribution. This leads to a much more general analysis than earlier work in SSL, both considering misspecification of the mixture and more than 2 classes. They propose several methods to recover the true mapping of decision regions to classes, for which they show both the sample complexity and show empirical results of the probability of correct recovery in three example simulations.  # Main comments I would like to thank the authors for this interesting work that further advances our understanding of sample complexities when using mixture model in a semi-supervised setting. While I like the analysis presented in the work, I have some questions, concerns and suggestions whose resolution might help me appreciate the work more.  First off, I would suggest reflecting the focus on mixture models in the title, to make it clearer what to expect.  Secondly, I have some confusion over the quantify of interest in the analysis. The goal is recovering pi^*. This is perfectly clear in the Lambda=Lambda^* setting, as was also considered in earlier work. However, it is never made precise in the paper (or I missed it) what pi^* represents if Lambda!=Lambda^*. Is it simply the best possible mapping for that particular Lambda (so like (4) but for the expected log likelihood)? Aside from this, how crucial is the restriction that the mixture contains the same number of components as the number of classes? Would it not be enough for the number of components to be bigger than K, or is this not a problem because the components themselves can be arbitrarily complex?  Third, I wonder if you could comment on the following. It seems that up except for section 4.4, the analysis does not really depend on being in a semi-supervised setting: regardless of where the mixture model comes from, your results that you can find a mapping from components to classes efficiently continues to hold. In this section, however, the rate in terms of the number of unlabeled samples is rather vague (but still more insightful than papers that only consider clairvoyance). Is there more that can be said about this rate. And for reasonable discrepancies between Lambda and Lambda^* (what are reasonable values for W_1), is it reasonable to assume that the bound in Theorem 4.7 becomes informative?  Fourth, I have three questions regarding the experiments. 1. What do the curves look like in terms of classification performance (perhaps these could be added to the appendix, including a supervised baseline). 2. Some of the curves seem to flatten out even though the bounds would suggest that they correct mapping should be obtained with high probability with some number n of samples. Could you explain the relationship between the results in the bounds and the curves (is n simply very large in these cases?).  Finally, the paper has a rather abrupt ending. I understand there are space constraints, but it would be nice if the paper could end with something like a concluding sentence.  In summary, I find the pros of the paper that it: offers an interesting extension to earlier analyses of mixture models in ssl; is clearly written; has a proper analysis and explanation of the differences between the proposed approaches. I find the cons that: as in earlier work, the focus is mostly on the labeled part of ssl, while the unlabeled part (section 4.4) is less extensive. I am looking forward to the author's response to my questions to better inform my opinion of the paper.  # Minor suggestions 15: MLE is not yet defined here 18: "more and more expensive": I'm not sure labeling data has gotten more expensive, labeling all the data might have gotten more expensive because there is more of it. 85: from this definition of M_K it is unclear to me why its elements also contain the lambda_k's, instead of just the f_k's. 101: supervised -> semi-supervised would also make sense here 138: "which is a more difficult problem": in what way? 194: the simplest 231: not -> note Figure 2: Perhaps the legibility of the figure can be improved by making the figure slightly bigger, for instance by sharing the axis annotations of the small multiples.   # After response Thank you for your response. Perhaps I had not made clear enough what my confusion with pi^* was in my review, since your response does not specifically address my confusion (or I simply don't understand it): While I understand pi^* is an interesting mapping to estimate if we have Lambda^*, if we have some \hat{Lambda} from the unlabeled data, what is the mapping we are trying to estimate (not necessarily pi^* I suppose), and how does it relate to pi^*. Hopefully this clarifies my question. I would also like to reiterate my suggestion to include "mixture models" or something related in the title to make the goal of the paper clearer up front. Overall, I still find it an interesting paper but I am not more convinced of its merits than I already was before the response.


Self-Supervised Generation of Spatial Audio for 360 • Video
We introduce an approach to convert mono audio recorded by a 360
• video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360
• video viewing, but spatial audio microphones are still rare in current 360
• video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis of audio and 360
• video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360
• videos uploaded with spatial audio. During training, ground-truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360
• video and a mono audio track.

The paper proposes an automatic method to spatialise mono-channel audio for 360degree videos, meaning that the output of the method are several audio channels (most commonly two when human-centered applications are targetted) from which one can understand where the sound source is located. The method is based on a CNN which inputs the video and the mono-channel, and that is trained with datasets for which the ambisonic sound tracks are available. The network has a specific path for sound, for RGB and for optical flow. RGB and flow are processed and the fused, further on with audio, in order to obtain an audio-visual embedding of the scene. This embedding is fed to the audio decoder, which produces a set of TF attenuation maps (one per source). The very same audio-visual embedding is used to produce the localization weights, which are combined with the STFT attenuated by the output attenuation maps. This combination allows to generate the spatialized (STFT and hence) wavemors.  I enjoyed reading the paper, the application is definitely useful, the methodology interesting, and the text reads well. However, I think the paper needs to be more sharp in some aspects that, right now, may be a bit fuzzy/confusing.  For instance, all the claims regarding the novelty of the paper should be very precise. This study is perhaps the first addressing spatialisation from video and mono-audio, but not the first work addressing spatialisation (the audio processing literature is populated with such studies).  I do not know how restrictive is to assume that "monaural audio is recorded with an omnidirectional microphone" (L122). I believe that many comercial recording devices do NOT satisfy this constraint, thus clearly limiting the applicative interest of the method. How could this be done for non-omnidirectional microphones?  The authors assume that it is required to separate $k$ sources. I think an analysis of what happens when there are more or less sources in the scene is required to understand the impact of the choice of k. Clearly, not all scenarios have the same number of sources. For instance, could (a variation of) the system be trained for k sources plus a "trash" extra source so that it can handle more than k sources? What happens when there are less sources: how do the extra separated sources sound?  There is a small notation problem that carries an underlying conceptual problem. The authors should write very clearly that the input of the network is i(t) and not \phi_w(t). I understand that under the right conditions, the former is a very good surrogate of the later, but in any case they are conceptually the same. Therefore I strongly suggest that the manuscript is revised in this direction.  When discussing the audio separation, the authors describe the inversion procedure very roughly. This deserve more details (for instance, I understand you use ISTFT and not only IFFT). Do you overlap-add? This details deserve to be mentioned. The very same happens with the envelope distance: its description is to rough not allowing for reproducibility.  Finally the reference format should be uniformized.


VideoCapsuleNet: A Simplified Network for Action Detection
The recent advances in Deep Convolutional Neural Networks (DCNNs) have shown extremely good results for video human action classification, however, action detection is still a challenging problem. The current action detection approaches follow a complex pipeline which involves multiple tasks such as tube proposals, optical flow, and tube classification. In this work, we present a more elegant solution for action detection based on the recently developed capsule network. We propose a 3D capsule network for videos, called VideoCapsuleNet: a unified network for action detection which can jointly perform pixel-wise action segmentation along with action classification. The proposed network is a generalization of capsule network from 2D to 3D, which takes a sequence of video frames as input. The 3D generalization drastically increases the number of capsules in the network, making capsule routing computationally expensive. We introduce capsule-pooling in the convolutional capsule layer to address this issue and make the voting algorithm tractable. The routing-by-agreement in the network inherently models the action representations and various action characteristics are captured by the predicted capsules. This inspired us to utilize the capsules for action localization and the class-specific capsules predicted by the network are used to determine a pixel-wise localization of actions. The localization is further improved by parameterized skip connections with the convolutional capsule layers and the network is trained end-to-end with a classification as well as localization loss. The proposed network achieves state-of-the-art performance on multiple action detection datasets including UCF-Sports, J-HMDB, and UCF-101 (24 classes) with an impressive ∼20% improvement on UCF-101 and ∼15% improvement on J-HMDB in terms of v-mAP scores.

The paper presents a method for video action detection. The proposed method extends the recently proposed 2D capsule networks to 3D. The proposed network consists of a 3D encoder (convolutional layers) followed by two capsule networks; the last of which is fully connected to C capsules (C the number of action classes). For localization, the wrong outputs of the capsules (the ones not belonging to the ground truth class) are masked out and the one remaining (that corresponds to the ground truth class) is first fed to a fully-connected layer and then to a series of transposed convolutions to produce the final pixel-level localization. The method is evaluated on three popular action localization datasets achieving state-of-the-art results.   The paper is well-written and easy to follow. It tackles the difficult problem of action detection, being able to localize actions not only with bounding boxes but also with pixel-wise localizations. The results of Table 1 show that using capsules really outperforms the state of the art for all datasets.   The method is mostly a simple extension of 2D capsules to 3D.  Besides this extension, the main difference from the original work is capsule pooling (Section 3.1). This seems a bit contradictory to the initial capsule works and to the whole idea behind capsules [7,11], where the authors state that pooling throws away information about the precise position of the entity within a region. If there was no memory limitation, do the authors think that the network without the pooling operation could perform better?   For capsule pooling, the authors use the mean of the capsules in the receptive field of each capsule type. Did the authors try to experiment with other types of pooling, such as max-pooling?   It would also be interesting to see where the remainder of the error comes from, eg. if it is due to incorrect localization and/or classification of the actions.   In lines 33-37 of the supplementary material, the authors attribute the low performance of some classes to different backgrounds or to cluttered scenes, which is not the case of other classes, such as cliff diving. It would be beneficial if the authors could comment on that. Moreover, It would be interesting to see what happens when there are multiple people in the scene, eg. ice dancing, with some examples or analysis maybe (given that Table 5 of the supplementary material shows that the proposed method achieves 100% video-mAP for this class).   The authors do not use any motion information. It would be interesting to see how the method would perform with optical flow as input, especially for the failure cases of RGB inputs, eg. cluttered background.   Minor: - line 78: work instead of works.  - line 263: contain instead of contains. - Section 5.3 should also mention that the results are reported in Table 2.  - A small part of the right side of Figure 2 is missing.   [About author feedback] The authors did not really explain the difference between f-map and v-map. Works like [2] and [4] also use multiple frames as input and the difference between their f-map and v-map is expected (temporally consistent detections vs per-frame detections). I am not convinced by the explanation provided by the authors. Regarding the “remainder of the error”: localization can mean temporal or spatial; hence, an explanation for this could be useful. Overall, I think that it’s a good submission and I stand by my initial score.