Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning
In reinforcement learning, agents learn by performing actions and observing their outcomes. Sometimes, it is desirable for a human operator to interrupt an agent in order to prevent dangerous situations from happening. Yet, as part of their learning process, agents may link these interruptions, that impact their reward, to specific states and deliberately avoid them. The situation is particularly challenging in a multi-agent context because agents might not only learn from their own past interruptions, but also from those of other agents. Orseau and Armstrong [16] defined safe interruptibility for one learner, but their work does not naturally extend to multi-agent systems. This paper introduces dynamic safe interruptibility, an alternative definition more suited to decentralized learning problems, and studies this notion in two learning frameworks: joint action learners and independent learners. We give realistic sufficient conditions on the learning algorithm to enable dynamic safe interruptibility in the case of joint action learners, yet show that these conditions are not sufficient for independent learners. We show however that if agents can detect interruptions, it is possible to prune the observations to ensure dynamic safe interruptibility even for independent learners.

This paper presents an extension of the safe interruptibility (SInt)
framework to the multi-agent case. The authors argue that the
original definition of safe interruptibility is difficult to use in
this case and give a more constrained/informed one called 'dynamic safe
interruptibility' (DSInt) based on whether the update rule depends
on the interruption probability. The joint action case is considered
first and it is shown that DSInt can be achieved. The case of
independent learners is then considered, with a first result showing
that independent Q-learners do not satisfy the conditions of the
definition of DSInt. The authors finally propose a model where the
agents are aware of each others interruptions, and interrupted
observations are pruned from the sequence, and claim that this model
verify the definition of DSInt.
The paper is mostly well-written, well motivated, offers novel ideas
and appears mostly technically correct. The level of formalism is
good, emphasizing on the ideas rather than rendering a boring
sequence of symbols, but this also hurts somewhat the proof reading
of the appendix, so I can't be 100% confident about the results.
* Main comments:
- What is the relation between the authors' definition of DSInt and
the original version? In particular, do we have DSInt implies SInt,
or the converse, or neither?
- In the proof of Lemma 1, p. 12, third equality, a condition on 'a'
is missing in the second P term. As far as I understand, this
condition cannot be removed, preventing this term from being pulled
out of the sum.
* Minor comments:
- The motivating example is pretty good. On a side note however, it
seems that in this scenario the knowledge of others' interruptions
may still be inadequate, as in the general case the different
vehicles may not even be of the same brand and thus may not exchange
information (unless some global norm is defined?). This is not so
much of a criticism, as one needs to start somewhere, but a
discussion about that could go in the conclusion.
- Definition 2: This should define "dynamic safe interruptibility",
not just "safe interruptibility" which is already defined in [16].
- The notation used for open intervals is at times standard (in
the appendix) and at other times non-standard (as in the main text).
This should be made homogeneous (preferably using the standard
notation).
- Def 5: "multi-agent systems" -> "multi-agent system"
- Proof of Thm 2: On the first equality in the display, we lost the
index (i).
- The second line after the display should probably read
\hat{Q^m_t} instead of Q^{(m)}_t.
- Missing period at the end of the proof.
- Proof of Thm 3: "two a and b" -> "two agents a and b" (or learner,
player?)
- \gamma is not defined AFAICT
- Proof of Thm 4, "same than" -> "same as"
- Does it mean that independent learners are DSInt but not SInt?
- Penultimate sentence of the conclusion, "amount exploration" -> +of
- p.12 after Lemma 2 "each agents do not" -> "each agent does" or
"no agent learns"
- last line of the display in the proof of Lemma 1: Use the \left[
and \right]  commands (assuming LaTeX is being used) to format
properly.
- proof of Thm 4: q is undefined.


Repeated Inverse Reinforcement Learning
We introduce a novel repeated Inverse Reinforcement Learning problem: the agent has to act on behalf of a human in a sequence of tasks and wishes to minimize the number of tasks that it surprises the human by acting suboptimally with respect to how the human would have acted. Each time the human is surprised, the agent is provided a demonstration of the desired behavior by the human. We formalize this problem, including how the sequence of tasks is chosen, in a few different ways and provide some foundational results.

## Summary
This is a theoretical work in which the authors present a new problem domain called repeated inverse reinforcement learning (RIRL). This domain describes interactions between a human (expert) and the RIRL agent, whereby the agent repeatedly presents solutions to tasks (in the form of a discounted state occupancy) and the human responds with a demonstration if surprised by the (suboptimality) of the agent's choice. At each iteration the agent also  potentially makes some choice about the domain, which can include the state transition structure, and/or the task specific part of the reward function. The goal is to determine the task independent reward function, which in real terms might refer to such things as norms, preferences and safety considerations. Towards the end of the paper, the authors relax the requirement for a full state occupancy to be presented, instead allowing trajectories to be presented. The solution is presented either as an optimal policy for any desired task, or by identifying the task independent reward function up to an equivalence class.
The main contributions of the paper are two algorithms and a series of convergence bounds proven for each setting. The algorithms making use of a 'volume reduction in ellipsoid' algorithm from the optimization literature, which reduces the space of possible reward functions to a minimal-volume enclosing ellipsoid. The paper is quite dense in theoretical results and pushes a number of proofs to appendices. Also, because of space considerations there are some very compact descriptions with high level intuition only. However, these results appear to be sound (to the best of my ability to judge) and to have general applicability in the newly defined problem domain. The authors could do a little more to provide both a concrete example to motivate the domain and some intuition to guide the reader (please see my comments below). Also, the tractability of this approach for 'interesting' domains is difficult to judge at this early theoretical stage.
## More detailed comments
# p2
# Authors use \Delta to represent a probability distribution over various sets, but don't define this notation anywhere, or scope its meaning (some authors forbid continuous sets, other authors forbid discrete distributions with zero components when using this notation).
# Or the identity function used for the state occupancy vector.
# p4, line 158
  Each task is denoted as a pair (X, R).
# The authors do not describe the meaning of R in the general bandit description. From later text it becomes apparent, but it should be stated here explicitly.
# p5, lines 171-177
# Is this really a generalisation? It seems more like a constraint to me. It certainly implies a constraint on the relationship between rewards for two different states which share features. Extension/modification might be a better word than generalisation.
# It is also implied, but not stated that the policy now maps from features to actions, rather than from states to actions.
# p5 line 183, language
    ...for normalization purpose,
# for normalization purposes,
# p5, line 186, meaning
  ...also contains the formal protocol of the process.
# It is unclear what the authors mean by this.
# p5 line 197, clarity
    Therefore, the update rule on Line 7...
# on Line 7 of Algorithm 1
# p5 line 205, clarity
    ...in the worst case...
# I think 'at it's smallest' (or similar) would be clearer. As this 'worst case' would represent the most accurate approximation to theta* possible after . A good thing.
# p5 line 206, clarity
  To simplify calculation, we relax this l∞ ball to its inscribed l2 ball.
# This took me a little while to interpret. It would be clearer if the authors talked about the lower bound represented by the l∞ ball being relaxed, rather than the ball being relaxed.
# p6 line 208 and Equation below
# When the authors say 'The unit sphere', I guess they are now talking about the l2 norm ball. This should be explicit. As should the vol function representing the volume of an ellipsoid. Also some intuition could be given as to why the third part of the inequality is as it is. If I am right then C_d sqrt(d)^d is the volume of an l2 ball that contains \Theta0 and C_d (ε/4)^d is the volume of the aforementioned lower bound l2 ball.
# p7 meaning
    We argue that this is not a reasonable protocol for two reasons: (1) in
expectation, the reward collected by the human may be less than that by the agent, which is due to us conditioning on the event that an error is spotted
# Is there a way of making this statement clearer.
# p7 clarity
  demonstration were still given in state occupancy
# Is the meaning 'demonstration were still given in terms of a state occupancy vector'
# and '(hence μ t )' could be expanded to '(we call the resulting state occupancy μ t ).
# p8 discussion for Algorithm 2, intuition.
# The authors could give some intuition behind the construction of the batch state vectors \bar{Z} and \bar{Z}^* in the Algorithm. The former appears to be an unnormalised state occupancy following \pi_t = \pi_{t-n} whose initial state distribution is uniformly sampled from the states with mistakes in between iterations t-n and t. Likewise, the \bar{Z}* vector is an unnormalised sum of discounted state visits from the n demonstrations initiated by the mistakes.  Is this correct?


Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones. Causes of failures include software bugs, network asynchrony, biases in local datasets, as well as attackers trying to compromise the entire system. Assuming a set of n workers, up to f being Byzantine, we ask how resilient can SGD be, without limiting the dimension, nor the size of the parameter space. We first show that no gradient aggregation rule based on a linear combination of the vectors proposed by the workers (i.e, current approaches) tolerates a single Byzantine failure. We then formulate a resilience property of the aggregation rule capturing the basic requirements to guarantee convergence despite f Byzantine workers. We propose Krum, an aggregation rule that satisfies our resilience property, which we argue is the first provably Byzantine-resilient algorithm for distributed SGD. We also report on experimental evaluations of Krum.

Intro:
The paper introduces a novel algorithm, called Krum, that combines partially calculated gradients in a Byzantine failure tolerant way. Such an algorithm is meant to be useful in distributed training of machine learning models on very large data sets.
I am a machine learner, but not a distributed computing expert. Hence, I review the manuscript from a customer perspective.
---
Strengths:
 * The research question is very interesting. Distributed techniques start to play a more important role in machine learning as the scales of the data sets grow exponentially.
 * The paper is written in a sophisticated academic language. It is also nicely structured.
 * The proposed algorithm is supported with rigorous theoretical argumentations.
---
Weaknesses:
 * Results in Figure 4 are not charming. When there are no Byzantine failures, the proposed algorithm lags far behind simple averaging. It starts to show its power only after one third of the workers are Byzantine. This looks like an extremely high failure rate. Apparently, averaging will remain to be extremely competitive when the failure rates are realistically low, making the motivation of the proposed approach questionable.
 * While the paper introduces an idea that starts to become useful when the data set is dramatically large, the reported experiments are on extremely small data sets. MNIST has 60000 instances, spambase has 4601. The real effect of the divergence between the true and the stochastic gradient starts to become visible when the data set is large enough. With today's hardware technology, a modest workstation with a few-thousand-dollar GPU can be easily trained on MILLIONS of data points without any need for such protocols as Krum. Furthermore, whether more data points than a few million are needed depends totally on applications. In many, the performance difference between 10 Million and 100 Million is ignorably small. For a machine learner to be convinced to slow down her model for the sake of safe distributedness, the authors should pinpoint applications where there is a real need for this and report results on those very applications.
 * The proposed algorithm is for updating global parameters from stochastic gradients calculated on minibatches. Although this approach is fundamental to many machine learning models, the central challenge of the contemporary techniques is different. Deep neural nets require distributed computing to distribute operations across "neurons" rather than minibatches. The fact that the proposed algorithm cannot be generalized to this scenario reduces the impact potential of this work significantly.
Minor point: The paper would also be stronger if it cited earlier pioneer work on distributed machine learning. One example is:

C.T. Chu et al., Map-Reduce for Machine Learning on Multicore, NIPS, 2007
---
Preliminary Evaluation:
While I appreciate the nice theoretical work behind this paper and that distributed machine learning is an issue of key importance, the reported results shed doubt on the usefulness of the proposed approach.
---
Final Evaluation:

I do acknowledge that 33% Byzantine failure rate is a standard test case for general distributed computing tasks, but here our concern is training a machine learning model. The dynamics are a lot different from "somehow" processing as large data bunches as possible. The top-priority issue is accuracy, not data size. According to Figure 4, Krum severely undermines the model accuracy if there is no attack. This literally means a machine learner will accept to use Krum only when she is ABSOLUTELY sure that i) a 10-20 million data point subset will not be sufficient for satisfactory accuracy (hence distributed computing is required), and ii) at least 33% of the nodes will act Byzantine (hence Krum is required). As a machine learner, I am trying hard but not managing to find out such a case. Essentially, it is not the readership's but the authors' duty to bring those cases to attention. This is missing in both the paper and the rebuttal. I keep my initial negative vote.


Sharpness, Restart and Acceleration
The Łojasiewicz inequality shows that sharpness bounds on the minimum of convex optimization problems hold almost generically. Sharpness directly controls the performance of restart schemes, as observed by Nemirovskii and Nesterov [1985]. The constants quantifying error bounds are of course unobservable, but we show that optimal restart strategies are robust, and searching for the best scheme only increases the complexity by a logarithmic factor compared to the optimal bound. Overall then, restart schemes generically accelerate accelerated methods.

This paper consider first-order algorithms for Holder-smooth convex optimization in the oracle model with an additional sharpness assumption, guaranteeing that, within a neighborhood of optimum, a reduction in objective value yields a reduction in distance from optimum. Recently, there has been growing interest in the algorithmic consequences of the presence of sharpness, particularly in the setting of alternating minimization and of compressed sensing.
Sharpness can be exploited to speed up the convergence of first-order methods, such as Nesterov's accelerated gradient descent, by appropriately restarting the algorithm after a certain number of iterations, possibly changing with the number of rounds. First, the authors provide asymptotically optimal restart schedules for this class of problem for given sharpness parameters mu and r. While this is interesting, the result is essentially the same as that appearing, in more obscure terms, in Nemirovski and Nesterov's original 1985 paper "Optimal methods of smooth convex optimization". See paragraph 5 of that paper.
More importantly, the authors show that a log-scale grid search can be performed to construct adaptive methods that work in settings when mu and r are unknown, which is typical in sharpness applications. This appears to be the main novel idea of the paper. From a theoretical point of view, I find this is to be a fairly straightforward observation. On the other hand, such observation may be important in practice. Indeed, the authors also show a small number of practical examples in the context of classification, in which the restart schedules significantly improve performance. At the same time, the fact that restarts can greatly help the convergence of accelerated methods has already been observed before (see O'Donoghue and Candes, as cited in the paper).
In conclusion, I find the paper interesting from a practical point of view and I wish that the authors had focused more on the empirical comparison of their restart schedule vs that of Nemirovski and Nesterov and others. From a theoretical point of view, my feeling is that the contribution is good but probably not good enough for NIPS. It might help if the authors, in their rebuttal, explained more clearly the relation of their non-adaptive bounds with those of Nemirovski and Nesterov.


Dynamic-Depth Context Tree Weighting
Reinforcement learning (RL) in partially observable settings is challenging because the agent's observations are not Markov. Recently proposed methods can learn variable-order Markov models of the underlying process but have steep memory requirements and are sensitive to aliasing between observation histories due to sensor noise. This paper proposes dynamic-depth context tree weighting (D2-CTW), a model-learning method that addresses these limitations. D2-CTW dynamically expands a suffix tree while ensuring that the size of the model, but not its depth, remains bounded. We show that D2-CTW approximately matches the performance of state-of-the-art alternatives at stochastic time-series prediction while using at least an order of magnitude less memory. We also apply D2-CTW to model-based RL, showing that, on tasks that require memory of past observations, D2-CTW can learn without prior knowledge of a good state representation, or even the length of history upon which such a representation should depend.

The paper develops a variation on Context Tree Weighting (CTW) which keeps memory costs low by adapting the depth of each branch to the extent that it aids prediction accuracy. The new algorithm, called Utile Context Tree Weighting (UCTW), is shown empirically in some illustrative examples to use less memory than fixed-depth CTW (since it can keep some branches short) and to be more effective under a memory bound (in which it must prune a node every time it expands a node).
---Quality---
As far as I can tell the technical claims and formalization of the algorithm are sensible. The experiments are, for the most part well designed to answer the questions being asked.
One experiment that felt less well-posed was the T-Maze. The text says "We consider a maze of length 4. Thus we set K = 3." What does that "thus" mean? Is the implication that K = 3 should be deep enough to represent the environment? Later it says that "during the initial stages of learning the agent may need more than 3 steps to reach the goal." I assume that means that the agent might move up and down the "stem" of the T for a while, before reaching the goal, thus forgetting the initial observation if the suffix is limited to depth 3. If that's the case, then K = 3 is only sufficient to make predictions under the *optimal* policy, so it's no surprise that CTW+UCT can't perform well (UCT does random rollouts!). In fact, under those dynamics no finite suffix is enough to represent the environment (for arbitrary action sequences), so even the depth 4 model that UCTW learns is incorrect -- it just happens to be deep enough to be sufficiently robust to suboptimal behavior to allow the planner to work. I guess I'm just not entirely sure what to conclude from these results. We see that CTW does poorly when given inadequate depth (no surprise) and that UCTW adapts its depth, so that's fine. But UCTW doesn't learn a good model either, and it's basically a coincidence of the domain that it happens to work out for planning purposes. The other experiments, which are more focused on performance under memory bounds, make a lot more sense to me.
---Clarity---
I think the paper was pretty clearly written. The theoretical framework of CTW is always a challenge to present, and I think the authors have done pretty well. The main idea of the algorithm is described well at an intuitive level as well as at a formal level.
I will say this: the name of the algorithm is confusing. The "utile" in utile suffix memory refers to the fact that the tree is expanded based on *utility* (i.e. value). The main point of that work was that the tree should only be as complicated as it needs to be in order to solve the control task. Here the tree is being split based on prediction error of the next observation, not utility, so it is strange to call it Utile CTW. I saw the footnote acknowledging and clarifying this mismatch...but the fact that you had to write that footnote is a pretty good sign that the name is confusing! How about "Incremental Expansion CTW", "Dynamic Depth CTW", or "Memory Bounded CTW"? UCTW is just not descriptive of what the algorithm does....
---Originality---
Clearly the work is directly built upon existing results. However, I would say that it combines the ideas in a novel way. It re-purposes an alternative formulation of CTW in a clever way and develops the necessary updates to repair the tree after an expansion or pruning.
---Significance---
I think UCTW is interesting and may have a significant practical impact. CTW is an important algorithm in the compression literature and gaining interest in the AI/ML literature. I agree with the authors that memory is a major bottleneck when applying CTW to interesting problems, so a memory-bounded version is definitely of interest. Empirically UCTW shows promise -- though the experiments were performed on basic benchmarks they do demonstrate the UCTW uses less memory than fixed-depth CTW and can cope  with a memory bound.
UCTW is a little bit of a strange beast, though. One of the appeals of CTW is that it has this very clear Bayesian interpretation of representing a distribution over all prunings of a tree. It's not at all clear what happens to that interpretation under UCTW. UCTW is *explicitly* expanding and pruning the tree using a statistical test rather than the posterior beliefs. The claim that UCTW will eventually do as well as fixed-depth CTW makes sense, and is comforting -- it eventually finds its way to the original Bayesian formulation and can overcome any funkiness in the initialization due to the process up until that point. Furthermore it's not clear what happens to the regret bounds that CTW enjoys once this expansion/pruning scheme is introduced. This is not really an objection -- sometimes some philosophical/mathematical purity must be sacrificed for the sake of practicality. But it does make things muddier and it is harder to interpret the relationship of this algorithm to other CTW variants.
Similarly, once we discard the clean interpretation of CTW, it does raise the question for me of why use CTW at all at this point? The authors raise the comparison to USM, but don't really compellingly answer the question "Why not just use USM?" The point is made that USM uses the K-L test, which is expensive, and doesn't have a memory bound. However, the main ideas used here (use likelihood ratio test instead and require a trade-off between expanding and pruning once the limit is hit) seem like they could just as easily be used in USM. I do no intend to suggest that the authors must invent a memory-bounded version of USM to compare to. However, if the authors do have a clear idea of why that's not a good idea, I think it would be valuable to discuss it. Otherwise I feel like the motivation of the work is a little bit incomplete.
***After Author Response***
I do think the name of the algorithm is misleading, and that leads to confusing comparisons too. For instance, in the author response the authors say "Also, the expansion tests in USM are performed over nonparametric representations of distributions over future reward, so the complexity of each test is a function of the sample size for each distribution." But, and I cannot stress this enough, *that is because USM is trying to predict value and UCTW is not.* They use different expansion tests because they address fundamentally different prediction problems. If one were to use a USM-like algorithm for predicting the next symbol from a finite alphabet, it would make perfect sense to represent the distribution using a histogram and use likelihood ratio tests instead of K-S; the complexity would be linear in the size of the alphabet, not the number of examples. USM uses K-S *because it is predicting a continuous value*. (In this case, I do nevertheless acknowledge that the CTW calculations have a nice side effect of making likelihood calculations efficient and thank the authors for that clarification).
I think this paper should be accepted, so, if that happens, obviously it will be up to the authors what they do with the title and the algorithm name and so on. My point is that the direct link between USM and UCTW is not sound -- USM and UCTW are solving different problems. Pretty much the *only* thing UCTW takes from USM is the fact that it incrementally grows its depth. So it's fine to draw this connection between two algorithms that incrementally expand a suffix tree, and its good to acknowledge inspiration, but they can't be directly compared. At best you can compare UCTW to a USM-like algorithm that predicts future symbols rather than utility, but then, because it has a different prediction problem, the design choices of USM might not make sense anymore. I think the name UCTW reinforces this flawed direct comparison because at first glance it implies that UCTW is solving the same problem as USM, and it is not. None of this is fatal; a motivated reader can untangle all of this. I just hope the authors will get really clear about the distinctions between these algorithms and then make sure the paper is as clear as it can possibly be.
I can see where the authors are coming from with the T-maze. I still think it's a bit of a wonky experiment, but adding a bit of the analysis given in the response to the paper would help a reader understand what the authors mean to extract from the results.


Fast Black-box Variational Inference through Stochastic Trust-Region Optimization
We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the "reparameterization trick." At each iteration, TrustVI proposes and assesses a step based on minibatches of draws from the variational distribution. The algorithm provably converges to a stationary point. We implemented TrustVI in the Stan framework and compared it to two alternatives: Automatic Differentiation Variational Inference (ADVI) and Hessianfree Stochastic Gradient Variational Inference (HFSGVI). The former is based on stochastic first-order optimization. The latter uses second-order information, but lacks convergence guarantees. TrustVI typically converged at least one order of magnitude faster than ADVI, demonstrating the value of stochastic second-order information. TrustVI often found substantially better variational distributions than HFSGVI, demonstrating that our convergence theory can matter in practice.

SUMMARY OF THE PAPER:
The paper transfers concepts known in the optimization community and applies them to variational inference. It introduces a new method to optimize the stochastic objective function of black box variational inference. In contrast to the standard SGD, which optimizes the objective using only first order derivatives, the proposed methods takes estimates of the Hessian into account and approximates the objective function on a small trust region around the current iterate by a quadratic function. The algorithm automatically adapts the size of the trust region based on estimates of the quality of the quadratic model. The paper proves theoretically that the algorithm converges, and shows experimentally that convergence is typically faster than standard SGD.
GENERAL IMPRESSION / POSITIVE ASPECTS:
The paper is well written and the proposed algorithm is potentially highly relevant given its generality and the reported improvements in speed of convergence. The experimental part summarizes results from a very large number of experiments, albeit it is not clear to me whether this includes any experiments on large-scale datasets that require minibatch sampling.
WHAT COULD BE IMPROVED:
1. I appreciate the fact that the paper includes an unambiguous definition of the algorithm and a rigorous proof that the proposed algorithm converges. To make the method more accessible, it would however be helpful to also briefly explain the algorithm in more intuitive terms. For example, a geometric interpretation of \gamma, \lambda, \eta, and \delta_0 would help practitioners to choose good values for these hyperparameters.
2. An often cited advantage of VI is that it scales to very large models when minibatch sampling (stochastic VI) is used. Minibatch sampling introduces an additional source of stochasticity. The paper discusses only stochasticity due to the black box estimation of the expectation under q(z). How does the proposed algorithm scale to large models? Are the convergence guarantees still valid when minibatch sampling is used and do we still expect similar speedups?
3. It is not clear to me how $\sigma_k$, defined in Condition 4, can be obtained in practice.
MINOR REMARKS:
1. Algorithm 1: R^d should be R^D in the first line.
2. Lines 108-114 discuss a set of hyperparameters, whose motivation becomes clear to the reader only after reading the convergence proofs. It might be clearer to present the hyperparameters in a table with a row for each hyperparameter, and columns for the symbol, a short and descriptive name, and the interval of allowed values.
3. Eq. 7: which norm is used for the Hessian matrix here?
4. Line 154: It may be helpful to point out at this point that one draws *new* samples from p_0, i.e., one may not reuse samples that were used to generate g_k and H_k. Otherwise, $\ell'_{ki}$ are not i.i.d. The algorithm box states this clearly, but it was not immediately clear to me from the main text.


Near Minimax Optimal Players for the Finite-Time 3-Expert Prediction Problem
We study minimax strategies for the online prediction problem with expert advice. It has been conjectured that a simple adversary strategy, called COMB, is near optimal in this game for any number of experts. Our results and new insights make progress in this direction by showing that, up to a small additive term, COMB is minimax optimal in the finite-time three expert problem. In addition, we provide for this setting a new near minimax optimal COMB-based learner. Prior to this work, in this problem, learners obtaining the optimal multiplicative constant in their regret rate were known only when K = 2 or K → ∞. We characterize, when K = 3, the regret of the game scaling as � 8/(9π)T ± log(T ) 2 which gives for the first time the optimal constant in the leading ( √ T ) term of the regret.

The paper studies the classic prediction with experts advice problem. There are a finite number k of experts and a finite number T of rounds. There is a player that makes sequential decisions for T rounds based on the advice of the k experts, and his goal is to minimize the maximum regret he can experience (minimax regret). Naturally, the optimal adversarial strategy is a key quantity to study here. This paper takes up the conjectured minimax optimal adversarial strategy called "Comb strategy" in the Gravin et al. paper and shows that it is indeed minimax optimal in the case of 3 experts. The high level proof structure is to show that the comb strategy is optimal assuming that in all the future rounds the comb strategy will be employed and then induct. (Comb strategy is not the exact optimal though --- thus one has to lose a bit in terms of optimality, and bound how the error accumulates as one inducts) One of the useful lemmas in doing this is the "exchangeability property", which shows that deviating from comb strategy in exactly one round will give a regret that is independent of which round one deviates off of it. The need for this property arises naturally when one takes the induction proof structure above. The proof of this property proceeds by considering another adversarial strategy twin-comb, also proved to be optimal for geometric horizon setting by Gravin et al., and then showing that playing Comb + non-comb is same as playing twin-comb + comb where non-comb is a mixture of {Comb, twin-comb}. The question is how to generalize this for arbitrary number of experts k. For general k, is it possible to push this idea through with the generalized twin-comb? And what would be the appropriate generalization? This is not immediately clear from the paper, but I still think the contribution as it is already meets the NIPS bar, and certainly has some neat ideas.
Given that this paper takes a conjectured optimal strategy, whose optimality for a small number of experts can be verified directly by writing a computer program to compute optimal adversarial values, its primary value can lie only in the new techniques it introduces. The main challenge is that since the comb adversary is not the exactly optimal adversary, all the nice properties of optimality are gone, and one has to control how the errors of this suboptimal strategy accumulate in order to establish approximate optimality. The "exchangeability" property is the main new idea for doing this. There is also the other nice contribution of proving that the player's algorithm that flows out of the comb adversary, which itself is not the exactly optimal but only approximately optimal, is an approximately optimal algorithm. Usually, when the adversary is not optimal, proving optimality for the algorithm that simulates it is not straight-forward. While the techniques don't immediately generalize, the general approach of controlling COMB adversary's errors with an exchangeability technique seems like the way to go for general k. Given this, and the fact that the work is technically interesting, I would say it meets the NIPS bar.


Multimodal Learning and Reasoning for Visual Question Answering
Reasoning about entities and their relationships from multimodal data is a key goal of Artificial General Intelligence. The visual question answering (VQA) problem is an excellent way to test such reasoning capabilities of an AI model and its multimodal representation learning. However, the current VQA models are oversimplified deep neural networks, comprised of a long short-term memory (LSTM) unit for question comprehension and a convolutional neural network (CNN) for learning single image representation. We argue that the single visual representation contains a limited and general information about the image contents and thus limits the model reasoning capabilities. In this work we introduce a modular neural network model that learns a multimodal and multifaceted representation of the image and the question. The proposed model learns to use the multimodal representation to reason about the image entities and achieves a new state-of-the-art performance on both VQA benchmark datasets, VQA v1.0 and v2.0, by a wide margin.

Summary
This paper proposes an approach that combines the output of different vision systems in a simple and modular manner for the task of visual question answering. The high level idea of the model is as follows. The question is first encoded into a bag of words representation (and passed through an MLP). Then different vision systems (which extract raw features, or compute attention on images using object detection outputs or face detection outputs or scene classification outputs) are all condensed into a representation compatible with the question. Finally the approach takes an outer product between the image representations from various tasks and the question, and concatenates the obtained representations. This concatenated feature is fed as input to an answering model which produces distributions over answer tokens. The entire model is trained with max-likelihood. Results on the VQA 1.0 as well as VQA 2.0 datasets show competitive performance with respect to the state of the art.
Strengths
1. At a high level the proposed approach is very well motivated since the vqa task can be thought of as an ensemble of different tasks at different levels of granularity in terms of visual reasoning. The approach has flavors of solving each task separately and then putting everything together for the vqa task.
2. The results feature ablations of the proposed approach which helps us understand the contributions of different modules in achieving the performance gains.
3. It is really encouraging that the approach obtains state of the art results on VQA. Traditionally there has been a gap between modular architectures which we “feel” should be good / right for the task of VQA and the actual performance realized by such models. This paper is a really nice contribution towards integrating different vision sub-problems for VQA, and as such is a really good step.
Weakness
1. When discussing related work it is crucial to mention related work on modular networks for VQA such as [A], otherwise the introduction right now seems to paint a picture that no one does modular architectures for VQA.
2. Given that the paper uses a billinear layer to combine representations, it should mention in related work the rich line of work in VQA, starting with [B] which uses billinear pooling for learning joint question image representations. Right now the manner in which things are presented a novice reader might think this is the first application of billinear operations for question answering (based on reading till the related work section). Billinear pooling is compared to later.
3. L151: Would be interesting to have some sort of a group norm in the final part of the model (g, Fig. 1) to encourage disentanglement further.
4. It is very interesting that the approach does not use an LSTM to encode the question. This is similar to the work on a simple baseline for VQA [C] which also uses a bag of words representation.
5. (*) Sec. 4.2 it is not clear how the question is being used to learn an attention on the image feature since the description under Sec. 4.2 does not match with the equation in the section. Speficially the equation does not have any term for r^q which is the question representation. Would be good to clarify. Also it is not clear what \sigma means in the equation. Does it mean the sigmoid activation? If so, multiplying two sigmoid activations (with the \alpha_v computation seems to do) might be ill conditioned and numerically unstable.
6. (*) Is the object detection based attention being performed on the image or on some convolutional feature map V \in R^{FxWxH}? Would be good to clarify. Is some sort of rescaling done based on the receptive field to figure out which image regions belong correspond to which spatial locations in the feature map?
7. (*) L254: Trimming the questions after the first 10 seems like an odd design choice, especially since the question model is just a bag of words (so it is not expensive to encode longer sequences).
8. L290: it would be good to clarify how the implemented billinear layer is different from other approaches which do billinear pooling. Is the major difference the dimensionality of embeddings? How is the billinear layer swapped out with the hadarmard product and MCB approaches? Is the compression of the representations using Equation. (3) still done in this case?
Minor Points:
- L122: Assuming that we are multiplying in equation (1) by a dense projection matrix, it is unclear how the resulting matrix is expected to be sparse (aren’t we mutliplying by a nicely-conditioned matrix to make sure everything is dense?).
- Likewise, unclear why the attended image should be sparse. I can see this would happen if we did attention after the ReLU but if sparsity is an issue why not do it after the ReLU?
Perliminary Evaluation
The paper is a really nice contribution towards leveraging traditional vision tasks for visual question answering. Major points and clarifications for the rebuttal are marked with a (*).
[A] Andreas, Jacob, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2015. “Neural Module Networks.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1511.02799.
[B] Fukui, Akira, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. “Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1606.01847.
[C] Zhou, Bolei, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. 2015. “Simple Baseline for Visual Question Answering.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1512.02167.


Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.

In this paper, the authors propose Batch Renormalization technique to alleviate the problem of batchnorm when dealing with small or non-i.i.d minibatches. To reduce the dependence of large minibatch size is very important in many applications especially when training large neural network models with limited GPU memory. The proposed method is vey simple to understand and implement. And experiments show that Batch Renormalization performs well with non-i.i.d minibatches, and improves the results of small minibatches compared with batchnorm.
Firstly, the authors give a clear review of batchnorm, and conclude that the key drawbacks of batchnorm are the inconsistency of mean and variance used in training and inference and the instability when dealing with small minibatches. Using moving averages to perform normalization would be the first thought, however this would lead to the model blowing up. So the authors propose a simple batch renormalization method to combine minibatch mean and variance with moving averages. In my opinion, what Batch Renormalization does is to gradually changing from origin batchnorm (normalizing with minibatch mean and variance) to batchnorm with (almost) only moving averages. In this way, the model can adopt part of the advantage of moving averages, and converge successfully.
And I have several questions about this paper.
(1)	When using large minibatch size (such as 32), why Batch Renormalization has no advantage compared with batchnorm. It seems that the consistency of mean and variance in training and inference does not help much in this case.
(2)	Experiments show that the result of small minibatch size (batchsize=4) is worse that the result of large minibatch size (batchsize=32). So I wonder if using two (multiple) moving averages (mean and variance) with different update rates (such as one with 0.1, one with 0.01) would help. Small update rate helps to solve the inconstancy problem, large update rate helps to solve the small minibatch size problem.
(3)	The results of how r_max and d_max will affect the performace are not provided. There seems to have plenty of parameter tuning work.
This work is good, and I am looking forward to seeing a more elegant solution to this problem.


MarrNet: 3D Shape Reconstruction via 2.5D Sketches
3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.

This paper describes a technique for estimating voxelized 3D shapes from single images. As directly predicting 3D shapes from images is hard, the paper proposes to separate the problem into 2 tasks - inspired by Marr's theory about vision. In the first part the method takes the image as input and predicts several intrinsic quantities ("2.5D sketch"), in particular depth, surface normals and the 2D silhouette using an encoder-decoder architecture. This information is fed into a second encoder-decoder network which predicts the volumetric 3D representation at 128^3 resolution. The advantage of the method is that the second part can be trained on synthetic data allowing for "domain transfer" based on the 2.5D sketches which are invariant to the actual appearance. Experiments (mainly qualitative) on Shapenet, IKEA and Pascal3D+ are presented.
Overall, I liked the idea. The paper is clear, well motivated and written and the qualitative results seem convincing. However, I have some concerns about the fairness of the evaluation and the self-supervised part which I like to have answered in the rebuttal before I turn towards a more positive score for this paper.
Positive aspects:
+ Very well written and motivated
+ Relevant topic in computer vision
+ Interesting idea of using intermediate intrinsic representations to facilitate the task
+ Domain transfer idea is convincing to me (though the self-supervision and fine-tuning of the real->sketch network are not fully clear to me, see below)
+ Qualitative results clearly superior than very recent baselines
Negative aspects:
- Self-supervision: It is not clear to me why this self-supervision should work. The paper says that the method fine-tunes on single images, but if the parameters are optimized for a single image, couldn't the network diverge to predict a different shape encapsulated in the shape space for an image depicting another object while still reducing the loss? Also, it is not very well specified what is finetuned exactly. It reads as if it was only the encoder of the second part. But how can the pre-trained first part then adapted to a new domain? If the first part fails, the second will as well.
- Domain adaptation: I agree that it is easy to replace the first part, but I don't see how this can be trained in the absence of real->sketch training pairs. To me it feels as the method profits from the fact that for cars and chairs pretraining on synthetic data already yields good models that work well with real data. I would expect that for stronger domain differences the method would not be able to work without sketch supervision. Isn't the major problem there to predict good sketches from little examples?
- Baselines: It is not clear if the direct (1-step) baseline in the experiments uses the same architecture or at least same #parameters as the proposed technique to have a fair evaluation. I propose an experiment where the same architecture is used but insteads of forcing the model (ie, regularizing it) to capture 2D sketches as intermediate representations, let it learn all parameters from a random initialization end-to-end. This would be a fair baseline and it would be interesting to see which representation emerges. Another fair experiment would be one which also has 2 encoder-decoder networks and the same number of parameters but maybe a different distribution of feature maps across layers. Finanlly a single encoder-decoder architecture with a similar architecture as the proposed one but increased by the number of parameters freed by removing the first part would be valuable. From the description in the paper it is totally unclear what the baseline in Fig. 4. is. I suggest such baselines for all experiments.
- DRC: It should be clarified how DRC is used here and if the architecture and resolution is adopted for fairness. Also, DRC is presented to allow for multi-view supervision which is not done here, so this should be commented upon. Further the results of the 3D GAN seem much more noisy compared to the original paper. Is there maybe a problem with the training?
- Quantitative Experiments: I do not fully agree with the authors that quantitative experiments are not useful. As the paper also evaluates on ShapeNet, quantitative experiments would be easily possible, and masks around the surfaces could be used to emphasize thin structures in the metric etc. I suggest to add this at least for ShapeNet.
Minor comments:
- Sec. 3.1: the deconvolution/upconvolution operations are not mentioned
- It is unclear how 128^3 voxel resolution can be achieved which typically doesn't fit into memory for reasonable batch sizes. Also, what is the batch size and other hyperparameters for training?
- Fig. 8,9,10: I suggest to show the predicted sketches also for the real examples for the above mentioned reasons.


A simple neural network module for relational reasoning
Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn. In this paper we describe how to use Relation Networks (RNs) as a simple plug-and-play module to solve problems that fundamentally hinge on relational reasoning. We tested RN-augmented networks on three tasks: visual question answering using a challenging dataset called CLEVR, on which we achieve state-of-the-art, super-human performance; textbased question answering using the bAbI suite of tasks; and complex reasoning about dynamic physical systems. Then, using a curated dataset called Sort-of-CLEVR we show that powerful convolutional networks do not have a general capacity to solve relational questions, but can gain this capacity when augmented with RNs. Thus, by simply augmenting convolutions, LSTMs, and MLPs with RNs, we can remove computational burden from network components that are not well-suited to handle relational reasoning, reduce overall network complexity, and gain a general ability to reason about the relations between entities and their properties.

The paper proposes a plug and play module (called Relation Networks (RNs)) specialized for relational reasoning. The module is composed of Multi Layer Perceptrons and considers relations between all pairs of objects. The proposed module when plugged into traditional networks achieves state of the art performance on the CLEVR visual question answering dataset, state of the art (with joint training for all tasks) on the bAbI textual question answering dataset and high performance (93% on one task and 95% on another) on a newly collected dataset of simulated physical mass-spring systems. The paper also collects a dataset similar to CLEVR to demonstrate the effectiveness of the proposed RNs for relational questions.
Strengths:
1. The proposed Relation Network is a novel neural network specialized for relational reasoning. The success of the proposed network is extensively shown by experimenting with three different tasks and clearly analyzing the effectiveness for relational questions by collecting a novel dataset similar to CLEVR.
2. The proposed RNs have been shown to be able to work with different forms of input -- explicit state representations as well as features from a CNN or LSTM.
3. The paper is well written and the details of model architecture including hyperparameters are provided.
4. As argued in the paper, I agree that relational reasoning is central to intelligence and since RNs are shown to be able to achieve this reasoning and a result perform better at tasks requiring such reasoning than existing networks, RNs seem to be of significant importance for designing reasoning networks.
Weaknesses:
1. Could authors please analyze and comment on how complicated relations can be handled by RNs. Is it the case that RNs perform well for single hop relations such as "what is the color of the object closest to the blue object" which requires reasoning about only one hop relation (distance between blue object and all other objects), but not so well for multiple hop relations such as "What shape is the small object that is in front of the yellow matte thing and behind the gray sphere?". From the failure cases in table 1 of supplementary material, it seems that the model has difficulty in answering questions involving multiple hops of relations.
2. L203-204, it is not clear to me what do authors mean by "we tagged ... support set". Is this referring to some form of human annotation? If so, could authors please elaborate on what happens at test time?
3. All the datasets experimented with in the paper are synthetic datasets. Could authors please comment on how they expect the RNs to work on real datasets such as the VQA dataset from Antol et al.?
Post-rebuttal comments:
Authors have provided satisfactory response to my question about multi-hop reasoning. However, I would still like to see experiments on real VQA dataset to see how effective RNs are at dealing with the amount of variation real datapoints show (in vision as well as in language). So it would be great if authors could include results on the VQA dataset (Antol et al., ICCV 2015) in camera-ready.


ADMM without a Fixed Penalty Parameter: Faster Convergence with New Adaptive Penalization
Alternating direction method of multipliers (ADMM) has received tremendous interest for solving numerous problems in machine learning, statistics and signal processing. However, it is known that the performance of ADMM and many of its variants is very sensitive to the penalty parameter of a quadratic penalty applied to the equality constraints. Although several approaches have been proposed for dynamically changing this parameter during the course of optimization, they do not yield theoretical improvement in the convergence rate and are not directly applicable to stochastic ADMM. In this paper, we develop a new ADMM and its linearized variant with a new adaptive scheme to update the penalty parameter. Our methods can be applied under both deterministic and stochastic optimization settings for structured non-smooth objective function. The novelty of the proposed scheme lies at that it is adaptive to a local sharpness property of the objective function, which marks the key difference from previous adaptive scheme that adjusts the penalty parameter per-iteration based on certain conditions on iterates. On theoretical side, given the local sharpness characterized by an exponent θ ∈ (0, 1], we show that the proposed ADMM enjoys an improved iteration complexity of O(1/ 1−θ ) 1 in the deterministic setting and an iteration complexity of O(1/ 2(1−θ) ) in the stochastic setting without smoothness and strong convexity assumptions. The complexity in either setting improves that of the standard ADMM which only uses a fixed penalty parameter. On the practical side, we demonstrate that the proposed algorithms converge comparably to, if not much faster than, ADMM with a fine-tuned fixed penalty parameter.

Summary: This paper shows that O(1/eps) iteration complexity of ADMM can be improved to O(1/eps^(1-theta)) where theta is a  parameter that characterizes how sharply the objective function increases with respect to increasing distance to the optimal solution. This improvement is shown under a locally adaptive version of the ADMM where the penalty parameter is increased after every $t$ steps of ADMM. The method is extended to stochastic ADMM whose O(1/eps^2) iteration complexity is shown to similarly improve.  The results are backed by experiments on generalized Lasso problems.
Overall, the paper is well written and makes an important contribution towards improving the analysis of ADMM under adaptive penalty parameters. On a practical note, I would have liked to see some comparisons against other "adaptive-rho" heuristics used the literature (see Boyd's monograph). As background, it may be valuable to some readers to see how the local error bound relates to the KL property, and some examples of "theta" for problems of interest in machine learning.
There are lots of grammatical errors, typos and odd phrases in the the abstract: "tremendous interests" --> "tremendous interest", "variants are"-->"variants is", "penalty scheme of lies at it is...",  "iterate message"..
The LADMM-AP behavior in plot 1(e) is somewhat strange. Any explanations?
If factor of 2 in LA-ADMM optimal in any sense? In practice ADMM stopping criteria include primal-dual tolerance thresholds. Shouldnt LA-ADMM use those instead of fixed t?
With regards to linearized ADMM, Eqn 7, please comment on its relationship to Chambelle-Pock updates (https://hal.archives-ouvertes.fr/hal-00490826/document)


YASS: Yet Another Spike Sorter
Spike sorting is a critical first step in extracting neural signals from large-scale electrophysiological data. This manuscript describes an efficient, reliable pipeline for spike sorting on dense multi-electrode arrays (MEAs), where neural signals appear across many electrodes and spike sorting currently represents a major computational bottleneck. We present several new techniques that make dense MEA spike sorting more robust and scalable. Our pipeline is based on an efficient multistage "triage-then-cluster-then-pursuit" approach that initially extracts only clean, high-quality waveforms from the electrophysiological time series by temporarily skipping noisy or "collided" events (representing two neurons firing synchronously). This is accomplished by developing a neural network detection method followed by efficient outlier triaging. The clean waveforms are then used to infer the set of neural spike waveform templates through nonparametric Bayesian clustering. Our clustering approach adapts a "coreset" approach for data reduction and uses efficient inference methods in a Dirichlet process mixture model framework to dramatically improve the scalability and reliability of the entire pipeline. The "triaged" waveforms are then finally recovered with matching-pursuit deconvolution techniques. The proposed methods improve on the state-of-the-art in terms of accuracy and stability on both real and biophysically-realistic simulated MEA data. Furthermore, the proposed pipeline is efficient, learning templates and clustering faster than real-time for a ' 500-electrode dataset, largely on a single CPU core.

[UPDATE AFTER AUTHOR RESPONSE]
The authors' response confirms my rating – it's a valuable paper. I'm optimistic that they will address mine and the other reviewers' concerns in their revision.
[ORIGINAL REVIEW]
The paper describes YASS: Yet Another Spike Sorter, a well-engineered pipeline for sorting multi-electrode array (MEA) recordings. The approach is sound, combining many state-of-the-art approaches into a complete pipeline. The paper is well written, follows a clear structure and provides convincing evidence that the work indeed advances the state of the art for sorting retinal MEA data.
While the paper is already a valuable contribution as is, I think it has much greater potential if the authors address some of the issues detailed below. In brief, my main concerns are
(1) the lack of publicly available code,
(2) the poor description of the neural network detection method,
(3) issues with applying the pipeline to cortical data.
Details:
(1) Code is not available (or at least the manuscript does not provide a URL). Since spike sorting is at this point mainly an engineering problem (but a non-trivial one), a mere description of the approach is only half as valuable as the actual code implementing the pipeline. Thus, I strongly encourage the authors to go the extra mile and make the code available.
(2) Neural network spike detection. This part seems to be the only truly innovative one (all other components of the pipeline have been described/used before). However, it remains unclear to me how the authors generated their training data. Section C.2 describes different ways of generating training data, but it is not clear which one (or which combination) the authors use.
(a) Using pre-existing sorts.
First, most labs do *not* have existing, properly sorted data available when moving to dense MEAs, because they do not have a pipeline for sorting such array data and – as the authors point out – existing methods do not scale properly.
Second, it is not clear to me how existing sorts should help training a more robust neural network for detection. Did the authors inspect every single waveform snippet and labeled it as clean or not? If not, based on what algorithm did they decide which waveform snippets in the training data are clean? Why do they need to train a neural network instead of just using this algorithm? How do they deal with misalignments, which create label noise?
If using pre-existing sorts is what the authors did, they need to provide more information on how exactly they did it and why it works. In the current form, their work cannot be reproduced.
(b) Generating synthetic training data by superimposing waveform templates on background noise. This could be a reasonable approach. Is it used for data augmentation or not at all and just described as a potential alternative? What is the evidence that this approach is useful? The synthetic data may not be representative of real recordings.
(3) Generalization to cortical data. I am quite confident that the pipeline works well for retinal data, but I doubt that it will do so for cortical data (some arguments below). I think this limitation needs to be discussed and acknowledged more explicitly (abstract, intro, conclusions).
(a) In cortical recordings, waveform drift is a serious issue that arises in pretty much all non-chronic recordings (and chronically working high-density MEAs are still to be demonstrated). Thus, modeling drift is absolutely crucial for recordings that last longer than a few minutes.
(b) Getting good training data for the NN detection is more difficult. Good ground truth (or well validated data such as described in appendix I) is not available and generating synthetic data as described in C.2 is not necessarily realistic, since background noise is caused by spikes as well and neurons often fire in highly correlated manners (thus rendering the approach of overlaying templates on spike-free noise problematic).
Minor comments:
- Fig. 3 bottom panel: Y axis is strange. Does 1^-x mean 10^-x? Also, it exaggerates tiny accuracy differences between 0.99 and 0.999, where both methods are essentially perfect.
- The authors use spatially whitened data (according to section 2.1), but I did not find a description of the spatial whitening procedure in the manuscript or supplement.


Independence clustering (without a matrix)
The independence clustering problem is considered in the following formulation: given a set S of random variables, it is required to find the finest partitioning {U 1 , . . . , U k } of S into clusters such that the clusters U 1 , . . . , U k are mutually independent. Since mutual independence is the target, pairwise similarity measurements are of no use, and thus traditional clustering algorithms are inapplicable. The distribution of the random variables in S is, in general, unknown, but a sample is available. Thus, the problem is cast in terms of time series. Two forms of sampling are considered: i.i.d. and stationary time series, with the main emphasis being on the latter, more general, case. A consistent, computationally tractable algorithm for each of the settings is proposed, and a number of fascinating open directions for further research are outlined.

ABOUT:
This paper is about clustering N random variable into k mutually independent clusters. It considers the cases when k is known and unknown, as well as the cases when the distribution of the N variables is known, when it needs to be estimated from n i.i.d. samples, and when it needs to be estimated from stationary samples. Section 3 provides an optimal algorithm for known distributions which uses at most a 2kN^2 oracle calls. Subsequent sections build on this algorithm in more complicated cases.
I do not know the clustering literature well enough to put this work in context. However, if the overview of the prior work is accurate, Sections 3 and 4 are a nice and important contribution. I have some reservations about Section 5. Overall, the exposition of results could be improved.
COMMENTS:
(1) I am not entirely convinced by the proof for Theorem 1. I believe the statement of the result is probably correct, but the proof needs to be more rigorous. It could be useful to state what exactly is meant by mutual independents for the benefit of the reader* and then to highlight in the proof how the algorithm does achieve a mutually independent clustering. For the Split function part, an inductive proof could be one way to make the proof rigorous and easier to follow.
(2) It does not seem that the stationary case in Section 5 is covered adequately. It kind of reads like the authors tried to do too much and actually accomplished little. In parts, it just looks like a literature review or a math tutorial (e.g. Section 5.1).
- I am not sure what the issue is with mutual information rate being zero is. One natural extension of the set up in Section 4 would be to cluster the N random variables according to their stationary distribution at time i with the n time series samples being then used to estimate this distribution as well as possible. In that case the mutual information is perfectly well defined and everything follows through. This does not seem to be what is actually being done, and instead the clustering is being performed over a whole infinite time series.
- The sum information in Definition 1 seems far less interesting and fundamental than the authors try to make it out to be. Also, a side note, people have spent time thinking about what a mutual information rate for a random process should be in full generality [A] and this question maybe deserves more attention than it is given here.
*I understand that some may find this too basic. But, having this definition side by side with the proof would help the reader confirm correctness. Somewhat more importantly, this issue of pairwise independence vs mutual independence is the crux of the paper. It should be stated explicitly (i.e. using math) what that means.
MINOR COMMENT
- Line 156: N not n
[A] Te Sun Han, Information-Spectrum Method in Information Theory .
Springer, 2003