Mean Field Residual Networks: On the Edge of Chaos
We study randomly initialized residual networks using mean field theory and the theory of difference equations. Classical feedforward neural networks, such as those with tanh activations, exhibit exponential behavior on the average when propagating inputs forward or gradients backward. The exponential forward dynamics causes rapid collapsing of the input space geometry, while the exponential backward dynamics causes drastic vanishing or exploding gradients. We show, in contrast, that by adding skip connections, the network will, depending on the nonlinearity, adopt subexponential forward and backward dynamics, and in many cases in fact polynomial. The exponents of these polynomials are obtained through analytic methods and proved and verified empirically to be correct. In terms of the "edge of chaos" hypothesis, these subexponential and polynomial laws allow residual networks to "hover over the boundary between stability and chaos," thus preserving the geometry of the input space and the gradient information flow. In our experiments, for each activation function we study here, we initialize residual networks with different hyperparameters and train them on MNIST. Remarkably, our initialization time theory can accurately predict test time performance of these networks, by tracking either the expected amount of gradient explosion or the expected squared distance between the images of two input vectors. Importantly, we show, theoretically as well as empirically, that common initializations such as the Xavier or the He schemes are not optimal for residual networks, because the optimal initialization variances depend on the depth. Finally, we have made mathematical contributions by deriving several new identities for the kernels of powers of ReLU functions by relating them to the zeroth Bessel function of the second kind.

This paper analytically investigates the properties of Resnets with random weights using a mean field approximation. The approach used is an extension of previous analysis of feed forward neural networks. The authors show that in contrast to feed forward networks Resnets do exhibit a sub-exponential behavior (polynomial or logarithmic) when inputs are propagated forward or gradients are propagated backwards through the layers.
The results are very interesting because they give an analytic justification and intuition of why Resnets with a large number of layers can be trained reliably. The paper also extends the mean field technique for studying neural network properties, which is of value for analyzing other architectures.  The presentation of the results is overall clear and while the paper is well written.
Overall this is an interesting paper with convincing results.


SGD Learns the Conjugate Kernel Class of the Network
We show that the standard stochastic gradient decent (SGD) algorithm is guaranteed to learn, in polynomial time, a function that is competitive with the best function in the conjugate kernel space of the network, as defined in Daniely et al. [2016]. The result holds for logdepth networks from a rich family of architectures. To the best of our knowledge, it is the first polynomial-time guarantee for the standard neural network learning algorithm for networks of depth more that two.
As corollaries, it follows that for neural networks of any depth between 2 and log(n), SGD is guaranteed to learn, in polynomial time, constant degree polynomials with polynomially bounded coefficients. Likewise, it follows that SGD on large enough networks can learn any continuous function (not in polynomial time), complementing classical expressivity results.

This paper studies the problem of learning a class of functions related to neural networks using SGD. The class of functions can be defined using a kernel that is related to the neural network structure (which was defined in [Daniely et al. 2016]). The result shows that if SGD is applied to a neural network with similar structure, but with many duplicated nodes, then the result can be competitive to the best function in the class.
Pros:
- This is one of the first analysis for SGD on multi-layer, non-linear neural networks.
- Result applies to various architectures and activation functions.
Cons:
- The definition of the kernel is not very intuitive, and it would be good to discuss what is the relationship between functions representable using a neural network and functions representable using this kernel?
- The same result seems to be easier to prove if all previous layers have random weights and SGD is applied only to last layer. It's certainly good that SGD in this paper runs on all layers (which is similar to what's done in practice). However, for most practical architectures just training the last layer is not going to get a good performance (when the task is difficult enough). The paper does not help in explaining why training all layers is more powerful than training the last layer.
Overall this is still an interesting result.


Adaptive Active Hypothesis Testing under Limited Information
We consider the problem of active sequential hypothesis testing where a Bayesian decision maker must infer the true hypothesis from a set of hypotheses. The decision maker may choose for a set of actions, where the outcome of an action is corrupted by independent noise. In this paper we consider a special case where the decision maker has limited knowledge about the distribution of observations for each action, in that only a binary value is observed. Our objective is to infer the true hypothesis with low error, while minimizing the number of action sampled. Our main results include the derivation of a lower bound on sample size for our system under limited knowledge and the design of an active learning policy that matches this lower bound and outperforms similar known algorithms.

The paper studies the problem of active hypothesis testing with limited information. More specifically, denoting the probability of ‘correct indication’ by q(j,w) if the true hypothesis is j and the chosen action is w, they assume that q(j,w) is bounded below by \alpha(w) for all j, and this quantity is available to the controller. They first study the incomplete-bayesian update rule, where all q’s are replaced with alpha’s, hence approximate, and show that with this IB update rule, one needs at least O(log(1/\delta)) to achieve (1-\delta) posterior error probability. Then, they show that their gradient-based policy achieves the order-wise optimal sample complexity.
I have a few technical questions.
On the upper bound: The authors claim that the upper bound is obtained by considering the worst case scenario where q(w,j) = \alpha(w) for every w if j is the true hypothesis. If that’s the case, how is different the analysis given in this paper from that for Bayesian update case? (If my understanding is correct, q(w,j) = \alpha(w) implies IB update rule = Bayesian update rule.) Further, what can happen in reality is that q(w,j) is arbitrarily close to 1, and alpha(w) is arbitrarily close to 1/2. In this case, can the upper bound provide any guarantee?
On the lower bound: Can this bound recover the existing lower bounds for the case of complete information?
On simulation results: It will be great if the authors can present the lower & upper bounds on E[T] and compare them with the simulation results.


Bayesian Dyadic Trees and Histograms for Regression
Many machine learning tools for regression are based on recursive partitioning of the covariate space into smaller regions, where the regression function can be estimated locally. Among these, regression trees and their ensembles have demonstrated impressive empirical performance. In this work, we shed light on the machinery behind Bayesian variants of these methods. In particular, we study Bayesian regression histograms, such as Bayesian dyadic trees, in the simple regression case with just one predictor. We focus on the reconstruction of regression surfaces that are piecewise constant, where the number of jumps is unknown. We show that with suitably designed priors, posterior distributions concentrate around the true step regression function at a near-minimax rate. These results do not require the knowledge of the true number of steps, nor the width of the true partitioning cells. Thus, Bayesian dyadic regression trees are fully adaptive and can recover the true piecewise regression function nearly as well as if we knew the exact number and location of jumps. Our results constitute the first step towards understanding why Bayesian trees and their ensembles have worked so well in practice. As an aside, we discuss prior distributions on balanced interval partitions and how they relate to an old problem in geometric probability. Namely, we relate the probability of covering the circumference of a circle with random arcs whose endpoints are confined to a grid, a new variant of the original problem.

This paper analyses concentration rates (speed of posterior concentration) for Bayesian regression histograms and demonstrates that under certain conditions and priors, the posterior distribution concentrates around the true step regression function at the minimax rate.
The notation is clear. Different approximating functions are considered, starting from the set of step functions supported on equally sized intervals, up to more flexible functions supported on balanced partitions. The most important part of the paper is building the prior on the space of approximating functions.
The paper is relatively clear and brings up an interesting first theoretical result regarding speed of posterior concentration for Bayesian regression histograms. The authors assume very simple conditions (one predictor, piecewise-constant functions), but this is necessary in order to get a first analysis.
Proof seems correct, although it is out of this reviewer's expertise. This reviewer wonders why the authors did not considered sending this work to the Annals of Statistics instead of NIPS, given the type of analysis and provided results.
Minor:
- You might want to check the reference of Bayesian Hierarchical Clustering (Heller et.al, 2005) and the literature of infinite mixture of experts.
- l. 116: You mention recommendations for the choice of c_K in Section 3.3, but this Section does not exist.
- Assumption 1 is referred to multiple times in page 4, but it is not defined until page 5.
Writing (typos):
- l. 83: we will
- l. 231: mass near zero
- l. 296: (b) not clear (extend to multi-dimensional?)


Value Prediction Network
This paper proposes a novel deep reinforcement learning (RL) architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network. In contrast to typical model-based RL methods, VPN learns a dynamics model whose abstract states are trained to make option-conditional predictions of future values (discounted sum of rewards) rather than of future observations. Our experimental results show that VPN has several advantages over both model-free and model-based baselines in a stochastic environment where careful planning is required but building an accurate observation-prediction model is difficult. Furthermore, VPN outperforms Deep Q-Network (DQN) on several Atari games even with short-lookahead planning, demonstrating its potential as a new way of learning a good state representation.

This paper proposes a new learning architecture, VPN, whose components can be
learned end-to-end to perform planning over pre-specified temporally extended actions. An advantage of this approach is that it explicitly avoids working in the original observation space and plans indirectly over a learned representation.
This is made possible by a applying a model-free DQN-like approach for learning values which are then used in combination with reward and duration models of options
to form planning targets. The proposed approach is shown to outperform DQN in a
variety of Atari games.
Model-based RL is a difficult problem when it comes to a large domain like ALE.
The development of new model-based methods which do not explicitly try to
reconstruct/predict the next observation seem to be going in the right direction.
VPN is a contribution in this new line of research.
However, there are a few conceptual elements which remain unclear.
First, why predicting the expected discount/duration separately from the
option-conditional expected cumulative return to termination ? Using the
Bellman-like equations for the reward model of options (see equation 17 of the options paper), the discount factor can effectively be folded into the reward prediction. This would avoid the need to learn a separate discount prediction for each option.
Second : The nature of the "next abstract state" (line 109) is a key element of the
proposed approach to avoid learning a transition model directly
in the original space. As you mention later on line 251, there is no incentive/constraint for the learned state representation to capture useful properties for planning and could in the extreme case become a tabular representation. The appendix then shows that you use a particular architecture for the transition module which predicts the difference. How much this prior on the architecture affects quality of the representation learned by the system ? What kind of representation would be learned in the absence of a residual connection ?
Can you also elaborate on the meaning of the weighting (1/d, d-1/d)  in equation (1) ? Why is it necessary ? Where does that come from ?
Line 21 : "predict abstractions of observations"
This expression is unclear, but I guess that it means a feature/representation
of the state. You also write : "not clear how to roll such predictions", but isn't that exactly what VPN is doing ?
Typo : line 31 "dyanmics"


From Bayesian Sparsity to Gated Recurrent Nets
The iterations of many first-order algorithms, when applied to minimizing common regularized regression functions, often resemble neural network layers with prespecified weights. This observation has prompted the development of learningbased approaches that purport to replace these iterations with enhanced surrogates forged as DNN models from available training data. For example, important NPhard sparse estimation problems have recently benefitted from this genre of upgrade, with simple feedforward or recurrent networks ousting proximal gradient-based iterations. Analogously, this paper demonstrates that more powerful Bayesian algorithms for promoting sparsity, which rely on complex multi-loop majorizationminimization techniques, mirror the structure of more sophisticated long short-term memory (LSTM) networks, or alternative gated feedback networks previously designed for sequence prediction. As part of this development, we examine the parallels between latent variable trajectories operating across multiple time-scales during optimization, and the activations within deep network structures designed to adaptively model such characteristic sequences. The resulting insights lead to a novel sparse estimation system that, when granted training data, can estimate optimal solutions efficiently in regimes where other algorithms fail, including practical direction-of-arrival (DOA) and 3D geometry recovery problems. The underlying principles we expose are also suggestive of a learning process for a richer class of multi-loop algorithms in other domains.

This paper proposes a recurrent network for sparse estimation inspired on sparse Bayesian learning (SBL). It first shows that a recurrent architecture can implement a variant of SBL, showing through simulations that different quantities have different time dynamics, which motivates leveraging ideas from recurrent-network design to adapt the architecture. This leads to a recurrent network that seems to outperform approaches based on optimization in simulations and two applications.
Strengths: The idea of learning a recurrent network for sparse estimation has great potential impact. The paper is very well written. The authors motivate their design decisions in detail and report numerical experiments that are quite thorough and indicate that the technique is successful for challenging sparse-decomposition problems.
Weaknesses: A lot of the details about the implementation of the method and the experiments are deferred to the supplementary material, so that the main paper is a bit vague. I find this understandable due to length limitations.


Convolutional Phase Retrieval
We study the convolutional phase retrieval problem, which considers recovery of an unknown signal x ∈ C n from m measurements consisting of the magnitude of its cyclic convolution with a known kernel a of length m. This model is motivated by applications to channel estimation, optics, and underwater acoustic communication, where the signal of interest is acted on by a given channel/filter, and phase information is difficult or impossible to acquire. We show that when a is random and m is sufficiently large, x can be efficiently recovered up to a global phase using a combination of spectral initialization and generalized gradient descent. The main challenge is coping with dependencies in the measurement operator; we overcome this challenge by using ideas from decoupling theory, suprema of chaos processes and the restricted isometry property of random circulant matrices, and recent analysis for alternating minimizing methods.

The authors of the article consider phase retrieval problems where the measurements have a specific form: they are the convolution of the n-dimensional input signal x with a random filter of length m. The authors show that, provided that m is larger than O(n), up to log factors, and up to a term that is related with the sparsity of x in the Fourier domain, a simple non-convex algorithm can solve such problems with high probability.
The proof is quite technical. Indeed, the measurement vectors are highly correlated with one another. To overcome this difficulty, the authors use "decoupling techniques", together with results from [Rauhut, 2010] and [Krahmer, Mendelson, Rauhut, 2014].
I liked this article. I had not realized it was so complex to prove the convergence of a simple non-convex phase retrieval method in a very structured case, and, in my opinion, it is good that it has been done. The result by itself is interesting, and the proof techniques developed by the authors seem general enough to me so that they can be reused in other settings.
As previously said, the proof is quite technical, and long. There are 38 pages in the appendix. I tried to read as much of it as possible, but, given the time constraints of the review process, I certainly did not have the time to deeply understand everything. However, what I read seems both clear and solid.
The organization is also good. However, maybe the authors could discuss in more detail the case of coded diffraction patterns ([CLS15b]). Indeed, the case of phase retrieval with coded diffraction patterns is, at least at first sight, very similar to the case considered in the article: seen in the Fourier domain, coded diffraction patterns consist in convolving the input signal with (m/n) random filters of length n, while, in convolutional phase retrieval, the input signal is convolved with a single random filter of length m. I would appreciate the authors to compare their result and their proof techniques with the ones obtained for coded diffraction patterns.
Minor remarks and typos:
- Before Equation (4): add "that"?
- l.95 (and also in the "basic notations" paragraph of the appendix): ", because" should be ". Because".
- Paragraph 3.1: I did not know the name "alternating direction method". I think that "alternating projection method" is more common.
- Equation (12), and the previous two: "i\phi" is missing in the exponential function.
- l.180: ". Controlling" should be ", controlling".
- l. 209: "are not" should be "is not".
- Equation (19) (and the one after Equation (46) in the appendix): it took me a while to understand why this equation was true. Maybe it could be proved?
- l.220: "this control obtains" should be "we obtain this control".
- Equations (21) and (23): \mathcal{Q} should be \hat\mathcal{L}.
- The paragraph starting at line 244 seems a bit redundant with the explanations around Equation (22).
- Experiments: the main theorem says that, when x is proportional to the all-one vector, the initialization step still works, but the local descent does not. It would be nice to add an experiment that checks whether this is really what happens in numerical simulations.
- [CC15] has been published.
- The accent on "Candès" is missing in some of the bibliography entries.
- Appendix, l.58: "we proof" should be "we prove".
- Proposition 2.2: some "that" and "such that" could be added.
- l.92: "be" should be "is".
- l.150: I do not understand "for \gamma_2 functional of the set \mathcal{A}".
- Lemma 3.11: the last equation contains a \sqrt{\delta} while, in the statement, it is a \delta (no square root).
- l.202: I think that a "1/m" is missing in front of the expectation.
- l.217: "for w" should be "for all w".
- l.220: "define" should be "be defined".
- l.222: "define" should be "defined".
- l.224: "is constant" should be "is a constant".
- Equation after line 226: I think that there should be no parentheses inside the sup.
- Theorem 4.3: "are" should be "be".
- l.268: a word is missing before "that".
- Second line of the equation after line 278: the upper bound in the second integral should depend on d, not v.
- l. 279: "where the" should be "where in the", and "Choose" should be "choosing".
- Equation after line 292: I do not understand why the first inequality is true.
- l. 297: "so that" should be removed.
- l.305: "is complex" should be "is a complex".
- Equation just before line 326: there should be parentheses aroung the logarithms.
- Lemma 5.3: I do not understand the meaning of the first inequality: what is the difference between the first and the second term?
- l.466: "is close" should be "are close".
- l.481: "are defined" should be "is defined".
- l.484 and 486: "first" and "second" have been exchanged.
- I think that "we have [Equation] holds" (used multiple times) could be replaced with "we have that [Equation] holds" or simply "[Equation] holds".
- Equation after line 492: it could be said that it is a consequence of Lemma 6.2.
- Lemma 6.5, main equation: a multiplicative constant is missing before \delta.
- Equation after l.510, sixth line and after, until the end of the proof: the \leq sign should be a \geq sign.
- Equation after l.513: why w^{(r)}? Also, I think that there should maybe be close to 1 multiplicative constants before the last two terms of the sum.
- Equation after l.551: I do not understand the presence of ||x||.
- Lemma 6.3 and lemma 8.1 are not perfectly identical (there is a 2 in 6.3 that becomes a 2+\epsilon in 8.1).
- References: the name of the authors is missing in reference [led07].


QMDP-Net: Deep Learning for Planning under Partial Observability
This paper introduces the QMDP-net, a neural network architecture for planning under partial observability. The QMDP-net combines the strengths of model-free learning and model-based planning. It is a recurrent policy network, but it represents a policy for a parameterized set of tasks by connecting a model with a planning algorithm that solves the model, thus embedding the solution structure of planning in a network learning architecture. The QMDP-net is fully differentiable and allows for end-to-end training. We train a QMDPnet on different tasks so that it can generalize to new ones in the parameterized task set and "transfer" to other similar tasks beyond the set. In preliminary experiments, QMDP-net showed strong performance on several robotic tasks in simulation. Interestingly, while QMDP-net encodes the QMDP algorithm, it sometimes outperforms the QMDP algorithm in the experiments, as a result of end-to-end learning.

The paper proposes a novel policy network architecture for partially observable environments. The network includes a filtering module and a planning module. The filtering module mimics computation of the current belief of the agent given its previous belief, the last action and the last observation. The model of the environment and the observation function are replaced with trainable neural modules. The planning module runs value iteration for an MDP, whose transition function and reward function are also trainable neural modules. The filtering model and the planning module are stacked to resemble the QMDP algorithm for solving POMDP. All the modules are trained jointly according to an imitation learning criterion, i.e. the whole network is trained to predict the next action of an expert given the past observations and actions. The experimental evaluation shows better generalization to new tasks.
The approach can be as an extension of Value Iteration Networks (VIN) to partially observable environments by means of adding the filtering network. The paper is in general well written. I did not understand though why the internal model of the network does not use the same observation and action spaces as the real model of the world. Sentences like "While the belief image is defined in M, action inputs are defined in \hat{M}" were therefore very hard to understand. The choice of notation f^t_T is quite confusing.
The experimental evaluation seems correct. I would suggest a convolutional LSTM as another baseline for the grid world. A fully-connected LSTM is obviously worse eqipped than the proposed model to handle their local structure. I am somewhat suspicious if the fact that expert trajectories came from QMDP could give QMDP-net an unfair advantage, would be good if the paper clearly argued why it is not the case.
One limitation is this line of research (including VIN) is that it assumes discrete state spaces and the amount of computation is proportional to the number of states. As far as I can I see this makes this approach not applicable to real-world problems with high-dimensional state spaces.
Overall, the paper proposes a novel and elegant approach to build policy networks and validates it experimentaly. I recommend to accept the paper.


Learning Neural Representations of Human Cognition across Many fMRI Studies
Cognitive neuroscience is enjoying rapid increase in extensive public brain-imaging datasets. It opens the door to large-scale statistical models. Finding a unified perspective for all available data calls for scalable and automated solutions to an old challenge: how to aggregate heterogeneous information on brain function into a universal cognitive system that relates mental operations/cognitive processes/psychological tasks to brain networks? We cast this challenge in a machine-learning approach to predict conditions from statistical brain maps across different studies. For this, we leverage multi-task learning and multi-scale dimension reduction to learn low-dimensional representations of brain images that carry cognitive information and can be robustly associated with psychological stimuli. Our multi-dataset classification model achieves the best prediction performance on several large reference datasets, compared to models without cognitive-aware low-dimension representations; it brings a substantial performance boost to the analysis of small datasets, and can be introspected to identify universal template cognitive concepts.
Due to the advent of functional brain-imaging technologies, cognitive neuroscience is accumulating quantitative maps of neural activity responses to specific tasks or stimuli. A rapidly increasing number of neuroimaging studies are publicly shared (e.g., the human connectome project, HCP [1]), opening the door to applying large-scale statistical approaches [2]. Yet, it remains a major challenge to formally extract structured knowledge from heterogeneous neuroscience repositories. As stressed in [3], aggregating knowledge across cognitive neuroscience experiments is intrinsically difficult due to the diverse nature of the hypotheses and conclusions of the investigators. Cognitive neuroscience experiments aim at isolating brain effects underlying specific psychological processes: they yield statistical maps of brain activity that measure the neural responses to carefully designed stimulus. Unfortunately, neither regional brain responses nor experimental stimuli can be considered to be atomic: a given experimental stimulus recruits a spatially distributed set of brain regions [4], while each brain region is observed to react to diverse stimuli. Taking advantage of the resulting data richness to build formal models describing psychological processes requires to describe each cognitive
conclusion on a common basis for brain response and experimental study design. Uncovering atomic basis functions that capture the neural building blocks underlying cognitive processes is therefore a primary goal of neuroscience [5], for which we propose a new data-driven approach.
Several statistical approaches have been proposed to tackle the problem of knowledge aggregation in functional imaging. A first set of approaches relies on coordinate-based meta-analysis to define robust neural correlates of cognitive processes: those are extracted from the descriptions of experimentsbased on categories defined by text mining [6] or expert [7]-and correlated with brain coordinates related to these experiments. Although quantitative meta-analysis techniques provide useful summaries of the existing literature, they are hindered by label noise in the experiment descriptions, and weak information on brain activation as the maps are reduced to a few coordinates [8]. A second, more recent set of approaches models directly brain maps across studies, either focusing on studies on similar cognitive processes [9], or tackling the entire scope of cognition [10,11]. Decoding, i.e. predicting the cognitive process from brain activity, across many different studies touching different cognitive questions is a key goal for cognitive neuroimaging as it provides a principled answer to reverse inference [12]. However, a major roadblock to scaling this approach is the necessity to label cognitive tasks across studies in a rich but consistent way, e.g., building an ontology [13].
We follow a more automated approach and cast dataset accumulation into a multi-task learning problem: our model is trained to decode simultaneously different datasets, using a shared architecture. Machine-learning techniques can indeed learn universal representations of inputs that give good performance in multiple supervised problems [14,15]. They have been successful, especially with the development of deep neural network [see, e.g., 16], in sharing representations and transferring knowledge from one dataset prediction model to another (e.g., in computer-vision [17] and audioprocessing [18]). A popular approach is to simultaneously learn to represent the inputs of the different datasets in a low-dimensional space and to predict the outputs from the low-dimensional representatives. Using very deep model architectures in functional MRI is currently thwarted by the signal-to-noise ratio of the available recordings and the relative little size of datasets [19] compared to computer vision and text corpora. Yet, we show that multi-dataset representation learning is a fertile ground for identifying cognitive systems with predictive power for mental operations.
Contribution. We introduce a new model architecture dedicated to multi-dataset classification, that performs two successive linear dimension reductions of the input statistical brain images and predicts psychological conditions from a learned low-dimensional representation of these images, linked to cognitive processes. In contrast to previous ontology-based approaches, imposing a structure across different cognitive experiments is not needed in our model: the representation of brain images is learned using the raw set of experimental conditions for each dataset. To our knowledge, this work is the first to propose knowledge aggregation and transfer learning in between functional MRI studies with such modest level of supervision. We demonstrate the performance of our model on several openly accessible and rich reference datasets in the brain-imaging domain. The different aspects of its architecture bring a substantial increase in out-of-sample accuracy compared to models that forgo learning a cognitive-aware low-dimensional representation of brain maps. Our model remains simple enough to be interpretable: it can be collapsed into a collection of classification maps, while the space of low-dimensional representatives can be explored to uncover a set of meaningful latent components.

This paper introduces a framework to analyze heterogeneous fMRI datasets combining multi-task supervised learning with multi-scale dictionary learning. The result shows that classification of task conditions in a target dataset can be improved by integrating the large-scale database of humman connectome project (HCP) using their method.
The proposed entire framework looks fairly reasonable to me. However, the method contains many different technical ideas and each of them is not necessarily well validated. The technical ideas include at least the following: 1) use of resting state data instead of task data for training the first stage; 2) use of multi-scale spatial dictionary combining multiple runs of sparse dictionary learning; 3) using Dropout for regularization; and 4) adding HCP data to improve classification on another target dataset. In my view, current results only support point 4. The following comments are related to the other points.
In the experiment (Sec2.2), the comparison between single and multi-scale dictionaries (methods 2 and 3) seems to be made for different dictionary sizes (256 and 336). However, I think they should be balanced for a fair comarison (for instance, learn 336 atoms at once for the non-multi-scale dictionary), since currently the effects of having different dimensionalities and different spatial scales cannot be dissociated. I would also suggest comparing with simply using PCA to reduce to the same dimensionality, which may clarify the advantage of using localized features.
Moreover, the methods 1-3 use L2-regularization but 4-5 uses Dropout, which may also not be a fair comparison. What happens if you use L2-regualarization for methods 4-5, or otherwise Dropout for 1-3?
I also wondered what happens if the spatial dictionary is trained based on the task data rather than resting-state data. The use of rest data for training sounds reasonable for the goal of "uncovering atomic bases" (Introduction) but it would be necessary to compare the two cases in terms of both test accuracy and interpretability. In particular, can similar spatial maps in Figs4,5 be obtained even without resting-state decomposition?
Other minor comments:
- Eq1,3: what does the objective actually mean, when taking expectation with respect to M? Is it correct that the objective is minimized rather than maximized? - Eq3: "log" seems to be dropped.
- Fig2: what are the chance levels for each dataset? No result by "full multinomial" for LA5c.
- Line235: what does "base condition" mean?


Multiplicative Weights Update with Constant Step-Size in Congestion Games: Convergence, Limit Cycles and Chaos
The Multiplicative Weights Update (MWU) method is a ubiquitous meta-algorithm that works as follows: A distribution is maintained on a certain set, and at each step the probability assigned to action γ is multiplied by (1 − C(γ)) > 0 where C(γ) is the "cost" of action γ and then rescaled to ensure that the new values form a distribution. We analyze MWU in congestion games where agents use arbitrary admissible constants as learning rates and prove convergence to exact Nash equilibria. Interestingly, this convergence result does not carry over to the nearly homologous MWU variant where at each step the probability assigned to action γ is multiplied by (1 − ) C(γ) even for the simplest case of two-agent, two-strategy load balancing games, where such dynamics can provably lead to limit cycles or even chaotic behavior.

The paper revisits the convergence result of multiplicative weighted update (MWU) in a congestion game, due to Kleinberg, Piliouras, and Tardos (STOC 2009), and establishes a connection between MWU and Baum-Welch algorithm in a neat and elegant style. By showing the monotonicity of the potential function in a congestion game, the authors prove that any MWU with linear updating rule converges to the set of fixed points, which is a superset of all the Nash equilibria of the congestion game. The Baum-Eagon inequality offers a new interpretation of the dynamics of the linear updating rule in MWU and their results in congestion games are quite general, and so are the conditions on initialization and isolated Nash equilibrium in a congestion game with finite set of agents and pure strategies.
The results in this paper hold for any congestion game irrespective of the topology of the strategy sets by the nature of their game, however,  one should emphasize the assumption that, individual earning rates $\epsilon_i$ are bounded above, in order for Baum-Eagon inequality to work in their context. An elaborate analysis on the upper bounds would be nicer, since for any $i$, this paper requires that
$$\frac{1}{\epsilon_i} >  \sup_{i,\mathbf{p}\in \Delta,\gamma\in S_i} \{c_{i\gamma}\}\geq \max_{e} c_e(|N|),$$ which can be huge when the number of agents is large enough.
Although Theorem 3.7 holds, the proof presented by the authors is flawed. To prove any point $\mathbf{y} \in \Omega$ is a fixed point of $\zeta$, one could guess from the argument in the paper that the authors intend to show that $\Phi(\mathbf{y}) = \Phi(\zeta(\mathbf{y}))$, thus by monotonicity of $\Phi$, $\zeta(\mathbf{y})=\mathbf{y}$. However, they applied the notation $\mathbf{y}(t)$, even though $\mathbf{y}$ is, according to their definition, the limit point of an orbit $\mathbf{p}(t)$ as $t$ goes to infinity. The subsequent argument is also hard to follow.
A easy way to prove theorem 3.7 is by contradiction. Suppose that $\mathbf{y}$ is not a fixed point, then by definition (line221), there exists some $i,\gamma$ s.t., $y_{i\gamma} > 0$ and $c_{i\gamma}\neq \hat{c}_i$.  If $c_{i\gamma} > \hat{c}_i$, then by convergence and continuity of the mapping, $\exists t_0$ and $\epsilon_0 > 0$, s.t., $\forall t > t_0$, $p_{i\gamma}(t) \geq \epsilon_0 > 0$, and $c_{i\gamma}(t) \geq  \hat{c}_i(t)+ \epsilon_0.$ Then it's not hard to see that
$$p_{i\gamma}(t+1) = p_{i\gamma}(t) \frac{\frac{1}{\epsilon_i} - c_{i\gamma}(t) }{\frac{1}{\epsilon_i} - \hat{c}_{i}(t)} \leq  p_{i\gamma}(t) (1- \epsilon_0 \epsilon_i),$$ which yields $\lim_{t\to \infty} p_{i\gamma}(t) = y_{i\gamma}=0$, contradicting with $y_{i\gamma} > 0$.
If $c_{i\gamma} < \hat{c}_i$, then by convergence and continuity of the mapping, $\exists t_0$ and $\epsilon_0 > 0$, s.t., $\forall t > t_0$, $p_{i\gamma}(t) \geq \epsilon_0 > 0$, and $c_{i\gamma}(t) + \epsilon_0 <  \hat{c}_i(t).$ Then it's not hard to see that $$p_{i\gamma}(t+1) = p_{i\gamma}(t) \frac{\frac{1}{\epsilon_i} - c_{i\gamma}(t) }{\frac{1}{\epsilon_i} - \hat{c}_{i}(t)} \geq  p_{i\gamma}(t) (1+ \epsilon_0)$$. Then $\lim_{t\to \infty} p_{i\gamma}(t)$ doesn't lie in the simplex.
In Remark 3.5, the authors claim that Lemma 3.3 and thus Theorem 3.4 and 3.1 hold in the extension of weighted potential games. However, a underlying step in the proof of Lemma 3.3 might not hold without adding extra conditions: the non-negativity of coefficients in $Q(p)$ in a weighted potential game. Similar to the proof of Lemma 3.3, in the weighted case, $\frac{\partial Q}{\partial p_{i\gamma}} = \frac{1}{\epsilon_i} - \frac{c_{i\gamma}}{w_i}$. Whether the RHS is non-negative is unknown simply by the definition of $\epsilon_i$ nor the fact that $w_i \in [0,1]$.
On Line 236, the constant in $Q(p)$ should rather be $Q(p)=const -\Phi$ and $const = \sum_{i\in N} 1/{\epsilon_i} + (|N|-1) 1/\beta$.


Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
Evaluating a policy by deploying it in the real world can be risky and costly. Off-policy policy evaluation (OPE) algorithms use historical data collected from running a previous policy to evaluate a new policy, which provides a means for evaluating a policy without requiring it to ever be deployed. Importance sampling is a popular OPE method because it is robust to partial observability and works with continuous states and actions. However, the amount of historical data required by importance sampling can scale exponentially with the horizon of the problem: the number of sequential decisions that are made. We propose using policies over temporally extended actions, called options, and show that combining these policies with importance sampling can significantly improve performance for long-horizon problems. In addition, we can take advantage of special cases that arise due to options-based policies to further improve the performance of importance sampling. We further generalize these special cases to a general covariance testing rule that can be used to decide which weights to drop in an IS estimate, and derive a new IS algorithm called Incremental Importance Sampling that can provide significantly more accurate estimates for a broad class of domains.

[I have read the other reviews and the author feedback, and updated my rating to be borderline accept. The paper will be substantially stronger with more realistic expts.
In the absence of such expts, I suspect that the IncrIS estimator will run into substantial computational challenges and sub-optimal MSE in realistic scenarios.]
The paper introduces new off-policy estimators to evaluate MDP policies. The options-based estimator of Theorem3 is a straightforward adaptation of per-step importance sampling.
Section6 makes an interesting observation: if at any step in the trajectory we know that the induced state distribution is the same between the behavior policy and the evaluation policy, we can partition the trajectory into two sub-parts and conduct per-step importance sampling on each sub-part. This observation may prove to be more generally useful; in this paper, they exploit this observation to propose the IncrIS estimator.
IncrIS uses sample variance and covariance estimates (presumably; this detail is missing in their experiment description) to detect what prefix of sampled trajectories can be safely ignored [i.e. yields smaller (estimated) MSE of per-step importance sampling on the resulting suffix]. This is an interesting approach to bias-variance trade-offs in importance-sampling-estimator design that can be explored further; e.g. covariance testing requires us to fix the choice of k (ignored prefix length) across the entire set of trajectories. Can we condition on the observed trajectories to adaptively discard importance weights?
In summary, the contributions up to Section 6 seem very incremental. IncrIS introduces an intriguing bias-variance trade-off but the experimental evidence (MSE in a constructed toy MDP)is not very convincing. In realistic OPE settings, it is likely that horizons will be large (so IncrIS can shine), but there may be lots of sampled trajectories (computational challenge for IncrIS to compute \hat{MSE}), and we almost certainly won't be in the asymptotic regime where we can invoke strong consistency of IncrIS (so, \hat{MSE} may yield severely sub-optimal choices, and IncrIS may have poor empirical MSE). Due to these intuitions, a more realistic experiment will be much more informative in exploring the strengths and weaknesses of the proposed estimators.
Minor:
66: \ldots
112: one *can* leverage *the* advantage?


Online Learning of Optimal Bidding Strategy in Repeated Multi-Commodity Auctions
We study the online learning problem of a bidder who participates in repeated auctions. With the goal of maximizing his T-period payoff, the bidder determines the optimal allocation of his budget among his bids for K goods at each period. As a bidding strategy, we propose a polynomial-time algorithm, inspired by the dynamic programming approach to the knapsack problem. The proposed algorithm, referred to as dynamic programming on discrete set (DPDS), achieves a regret order of O( √ T log T ). By showing that the regret is lower bounded by Ω( √ T ) for any strategy, we conclude that DPDS is order optimal up to a √ log T term. We evaluate the performance of DPDS empirically in the context of virtual trading in wholesale electricity markets by using historical data from the New York market. Empirical results show that DPDS consistently outperforms benchmark heuristic methods that are derived from machine learning and online learning approaches.

This paper  studies the online learning (stochastic and full-information) problem of bidding in multi commodity first price auctions. The paper introduces a polynomial time algorithm that achieves a regret of \sqrt{T log(T)} that has a near optimal dependence on T.
The main challenge that the paper has to deal with is to find a computationally efficient algorithm for computing the best biding strategy given a known distribution.The authors first demonstrate that natural approaches for solving this problem exactly are not computationally efficient (this is not a formal np-hardness proof). Then, they provide a FPTAS for solving the problem using dynamic programming. Once they have a FPTAS for the offline problem, their results hold for the stochastic online setting using existing reductions. I haven’t carefully looked in to the details of their analysis of the dynamic programming, but I think the effectiveness of it here is interesting and surprising — specially given that the variation of this problem for the second price auctions is hard to approximate.
As for the weaknesses, I find the theorems to be hard to interpret. Particularly because there are a lot of variables in them. The authors should provide one simple variation of theorem 1 with fewer parameters. For example, when l = 0, and u = 1, the lipschitz conditions holds with L = 1 and p = 1, and B = K. Assigning \alpha = \sqrt{t} gives a regret of \tilde O(\sqrt{t \log(T)}).
As mentioned above the results mentioned here hold for the full-information online stochastic setting, that is, the valuation and the clearing price are drawn from a the same distribution at every step. I find this to be a weird choice of a setting. In most cases, full information settings are accompanied by adversarial arrival of the actions, rather than stochastic. This is indeed due to the fact that the statistical and computational aspects of online stochastic full-information setting is very similar to the offline stochastic full-information setting. So, I think a much more natural setting would have been to consider the adversarial online setting.
In this regard, there are two very relevant works on no-regret learning in auctions that should be cited here. Daskalakis and Syrgkanis FOCS’16 and later on Dudik et al. FOCS’17 study the problem of online learning in Simultaneous Second Price auctions (SiSPA). This is also the problem of optimal bidding in a repeated multi-commodity auction where each item is assigned based on a second price auctions, rather than a first price auction that is considered in this paper. In particular, the latter paper discusses computationally efficient mechanisms for obtaining a no-regret algorithm in the adversarial setting when one can solve the offline problem efficiently. Their methods seem to apply to the problem discussed in this paper and might be specially useful since this paper provides an approximation method for solving the offline problem. It is worth seeing whether the machinery used in their paper can directly strengthen your result so that it holds for the adversarial online setting.
After rebuttal:
I suggest that the authors take a closer look at the works of Dudik et al. and Daskalakis and Syrgkanis. In their response, the authors mention that those works are inherently for discrete spaces. From a closer look at the work of Dudik et al. it seems that they also work with some auctions specifically in the continuous space by first showing that they can discretize the auction space. This discretization step is quite common in no-regret learning in auctions. Given that the current submission also shows that one can discretize the auctions space first, I think the results of Dudik et al. can be readily applied here. This would strengthen the results of this paper and make the setting more robust. I suggest that the authors take a closer look at these works when preparing their discussion.


Optimized Pre-Processing for Discrimination Prevention
Non-discrimination is a recognized objective in algorithmic decision making. In this paper, we introduce a novel probabilistic formulation of data pre-processing for reducing discrimination. We propose a convex optimization for learning a data transformation with three goals: controlling discrimination, limiting distortion in individual data samples, and preserving utility. We characterize the impact of limited sample size in accomplishing this objective. Two instances of the proposed optimization are applied to datasets, including one on real-world criminal recidivism. Results show that discrimination can be greatly reduced at a small cost in classification accuracy.

This paper presents a constrained optimization framework to address (ethically/socially sensitive) discrimination in a classification setting. Such discrimination can occur in machine learning because (1) the training data was obtained from an unfair world and reflects that unfairness, (2) the algorithm trained on the data may introduce biases or unfairness even if there are no issues in the training data, or (3) the output is used in a discriminatory fashion. The present paper falls in the category of preprocessing, meaning it attempts to address unfairness coming from (1) by transforming the data to reduce or eliminate its discriminatory aspects.
As notation, D represents one or more variables of a sensitive nature for which we wish to reduce discrimination, such as race or sex, X represents other predictors, and Y the outcome variable for (binary) classification. Existing literature on fairness and discrimination contains a variety of formal definitions of (un)fairness. Two such definitions are used here, (I) dependence of the predicted outcome \hat Y on D should be low, and (II) similar individuals should receive similar predicted outcomes. These two criteria form the constraints of the optimization problem posed in this paper, and the objective is to minimize a probability distance measure between the joint distribution of the transformed data and the joint distribution of the original data. The paper gives conditions under which the optimization problem is (quasi)convex, and proves a result bounding errors resulting from misspecification of the prior distribution of the data. The utility of the method is demonstrated empirically on two datasets and shown to be competitive with the most similar existing method of Zemel et al. (2013).
One important limitation of this paper is the assumption that the joint distribution of the data is known. Proposition 2 is a good attempt at addressing this limitation. However, there may be subtle issues related to fairness encoded in the joint distribution which are only hinted at in the Supplementary Material in Figures 2-4. For example, what does it mean for two individuals with different values of D to be similar otherwise? Are there important aspects of similarity not captured in the available data--i.e. confounding variables? These questions are fundamental limitations of individual fairness (IF), and while I may have objections to (IF) I recognize it has been accepted in previous literature and is not a novel proposal of the present work. However, even accepting (IF) without objections, Proposition 2 appears only to address the other two aspects of the optimization problem, and does not address the effect of prior distribution misspecification on (IF). If this could be changed it would strengthen the paper. Some acknowledgement of the subtlety in defining fairness--issues concerning the relations between D and other variables in X, and the implications of those relations on Y, which are addressed fairly implicitly in (IF)--would also strengthen the exposition. See, for example, Johnson et al. "Impartial Predictive Modeling: Ensuring Fairness in Arbitrary Models," or Kusner et al. "Counterfactual Fairness."
Another limitation of the present work is the large amount of calibration or tuning parameters involved. There are tolerances for each constraint, and the potentially contradictory nature of the two kinds of constraints (discrimination vs distortion) is not acknowledged except in the Supplementary Materials where Figure 1 shows some constraint tolerances may be infeasible. Addressing this in the main text and giving conditions (perhaps in the Supplementary Materials) under which the optimization problem is feasible would also strengthen the paper.
Finally, the choice of the distortion metric is both a subtle issue where the fundamental goal of fairness is entirely implicit--a limitation of (IF)--and a potentially high dimensional calibration problem. The metric can be customized on a per-application basis using domain and expert knowledge, which is both a limitation and a strength. How should this be done so the resulting definition of (IF) is sensible? Are there any guidelines or default choices, or references to other works addressing this? Figure 2 is rather uninformative since neither the present method nor its competitor have been tuned. The comparison is (probably) fair, but the reader is left wondering what the best performance of either method might look like, and whether one strictly dominates the other.
* Quality
The paper is well supported by theoretical analysis and empirical experiments, especially when the Supplementary Materials are included. The work is fairly complete, though as mentioned above it is both a strength and a weakness that there is much tuning and other specifics of the implementation that need to be determined on a case by case basis. It could be improved by giving some discussion of guidelines, principles, or references to other work explaining how tuning can be done, and some acknowledgement that the meaning of fairness may change dramatically depending on that tuning.
* Clarity
The paper is well organized and explained. It could be improved by some acknowledgement that there are a number of other (competing, often contradictory) definitions of fairness, and that the two appearing as constraints in the present work can in fact be contradictory in such a way that the optimization problem may be infeasible for some values of the tuning parameters.
* Originality
The most closely related work of Zemel et al. (2013) is referenced, the present paper explains how it is different, and gives comparisons in simulations. It could be improved by making these comparisons more systematic with respect to the tuning of each method--i.e. compare the best performance of each.
* Significance
The broad problem addressed here is of the utmost importance. I believe the popularity of (IF) and modularity of using preprocessing to address fairness means the present paper is likely to be used or built upon.


Nonlinear Acceleration of Stochastic Algorithms
Extrapolation methods use the last few iterates of an optimization algorithm to produce a better estimate of the optimum. They were shown to achieve optimal convergence rates in a deterministic setting using simple gradient iterates. Here, we study extrapolation methods in a stochastic setting, where the iterates are produced by either a simple or an accelerated stochastic gradient algorithm. We first derive convergence bounds for arbitrary, potentially biased perturbations, then produce asymptotic bounds using the ratio between the variance of the noise and the accuracy of the current point. Finally, we apply this acceleration technique to stochastic algorithms such as SGD, SAGA, SVRG and Katyusha in different settings, and show significant performance gains.

Summary of the paper
====================
In principle, the extrapolation algorithm devised by Scieur et al. (2016) forms a scheme which allows one to speed up a given optimization algorithm by averaging iterates. The current paper extends this idea to the stochastic case by using the same the averaging technique only this time for noisy iterates. After recalling the results for the deterministic case (as shown in Scieur et al. (2016)), a theoretical upper bound is derived, based on theses results, by quantifying the amount of noise introduced into the optimization process. Finally, numerical experiments which demonstrate the speed-up offered by this technique for least-square and finite sum problems are provided.
Evaluation
==========
Extending this technique for stochastic algorithms is a crucial step for this technique as this family of algorithms dominants many modern applications. From practical point of view, this seems to provide a significantly acceleration to some (but not all) of the state-of-the-art algorithms. Moreover, the theoretical analysis provided in the paper shows that this stochastic generalization effectively subsumes the deterministic case. However, this paper does not provide a satisfying analysis of the theoretical convergence rate expected by applying the Algorithm 1 on existing stochastic algorithms (especially, for finite sum problems). It is therefore hard to predict in what settings, for which algorithms and to what extent should we expect acceleration. In my opinion, this forms a major weakness point of this paper.
General Comments
================
- As mentioned above, I would suggest providing a more comprehensive discussion which allows more intuition regarding the benefits of this technique with respect to other existing algorithms.
- Consider adding a more precise statement to address the theoretical convergence rate expected when applying this technique to a given optimization algorithm with a given convergence rate.
Minor Comments
==============
L13: f(x) -> f ?
L13: Smooth and strong convexity are not defined\addressed at this point of the paper.
L25: depends on
L31: Many algorithms do exhibit convergence properties which are adaptive to the effective strong convexity of the problem: vanilla gradient descent, SAG (https://arxiv.org/abs/1309.2388, Page 8, 'an interesting consequence').
L52: f(x) -> f ?
L55: Might be somewhat confusing to diverse from the conventional way of defining condition number.
L61: The linear approximation may not be altogether trivial (also may worth to mention that the gradient vanishes..)
L72: in the description of Algorithm 1, using boldface font to denote the vector of all ones is not aligned with vector notation used throughout the paper, and thus, is not defined a-priori.
L81: can you elaborate more on why 'Convergence is guaranteed as long as the errors' implies to 'ensures a good rate of decay for the regularization parameter'.
L103: lambda..
L118: I would split the technical statement from the concrete application fro SGD.
L121: Maybe elaborate more on the technical difficulty you encounter to prove a 'finite-step' version without lim?
L136: G is symmetric?
L137: corresponds?
L137: I'd suggest further clarifying the connection with Taylor reminder by making it more concrete.
L140: corresponds
L140: h in I-h A^\top A
L149: The
Figure 2 Legend: the 'acc' prefix is a bit confusing as 'acceleration' in the context of continuous optimization this term is used somewhat differently.
L227: Can you give more intuition on why 'the algorithm has a momentum term' implies 'underlying dynamical system is harder to extrapolate.'?


Inverse Filtering for Hidden Markov Models
This paper considers a number of related inverse filtering problems for hidden Markov models (HMMs). In particular, given a sequence of state posteriors and the system dynamics; i) estimate the corresponding sequence of observations, ii) estimate the observation likelihoods, and iii) jointly estimate the observation likelihoods and the observation sequence. We show how to avoid a computationally expensive mixed integer linear program (MILP) by exploiting the algebraic structure of the HMM filter using simple linear algebra operations, and provide conditions for when the quantities can be uniquely reconstructed. We also propose a solution to the more general case where the posteriors are noisily observed. Finally, the proposed inverse filtering algorithms are evaluated on real-world polysomnographic data used for automatic sleep segmentation.

The paper addresses recovery of the observation sequence given known posterior state estimates, but unknown observations and/or sensor model and also in an extension, noise-corrupted measurements.  There is a nice progression of the problem through IP, LP, and MILP followed by a more careful analytical derivation of the answers in the noise-free case, and a seemingly approximate though empirically effective approach (cf. Fig 2) using spherical K-means as a subroutine in the noisy case.
Honestly, most of the motivations seem to be unrealistic, especially the cyber-physical security setting where one does not observe posteriors, but simply an action based on a presumed argmax w.r.t. posteriors.  The EEG application (while somewhat narrow) seems to be the best motivation, however, the sole example is to compare resconstructed observations to a redundant method of sensing -- is this really a compelling application?  Is it actually used in practice?
Some additional details are provided on page 8, but this is not enough for me to fully understand the rationale for the approach.  One is given posterior state estimates of sleep stage for a new patient and the goal is to provide the corresponding mean EEG frequency observations?  I don't see the point.
Minor terminological question: I've always viewed the HMM to be a type of sequential graphical model structure allowing any type of random variable (discrete or continuous) of which discrete HMMs and Kalman filters (continuous Gaussian) are special cases.  Does the terminology "HMM" really assume discrete random variables?
Overall, though the motivation for the work is not very strong for me, the results in this paper address a novel problem, and make technically innovative and honestly surprising insights in terms of the tractability and uniqueness of the recovery problems addressed in the paper.  I see this paper as a very technically interesting solution that may find future application in use cases the authors have not considered.


Learning Overcomplete HMMs
We study the problem of learning overcomplete HMMs-those that have many hidden states but a small output alphabet. Despite having significant practical importance, such HMMs are poorly understood with no known positive or negative results for efficient learning. In this paper, we present several new results-both positive and negative-which help define the boundaries between the tractable and intractable settings. Specifically, we show positive results for a large subclass of HMMs whose transition matrices are sparse, well-conditioned, and have small probability mass on short cycles. On the other hand, we show that learning is impossible given only a polynomial number of samples for HMMs with a small output alphabet and whose transition matrices are random regular graphs with large degree. We also discuss these results in the context of learning HMMs which can capture long-term dependencies.

The paper studies the learnability of HMMs in the setting when the output label size m is smaller than the number of states n. This setting is particularly tricky since the usual full-rank assumption that comes up in tensor decomposition based methods is not valid. This paper gives both algorithms and information-theoretic lower-bounds on the learnability of overcomplete HMMs, which clearly increases our state of understanding.
1. They show that HMMs with m=polylog(n) even for transition-matrix being a random dense regular graphs, can not be learning using poly(n) samples even when the window size is poly(n).
2. They show that overcomplete HMMs can be learned efficiently with window size O(log_m n), under some additional four conditions (well-conditioned-ness of T, no-short-cycles, sparsity of graph given by T and random sparse output distributions)

Positives:
1. The lower bound is interesting and surprisingly strong. This is unlike the parity-based lower bounds in Mossel-Roch which are more computational by nature, and more similar to moment-based identifiability lower bounds known for Gaussian mixtures and multiview models (see Rabani-Swamy-Schulman).
2. This paper tries to understand in depth the effect of the structure of the transition matrix on learnability in the overcomplete setting.
3. The techniques developed in this paper are new and significantly depart from previous works. For instance, the algorithmic result using tensor decompositions departs significantly from previous techniques in analyzing the Khatri-Rao (KR) product. Analyzing overcomplete tensor decompositions and these KR products are challenging, and the only non-generic analysis I am aware of (by Bhaskara et al. for smoothed analysis) can not be used here since the different tensor modes are dependent. At the heart of this argument is an interesting coupling argument that shows that for any two different starting states, the random walk distributions after O(log_m n) steps will be significantly different if the observation matrices are random. This crucially depends on the well-conditioned-ness of T and no-short-cycle condition which they argue is needed for identifiability.
4. I like the discussion of assumptions section 2.4; while the paper assumes four conditions, and they give qualitative reasons for why most of them are necessary.
Negatives:
1. The proofs are fairly dense (especially Theorem 1 , Lemma 3) and the error analysis is hand-wavy at places . Though I have no reason to suspect the correctness, it would be good to improve the readability of these proofs.
2. It is not clear why some of the conditions are necessary. For instance, the random sparse support assumption seems stronger than necessary. Similarly, while the lower bound example in Proposition 1 explains why an example with just short cycles is not identifiable, this doesn't explain why there should be no short-cycles in a connected, well-conditioned T.

Summary: Overall, the paper presents a nice collection of results  that significantly improves our understanding on overcomplete HMMs. I think the techniques developed here may be of independent interest as well. I think the paper is easily above the bar.
Comments:
The identifiability results for HMMs using O(log_m n ) should also be attributed to Allman-Mathias-Rhodes'2010. They already show that the necessary conditions are true in a generic sense, and they introduced the algorithm based on tensor decompositions.
Typo in Section 2.2: "The techniques of Bhaskara et al. can be applied.." --> ".. can not be applied ..."


Nonlinear random matrix theory for deep learning
Neural network configurations with random weights play an important role in the analysis of deep learning. They define the initial loss landscape and are closely related to kernel and random feature methods. Despite the fact that these networks are built out of random matrices, the vast and powerful machinery of random matrix theory has so far found limited success in studying them. A main obstacle in this direction is that neural networks are nonlinear, which prevents the straightforward utilization of many of the existing mathematical results. In this work, we open the door for direct applications of random matrix theory to deep learning by demonstrating that the pointwise nonlinearities typically applied in neural networks can be incorporated into a standard method of proof in random matrix theory known as the moments method. The test case for our study is the Gram matrix
, where W is a random weight matrix, X is a random data matrix, and f is a pointwise nonlinear activation function. We derive an explicit representation for the trace of the resolvent of this matrix, which defines its limiting spectral distribution. We apply these results to the computation of the asymptotic performance of single-layer random feature networks on a memorization task and to the analysis of the eigenvalues of the data covariance matrix as it propagates through a neural network. As a byproduct of our analysis, we identify an intriguing new class of activation functions with favorable properties.

This looks like a good theoretical contribution and an interesting direction in the theory of deep learning to me.
In this paper, the authors compute the correlation properties (gram matrix) of the vector than went through some step of feedforward network with non linearities and random weights. Given the current interest in the theoretical description of neural nets, I think this is a paper that will be interesting for the NIPS audience and will be a welcome change between the hundreds of GAN posters. Some findings are particularly interesting.They applied their result to a memorization task and obtained an explicit characterizations of the training error. Also the fact that for some type of activation the eigenvalues of the data covariance matrix are constant  suggests interesting directions for the future.
Few comments and suggestions follows;:
- In section 1.3 : I think that the interest for Random Network is slightly older than the authors (maybe) know. In particular, these were standard setting in the 80s (more than 20 years before any work presented by the authors) and have generated a lot of attention, especially in the statistical physics community where the generalization properties were extensivly studied, (see, just to name a few:  -https://doi.org/10.1209/0295-5075/27/2/002 -https://doi.org/10.1088/0305-4470/21/1/030-10.1103/PhysRevLett.82.2975 )
- In connection with the random kitchen sinks: isn't the theory presented here a theory of what is (approximatly) going in in the Kernel space ? Given the connection between random projections and kernel methods there seems to be something interesting in this direction. Section  4.2 "Asymptotic performance of random feature methods" should propbably discuss this point. Perharps it is linked to the work of El Karoui, et al ?
- Can the author provided an example of an activation that leads to the 0 value for the parameter zeta? This is not clear from the definition (10) (or maybe I missed something obvious ?) is it just ANY even function?
I did not check all the computations in the supplementary materials (espcially the scary expansion in page 11!) but I am a bit confuse about the statement "t is easy to check many other examples which all support that eqn. (S19) is true, but a proof is still lacking." . Why is it called a lemma then? I am missing something?


Consistent Multitask Learning with Nonlinear Output Relations
Key to multitask learning is exploiting the relationships between different tasks in order to improve prediction performance. Most previous methods have focused on the case where tasks relations can be modeled as linear operators and regularization approaches can be used successfully. However, in practice assuming the tasks to be linearly related is often restrictive, and allowing for nonlinear structures is a challenge. In this paper, we tackle this issue by casting the problem within the framework of structured prediction. Our main contribution is a novel algorithm for learning multiple tasks which are related by a system of nonlinear equations that their joint outputs need to satisfy. We show that our algorithm can be efficiently implemented and study its generalization properties, proving universal consistency and learning rates. Our theoretical analysis highlights the benefits of non-linear multitask learning over learning the tasks independently. Encouraging experimental results show the benefits of the proposed method in practice.

The paper tackles multi-task learning problems where there are
non-linear relationships between tasks. The relationships between
tasks is encoded as a set of non-linear constraints that the outputs
of each task must satisfy (e.g . y1^2 + y2^2 = 1). In a nutshell, he
proposed technique can be summarized as: use kernel regression to make
predictions for each task independently, then project the prediction
vector onto the constrained set.
Overall, I like the idea of being able to take advantage of non-linear
relationships between tasks. However, I am not sure how practical it
is to specify the non-linear constraints between tasks in practice.
The weakest point of the paper is the empirical evaluation.
It would be good if the authors can discuss real applications where
one can specify a reasonable set non-linear relationships between
tasks. The examples provided in the paper are not very
satisfactory. One imposes deterministic relationships between tasks
(i.e. if the output for task 1 is known then the output for task 2 can
be inferred exactly), and the other approximates the relationship
through a point cloud (which raises the question why was the data in
the point cloud not used as training data. Also to obtain a point
cloud one needs to be in a vector-valued regression setting).
The paper could be significantly improved by providing a more thorough
empirical evaluation section that analyses the performance of the
proposed method on multiple real problems.  One of the main
contributions of the paper, as stated by the authors, is the extension
of the SELF approach beyond vector-valued regression. But evaluation
is done only on vector-valued regression problems.
The technique depends on using a kernel K for the kernel
regression. Is there any guidance on how to select such a kernel? What
kernel was used in the experimental section?
It would also be good if the authors analyze the effect of various
design and parameter choices. How sensitive are the results to the
kernel K, to the regularization parameter \lambda, to the parameters
\delta and \mu?
How were all the above parameters set in the experimental section?
Looking at figure 1, I am very surprised that the performance of STL
seems to converge to a point significantly worse than that of
NL-MTL. Given a large amount of data I would expect STL to perform
just as well or better than any MTL technique (information transfer
between tasks is no longer relevant if there is enough training
data). This does not seem to be the case in figure 1. Why is that?


Minimax Estimation of Bandable Precision Matrices
The inverse covariance matrix provides considerable insight for understanding statistical models in the multivariate setting. In particular, when the distribution over variables is assumed to be multivariate normal, the sparsity pattern in the inverse covariance matrix, commonly referred to as the precision matrix, corresponds to the adjacency matrix representation of the Gauss-Markov graph, which encodes conditional independence statements between variables. Minimax results under the spectral norm have previously been established for covariance matrices, both sparse and banded, and for sparse precision matrices. We establish minimax estimation bounds for estimating banded precision matrices under the spectral norm. Our results greatly improve upon the existing bounds; in particular, we find that the minimax rate for estimating banded precision matrices matches that of estimating banded covariance matrices. The key insight in our analysis is that we are able to obtain barely-noisy estimates of k×k subblocks of the precision matrix by inverting slightly wider blocks of the empirical covariance matrix along the diagonal. Our theoretical results are complemented by experiments demonstrating the sharpness of our bounds.

+++++++
Summary
+++++++
The paper establishes the first theoretical guarantees for statistical estimation of bandable precision matrices. The authors propose an estimator for the precision matrix given a set of independent observations, and show that this estimator is minimax optimal. Interestingly the minimax rate for estimating banded precision matrices is shown to be equal to the corresponding rate for estimating banded covariance matrices. The upper bound follows by studying the behavior of inverses of small blocks along the diagonal of the covariance matrix together with classical random matrix theory results. The lower bound follows by constructing two specially defined subparameter spaces and applying testing arguments. The authors support their theoretical development via representative numerical results.
+++++++
Strengths
+++++++
Clarity: The paper is very nicely written and the authors appropriately convey the ideas and relevance of their work.
Novelty: While I am not too familiar with the surrounding literature, the ideas in this paper appear novel. In particular, the estimator is both intuitive, as well as (relatively) easy to compute via the tricks that the authors discuss in Section 2.2. The techniques used to prove both upper and lower bounds are interesting and sound.
Significance: The authors establish matching upper and lower bounds, thereby effectively resolving the question of bandable precision matrix estimation (at least in the context of Gaussian random variables).
Typo: Should it be F_alpha in (3), not P_alpha?
Update after authors' response: Good paper; no further comments to add.


EX 2 : Exploration with Exemplar Models for Deep Reinforcement Learning
Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, where classifiers are trained to discriminate each visited state against all others. Intuitively, novel states are easier to distinguish against other states seen during training. We show that this kind of discriminative modeling corresponds to implicit density estimation, and that it can be combined with countbased exploration to produce competitive results on a range of popular benchmark tasks, including state-of-the-art results on challenging egocentric observations in the vizDoom benchmark.

Review of submission 1489:
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
Summary:
A discriminative novelty detection algorithm is proposed to improve exploration for policy gradient based reinforcement learning algorithms. The implicitly-estimated density by the discriminative novelty detection of a state is then used to produce a reward bonus added to the original reward for down-stream policy optimization algorithms (TRPO). Two techniques are discussed to improve the computation efficiency.
Comments
- One motivation of the paper is to utilize implicit density estimation to approximate classic count based exploration. The discriminative novelty detection only maintains a density estimation over the states, but not state-action pairs. To the best of my knowledge, most theoretical results on explorations are built on the state-action visit count, not state visit count. Generally, exploration strategies for RL algorithms investigate reducing the RL agent’s uncertainty over the MDP’s state transition and reward functions. The agent’s uncertainty (in terms of confidence intervals or posterior distributions over environment parameters) decreases as the inverse square root of the state-action visit count.
- It is not clear from the texts how the added reward bonus is sufficient to guarantee the novel state action pairs to be visited for policy gradient based RL methods. Similar reward bonuses have been shown to improve exploration for value-based reinforcement learning methods, such as MBIE-EB (Strehl & Littman, 2009) and BEB (Kolter & Ng, 2009), but not policy gradient based RL methods. Value-based and policy gradient based RL methods differ a lot on the technical details.
- Even we assume the added reward bonuses could result in novel state action pairs to be visited as desired, it is not clear why policy gradient methods could benefit from such exploration strategy. The paper, State-Dependent Exploration for Policy Gradient Methods (Thomas Ruckstieß, Martin Felder, and Jurgen Schmidhuber, ECML 2008), showed that limiting the exploration within an episode (i.e., returning the same action for any given state within an episode) could be helpful for policy gradient RL methods by reducing the variance in the gradients and improving the credit assignment. The soundness of the proposed techniques would be improved if necessary explanations were provided.
- There is one closely related recent paper on improving exploration for policy gradient methods, IMPROVING POLICY GRADIENT BY EXPLORING UNDER-APPRECIATED REWARDS (Ofir Nachum et al. ICLR 2017). It would be good for the paper to discuss the relationship and even compare against it empirically.


Gradients of Generative Models for Improved Discriminative Analysis of Tandem Mass Spectra
Tandem mass spectrometry (MS/MS) is a high-throughput technology used to identify the proteins in a complex biological sample, such as a drop of blood. A collection of spectra is generated at the output of the process, each spectrum of which is representative of a peptide (protein subsequence) present in the original complex sample. In this work, we leverage the log-likelihood gradients of generative models to improve the identification of such spectra. In particular, we show that the gradient of a recently proposed dynamic Bayesian network (DBN) [7] may be naturally employed by a kernel-based discriminative classifier. The resulting Fisher kernel substantially improves upon recent attempts to combine generative and discriminative models for post-processing analysis, outperforming all other methods on the evaluated datasets. We extend the improved accuracy offered by the Fisher kernel framework to other search algorithms by introducing Theseus, a DBN representing a large number of widely used MS/MS scoring functions. Furthermore, with gradient ascent and max-product inference at hand, we use Theseus to learn model parameters without any supervision.

This paper introduces Theseus, an algorithm for matching MS/MS spectra to peptide in a D.B.
This is a challenging and important task. It is important because MS/MS  is currently practically the only common high-throughput method to identify which proteins are present in a sample. It is challenging because the data is analog (intensity vs. m/z  graphs) and extremely noisy. Specifically, insertions, deletions, and variations due to natural losses (e.g. carbon monoxide) can occur compared to a theoretical peptide spectra.
This work builds upon an impressive body of work that has been dedicated to this problem. Specifically, the recently developed DRIP (Halloran et al 2014, 2016)  is a DBN, i.e. a generative model that tries to match the observed spectra to a given one in the D.B. modeling insertions/deletions etc. and computes for it a likelihood and the most likely assignment using a viterbi algorithm.
A big issue in the field is calibration of scores (in this case - likelihoods for possible peptide matches from the D.B to an observed spectra) as these vary greatly between spectra (e.g. different number of peaks to match) so estimating which score is “good enough” (e.g. in terms of overall false positive rate) is challenging.  Another previous algorithm, Percolator, has been developed to address this by training an SVM to discriminate between “good” matching peptides scores and “negative” scores from decoy peptides. A recent work  (Halloran et al 2016) tried to then map the DRIP viterbi path to a fixed sized vector so Percolator could be applied atop of DRIP.
Here, the authors propose an alternative to the above heuristic mapping. Instead, they employ Fisher Kernels where two samples (in our case - these would be the viterbi best path assigned to two candidates matches) are compared by their derivative in the feature space of the model. This elegant solution to apply discriminative models (here Percolator) to generative models (here DRIP) was suggested for other problems in the comp bio domain by Jaakkola and Haussler 1998. They derive the likelihood and its derivatives and the needed update algorithms for this, and show employing this to several datasets improve upon state of the art results.
Pros:
This work tackles an important problem. It results in what seems to be a significant improvement upon state of the art. It offers a new model (Theseus) and an elegant application Jaakkola and Haussler’s Fisher Kernel idea to the problem at hand. Overall, this appears to be high-quality computational work advancing an important topic.
Cons:
Major points of criticism:
1. Writing: While the paper starts well (intro) it quickly becomes impenetrable. It is very hard to follow the terms, the variables and how they all come together. For example, Fig1 is referred to in line 66 and includes many terms not explained at that point. Sec. 2.1 refers to c_b which is not defined (is it like c(x) ?) How are the natural losses modeled in this framework, Lines 137-145 are cryptic, evaluation procedure is too condensed (lines 221-227). In general, the authors spend too much space deriving all the equations in the main text instead of actually explaining the model and what is new compared to previous work. This makes evaluating both the experiments and the novelty hard. To make things worse, the caption of the main figure (Fig3) refers to labels/algorithms (“Percolator”, “DRIP Percolator Heuristic”) which do not appear in the figure.
Given the complexity of the domain and the short format of NIPS papers, we recommend the authors revise the text to make the overall model (Theseus) and novel elements more clear, and defer some of the technical details to a longer more detailed supplementary (which in this case seems mandatory).


Few-Shot Learning Through an Information Retrieval Lens
Few-shot learning refers to understanding new concepts from only a few examples. We propose an information retrieval-inspired approach for this problem that is motivated by the increased importance of maximally leveraging all the available information in this low-data regime. We define a training objective that aims to extract as much information as possible from each training batch by effectively optimizing over all relative orderings of the batch points simultaneously. In particular, we view each batch point as a 'query' that ranks the remaining ones based on its predicted relevance to them and we define a model within the framework of structured prediction to optimize mean Average Precision over these rankings. Our method achieves impressive results on the standard few-shot classification benchmarks while is also capable of few-shot retrieval.

This work is a great extension of few shot learning paradigms in deep metric learning. Rather than considering pairs of samples for metric learning (siamese networks), or 3 examples (anchor, positive, and negative -- triplet learning), the authors propose to use the full minibatchs' relationships or ranking with respect to the anchor sample. This makes sense from the structured prediction, and efficient learning. Incorporating mean average precision into the loss directly, allows for optimization of the actual task at hand.
Some comments:
1. t would be great to see more experimental results, on other datasets (face recognition).
2. Figure 1 is not very powerful, it would be better to show how the distances change as training evolves.
3. Not clear how to set \lambda, as in a number of cases, a wrong value for \lambda leads to weak results.
4. Authors must provide code for this approach to really have an impact in deep metric learning, as this work requires a bit of implementation.


Action Centered Contextual Bandits
Contextual bandits have become popular as they offer a middle ground between very simple approaches based on multi-armed bandits and very complex approaches using the full power of reinforcement learning. They have demonstrated success in web applications and have a rich body of associated theoretical guarantees. Linear models are well understood theoretically and preferred by practitioners because they are not only easily interpretable but also simple to implement and debug. Furthermore, if the linear model is true, we get very strong performance guarantees. Unfortunately, in emerging applications in mobile health, the time-invariant linear model assumption is untenable. We provide an extension of the linear model for contextual bandits that has two parts: baseline reward and treatment effect. We allow the former to be complex but keep the latter simple. We argue that this model is plausible for mobile health applications. At the same time, it leads to algorithms with strong performance guarantees as in the linear model setting, while still allowing for complex nonlinear baseline modeling. Our theory is supported by experiments on data gathered in a recently concluded mobile health study.

In the context of multi-armed bandit problems for mobile health, modelling correctly the linearity or non-linearity of the rewards is crucial.
The paper tries to take advantage of the fact that a general "Do nothing" action is generally available, that corresponds to the baseline. The idea is that, while the reward for each action may not be easily captured by a linear model, it is reasonable to assume that the relative error between playing an action and playing the "Do thing" action follows a linear model.
The article revisits bandit strategies in this context.
Comments:
1) Line 29: See also the algorithm OFUL, and more generally Kernel-UCB, GP-UCB, to cite a few other algorithms with provable regret minimization guarantees.
2) Line 35, 51, and so on: I humbly suggest you replace "mhealth" with something else, e.g. "M-health", since "mhealth" looks like a typo.
3) Question about the model: is \bar {f}_t assumed to be known? If not and if it is completely arbitrary, it seems counter-intuitive that one can prove any learning guarantee, given that a bandit setting is considered (only one action per step), and \bar {f}_t may change at any time to create maximal confusion.
4) Line 132: "might different for different"
5) Line 162: What is \bar{a}_t ? I can only see the definition for a_t and \bar{a}_t^\star.
Likewise, what is \bar{a} on line 165? It seems these are defined only later in the paper, which is confusing. Please, fix this point.
6) Question: What is the intuition for choosing \pi_t to be (up to projection) P(s_{t,\bar a}^\top \bar \theta \geq 0) ? Especially, please comment on the value 0 here.
7) Lemma 5: Should 0 be replaced with a in the probability bound?
8) Section 9: I understand that you heavily used the proof of Agrawal and Goyal, but I discourage the use of sentences such as "For any filtration such that ... is true". I suggest you rather control probability of events.
9) Proof, line 62: there is a missing reference.
.................
In the end the setting of the paper and definition of regret look a bit odd, but the authors provide an interesting idea to overcome the possibly nasty non-stationary term, thanks to randomization. Also deriving the weighted least-squares estimate and TS analysis in this context is interesting, even though relying heavily on previous existing analysis for TS. Finally, the idea to consider that only the relative errors follow a linear model might be generalized to other problems.
Weak accept.


Clustering Billions of Reads for DNA Data Storage
Storing data in synthetic DNA offers the possibility of improving information density and durability by several orders of magnitude compared to current storage technologies. However, DNA data storage requires a computationally intensive process to retrieve the data. In particular, a crucial step in the data retrieval pipeline involves clustering billions of strings with respect to edit distance. Datasets in this domain have many notable properties, such as containing a very large number of small clusters that are well-separated in the edit distance metric space. In this regime, existing algorithms are unsuitable because of either their long running time or low accuracy. To address this issue, we present a novel distributed algorithm for approximately computing the underlying clusters. Our algorithm converges efficiently on any dataset that satisfies certain separability properties, such as those coming from DNA data storage systems. We also prove that, under these assumptions, our algorithm is robust to outliers and high levels of noise. We provide empirical justification of the accuracy, scalability, and convergence of our algorithm on real and synthetic data. Compared to the state-of-the-art algorithm for clustering DNA sequences, our algorithm simultaneously achieves higher accuracy and a 1000x speedup on three real datasets.

The paper presents a solution to a new type of clustering problem that has emerged from studies of DNA-based storage. Information is encoded within DNA sequences and retrieved using short-read sequencing technology. The short-read sequencer will create multiple short overlapping sequence reads and these have to be clustered to establish whether they are from the same place in the original sequence. The characteristics of the clustering problem is that the clusters are pretty tight in terms of edit distance (25 max diameter here - that seems quite broad given current sequencing error rates) but well separated from each other (much larger distance between them than diameter).
I thought this was an interesting and timely application. DNA storage is a hot topic and this kind of clustering is one of the computational bottlenecks to applying the method. I guess the methods presented will also be of general interest to the NIPS audience as an example of a massive data inference problem.
The methods look appropriate for the task and the clustering speed achieved looks very impressive.
The paper is well written and the problem, methods and results were easy to follow.
My main concern with the work centred around how realistic is the generative model of the data. The substitution error rate and the insertion/deletion error rates are set to be the same parameter p/3. I think that current Illumina high-throughput sequencing is associated with much higher substitution error rates than insertion/deletion rates, I think  perhaps 100-fold less insertion/deletion than substitution for some illumina technologies. If insertion/deletion rates are much lower than substitution errors then perhaps the problem is less challenging than stated here since there would be less requirement for dealing with edit distances in the case insertions/deletions are rare. Also, for most technologies the error rates from sequencing are can be well established and therefore methods can be fine-tuned given knowledge of these parameters.