Generalized Correspondence-LDA Models (GC-LDA) for Identifying Functional Regions in the Brain
This paper presents Generalized Correspondence-LDA (GC-LDA), a generalization of the Correspondence-LDA model that allows for variable spatial representations to be associated with topics, and increased flexibility in terms of the strength of the correspondence between data types induced by the model. We present three variants of GC-LDA, each of which associates topics with a different spatial representation, and apply them to a corpus of neuroimaging data. In the context of this dataset, each topic corresponds to a functional brain region, where the region's spatial extent is captured by a probability distribution over neural activity, and the region's cognitive function is captured by a probability distribution over linguistic terms. We illustrate the qualitative improvements offered by GC-LDA in terms of the types of topics extracted with alternative spatial representations, as well as the model's ability to incorporate a-priori knowledge from the neuroimaging literature. We furthermore demonstrate that the novel features of GC-LDA improve predictions for missing data.

The submission #635, entitled " Generalized Correspondence-LDA Models (GC-LDA) for Identifying Functional Regions in the Brain" presents a topic model that accounts for both word occurrence and associated locations to build a cognitive atlas from the brain imaging literature. The main contribution of the paper is a novel "generalized correspondence model" that is derived into three versions, e.g. including (or not) symmetry constraints on the spatial distributions involved. The model is handled in a Bayesian framework, and the results display an improved predictive accuracy in cross-validation when the full power of the correspondence model is used.
This paper addresses an important topic, and presents a clear, well-detailed and well-presented framework. I find the results and their illustration compelling. In enjoyed reading the paper.  my only major concern regards the cross-validation scheme adopted in the paper (see below).   More in detail --------------  8: 'probability distribution over neural activity' is a bit vague and potentially misleading to describe the kind of statistics made in coordinate-based meta-analysis.  99-111: I find this paragraph a bit awkward. To me the message boils down to: although gamma does not enjoy a very clear interpretation in the generative model, it plays the role of a regularizer. Presenting it in such a way would clarify the presentation.  125 why not 'semantic' instead of 'linguistic' ? Would sound better to me.  130-132: Again, a rather awkward way to introduce symmetry about the x=0 plane.  Fig. 2: I did not get the meaning of the colors. Usually, red/blue refer to positive/negative values, but it cannot be the case here.  192 the authors probably mean A2.  238-241: I would _really_ prefer a cross-validation across documents, as it would probably fit with the properties that you expect from a cognitive atlas. The authors should --at least-- explain why they did not do it that way, and if possible, describe the performance of cross-document generalization.  The paper did not address model selection (how many classes ?). While I understand that this is ill-posed and potentially impossible in their unsupervised setting, I think that the question should be discussed.  Using GMMs with two classes is fairly limited. Why not using more general spatial distributions, such as kernel density estimators ?  Gibbs sampling has a strong drawback wrt variational schemes: its computational cost. I have no idea of the computation time for the model (this should be stated at least in SM) , but I think that it would be great to perform a comparison: either confirm the weakness of the variational approach or propose  it as an efficient alternative.


Convolutional Neural Fabrics
Despite the success of CNNs, selecting the optimal architecture for a given task remains an open problem. Instead of aiming to select a single optimal architecture, we propose a "fabric" that embeds an exponentially large number of architectures. The fabric consists of a 3D trellis that connects response maps at different layers, scales, and channels with a sparse homogeneous local connectivity pattern. The only hyper-parameters of a fabric are the number of channels and layers. While individual architectures can be recovered as paths, the fabric can in addition ensemble all embedded architectures together, sharing their weights where their paths overlap. Parameters can be learned using standard methods based on backpropagation, at a cost that scales linearly in the fabric size. We present benchmark results competitive with the state of the art for image classification on MNIST and CIFAR10, and for semantic segmentation on the Part Labels dataset.

The authors propose a convolutional neural fabric model, which addresses the issue of selecting an optimal CNN architecture. The proposed "fabric" model consists of 3D trellis, which are essentially the response maps that capture scale, layer, and channel information of the traditional CNN architectures. Each node in the trellis is connected to a 3x3 scale-channel neighborhood from the previous layer. The authors show that the traditional CNN architectures can be embedded and implemented via neural fabric models. Finally, the authors present results on MNIST, CIFAR and "Part Labels" dataset and show competitive results.
I think the authors present an interesting and a novel idea that addresses an important problem of how to select an optimal CNN architecture. I think the major strength of this paper is the novelty in the technical approach.  I had several comments in regards to the experiments. As is standard for many deep learning architecture papers, the authors conduct experiments on MNIST dataset, CIFAR, and in this case "Part Labels" dataset. In my opinion, the tasks related to these datasets (especially MNIST) are not very challenging, and the results are extremely saturated (over >95% accuracy). I would like to see some of the experiments on more challenging tasks such as semantic segmentation on PASCAL, etc. Having said this, I think the key contribution of this paper is the novelty of the method, and I would be willing to overlook the experiments on these "less challenging" tasks, especially since it seems like this is a common practice in this field (a practice, which is not good in my opinion).  Additionally, I also didn't find some of the experiments that convincing. I thought that the results for the Parts Labeling task were solid. However, the disparities between the error rates on the MNIST dataset seem pretty large (considering how saturated the performance is). Especially if we look at the top 2 baseline methods, we observe that without using data augmentation , these methods achieve better results with significantly less number of parameters (>3M) than CNF-dense. The same trend occurs on the CIFAR dataset, where we have top 2 methods significantly outperform CNF-dense method with almost 30M less parameters than CNF-dense. Therefore, based on these experiments, I am not convinced that the method would work very well on some of the more challenging tasks (such as semantic segmentation).   In summary, even though I didn't find many of these experiments convincing, I would still like to see this paper published. There has been a lot of incremental work on CNNs recently, that do not contribute much to the field despite very good performance in certain tasks. In comparison,  even though this paper may not have the most impressive empirical results, it presents a novel alternative to CNNs, which addresses an important CNN architecture selection problem.  ---POST REBUTTAL---  I read the rebuttal. I agree with AR5 that the authors should include more experiments clearly outlining theoretical and practical costs of training compared to the traditional CNNs. As I mentioned previously, I am not convinced that the proposed approach is practical at its current state. It doesn't seem to outperform other techniques on the more challenging datasets, even if it uses a larger number of parameters than other methods. However, I like the idea of the paper, and I think that it could have a solid impact on deep learning community, as the CNN architecture selection problem is an important one.


An Online Sequence-to-Sequence Model Using Partial Conditioning
Sequence-to-sequence models have achieved impressive results on various tasks. However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences. This is because they generate an output sequence conditioned on an entire input sequence. In this paper, we present a Neural Transducer that can make incremental predictions as more input arrives, without redoing the entire computation. Unlike sequence-to-sequence models, the Neural Transducer computes the next-step distribution conditioned on the partially observed input sequence and the partially generated sequence. At each time step, the transducer can decide to emit zero to many output symbols. The data can be processed using an encoder and presented as input to the transducer. The discrete decision to emit a symbol at every time step makes it difficult to learn with conventional backpropagation. It is however possible to train the transducer by using a dynamic programming algorithm to generate target discrete decisions. Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in. We also find that the Neural Transducer performs well for long sequences even when attention mechanisms are not used.

The authors introduced an encoder-decoder architecture with attention over blocks of input and variable length decoder structure. The encoder is a multi-layer LSTM RNN. The decoder is an RNN model conditioned on weighted sums of the last layer of the encoder and it's previous output. The weighting schemes (attention) varies and can be conditioned on the hidden states or also previous attention vectors. The decoder model produces a sequence of symbols, until it outputs a special end character "e" and is moved to the next block (other mechanisms where explored as well (no end-of-block-symbol and separately predicting the end of a block given the attention vector). It is then fed the weighted sum of the next block of encoder states. The resulting sequence of symbols determines an alignment of the target symbols over the blocks of inputs, where each block may be assigned a variable number of characters. The system is trained by fixing an alignment, that approximately resembles the best alignment. Finding this approximately best alignment is akin to a beam-search with a beam size of M (line 169), but a restricted set of symbols conditional on the last symbol in a particular hypothesis (since the target sequence is known). Alignments are computed less frequently than model updates (typically every 100 to 300 sequences). For inference, an unconstrained beam-search procedure is performed with a threshold on sequence length and beam size.
First of all, I think the paper is well written and clear.  Judging by the title the main contribution of this model is a novel model architecture. Looking at the references, there have previously been speech recognition systems, that use attention ([2] and [5]). Sequence transduction has also been introduced by Graves in "Sequence Transduction with Recurrent Neural Networks" in 2012. The follow up paper is cited as [7] uses this transduction method. Grave's describes his transducer to "extend CTC by defining a distribution over output sequences of all lengths, and by jointly modeling both input-output and output-output dependencies.". To me the novelty of this paper then is the idea of applying the transducer on a block-by-block basis and defining a RNN conditional on it's last definite output. Other methods integrate over alignments using the output of an RNN, whereas here the authors define a separate complex network structure to perform the transduction. However, Graves and Hinton also use a separate prediction network to feed into the transduction graph. To me, the disadvantage of the method presented here is the complicated training procedure, since exact constrained inference becomes intractable. I'm am also not sure how to judge the novelty of this approach, because I might be missing literature, since there has been a lot of developments in this area. It seems very similar to Graves and Hinton [7], especially since they also use the separate prediction network, except for the attention mechanism (but in later experiments we see, that the attention mechanism doesn't necessarily help that much (unless larger window sizes have advantages (one of which might be speed))).  The experimental evaluation of this method is performed on an addition toy task and TIMIT. For the addition task the authors constrain themselves to adding two three-digit numbers. It would be interesting to see how the model generalizes to longer numbers. That is, if trained on additions of three-digit numbers, how well does it add four-digit numbers?  For TIMIT the authors first compare having recurrent state across blocks, vs. not having a recurrent state. I think, if you don't have recurrent state in the transducer, you might be able to perform more efficient inference using Graves' transduction method. I'm not too sure on this point. Figure 3 shows that for some block sizes no attention performs equally well. I think no-attention means, that the encoder hidden states are averaged per block and then fed to the transducer. This is then akin to a convolution of size W and stride W. Maybe it is interesting to see how the encoder would perform, if W states are combined and fed into a simple RNN, that produces a softmax over symbols, which is then trained with CTC or Graves' Transducer. The best performance of Graves' Transducer on a unidirectional LSTM RNN has been reported at 19.6%, which is close to the best 19.8% reported here, albeit I understand, that the authors argue, that this model needs to be tuned more, which can be in itself a lot of work. Their best network uses a 3 layer LSTM RNN transducer. I think four layer transducers might vary much in accuracy, because they start overfitting a lot on TIMIT. It would be interesting to see how much these models overfit.   Overall, given the complexity of the model and the performance, plus similarities to other methods, I'm not very sure how interesting this method is (I might lack the breath in literature to point out exactly how similar this model is). I would be interested in understanding how exactly it differs from Graves' and Hinton's Transducer network. I am not very sure on my review of the novelty of this method. The performance in itself is competitive on TIMIT, but other, simpler methods perform equally well. I am not sure if the extra capacity and context is necessary in comparison to other methods or for this dataset. Perhaps, if the model was trained on more data, it would become more apparent, but I'm not sure how well it scales. I wonder how long it takes to fully train a model?


Spatiotemporal Residual Networks for Video Action Recognition
Two-stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two-stream architecture to allow spatiotemporal interaction between the two streams. Second, we transform pretrained image ConvNets into spatiotemporal networks by equipping them with learnable convolutional filters that are initialized as temporal residual connections and operate on adjacent feature maps in time. This approach slowly increases the spatiotemporal receptive field as the depth of the model increases and naturally integrates image ConvNet design principles. The whole model is trained end-to-end to allow hierarchical learning of complex spatiotemporal features. We evaluate our novel spatiotemporal ResNet using two widely used action recognition benchmarks where it exceeds the previous state-of-the-art.

This paper proposed a deep CNN architecture for video action recognition by extending residual networks. The proposed architecture combines both spatial stream and temporal stream into a single network, and extends to model longer period of temporal information. This work achieves state-of-the-art performance on two public action recognition datasets.
This paper presents a framework that improves two stream networks for video action recognition by extending residual network to combine information from two streams into one single network. It significantly improves over previous state-of-the-art on two popular video action recognition benchmark. The downside of this paper is the limited novelty. There are previous work tried to combine two streams into a single network [1,2], and the temporal convolution is not new either [3]. Although the way to combine two streams is slightly different from previous work, the proposed approach is still pretty straightforward.  Since this paper mainly consists of incremental ideas on previous work, more details could be given on how those ideas help in solving the tasks to gain more insights. For example, separate ablation studies would be useful to understand how the techniques in section 3.1 and 3.2 help.  The temporal convolution described in section 3.2 is just replacing spatial convolution with temporal convolution, which is not really related to residual connections. The term “residual” could be dropped in the title of section 3.2 and line 166.  Some details in the paper are unclear: - Are the HMDB51 networks initialized from UCF101 networks as in previous work? - The proposed network sees 705 frames (line 193). How do you handle videos with less than 705 frames? - Line 233: “Batch-normalization uses a smaller batch size of 128/32.” Does that mean the batch size is 128/32 = 4 as discussed above, or the batch size is 128 or 32? This is a bit confusing. - Is the loss of ST-ResNet defined as the sum of loss of both streams?  Typo: Line 255: UCF0101 -> UCF101  Overall, this paper is clearly written, and produces very good results on video action recognition datasets.    [1] C. Feichtenhofer, A. Pinz, A. Zisserman. Convolutional Two-Stream Network Fusion for Video Action Recognition. https://arxiv.org/abs/1604.06573  [2] E. Park, X. Han, T. L. Berg, A. C. Berg. Combining Multiple Sources of Knowledge in Deep CNNs for Action Recognition. WACV 2016 [3] L. Pigou, A. van den Oord, S. Dieleman, M. Van Herreweghe, J. Dambre, Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video. http://arxiv.org/abs/1506.01911


Search Improves Label for Active Learning
We investigate active learning with access to two distinct oracles: LABEL (which is standard) and SEARCH (which is not). The SEARCH oracle models the situation where a human searches a database to seed or counterexample an existing solution. SEARCH is stronger than LABEL while being natural to implement in many situations. We show that an algorithm using both oracles can provide exponentially large problem-dependent improvements over LABEL alone.
Counterexamples can be ineffective or misleading in practice as well. Reconsidering the intervals example above, a counterexample on the boundary of an incorrect interval provides no useful information about any other examples. What is a good counterexample? What is a natural way to restrict the searchable space? How can the intervals problem be generalized?
We define a new oracle, SEARCH, that provides counterexamples to version spaces. Given a set of possible classifiers H mapping unlabeled examples to labels, a version space V ⊆ H is the subset of classifiers still under consideration by the algorithm. A counterexample to a version space is a labeled example which every classifier in the version space classifies incorrectly. When there is no counterexample to the version space, SEARCH returns nothing.
How can a counterexample to the version space be used? We consider a nested sequence of hypothesis classes of increasing complexity, akin to Structural Risk Minimization (SRM) in passive learning [see, e.g., Vapnik, 1982, Devroye et al., 1996. When SEARCH produces a counterexample to the version space, it gives a proof that the current hypothesis class is too simplistic to solve the problem effectively. We show that this guided increase in hypothesis complexity results in a radically lower LABEL complexity than directly learning on the complex space. Sample complexity bounds for model selection in LABEL-only active learning were studied by Balcan et al. [2010], Hanneke [2011]. SEARCH can easily model the practice of seeding discussed earlier. If the first hypothesis class has just the constant always-negative classifier h(x) = −1, a seed example with label +1 is a counterexample to the version space. Our most basic algorithm uses SEARCH just once before using LABEL, but it is clear from inspection that multiple seeds are not harmful, and they may be helpful if they provide the proof required to operate with an appropriately complex hypothesis class.
Defining SEARCH with respect to a version space rather than a single classifier allows us to formalize "counterexample far from the boundary" in a general fashion which is compatible with the way LABEL-based active learning algorithms work.
Related work. The closest oracle considered in the literature is the Class Conditional Query (CCQ) [Balcan and Hanneke, 2012] oracle. A query to CCQ specifies a finite set of unlabeled examples and a label while returning an example in the subset with the specified label, if one exists.
In contrast, SEARCH has an implicit query set that is an entire region of the input space rather than a finite set. Simple searches over this large implicit domain can more plausibly discover relevant counterexamples: When building a detector for penguins in images, the input to CCQ might be a set of images and the label "penguin". Even if we are very lucky and the set happens to contain a penguin image, a search amongst image tags may fail to find it in the subset because it is not tagged appropriately. SEARCH is more likely to discover counterexamples-surely there are many images correctly tagged as having penguins.
Why is it natural to define a query region implicitly via a version space? There is a practical reason-it is a concise description of a natural region with an efficiently implementable membership filter [Beygelzimer et al., 2010, 2011, Huang et al., 2015. (Compare this to an oracle call that has to explicitly enumerate a large set of examples. The algorithm of Balcan and Hanneke [2012] uses samples of size roughly dν/�

This article studies a type of active learning in which the algorithm has access both to the usual Label query oracle and also a "Search" oracle.  The Search oracle takes a set of classifiers as input, and returns a correctly labeled example that contradicts all of them, if one exists.  They first relate this oracle to others that have appeared in the literature previously, and provide several nice examples.  They then propose a general active learning algorithm that uses a nested sequence of hypothesis classes, and uses the Search oracle to perform model selection, and to construct a seed data set for an active learning algorithm for each hypothesis class.  They study this both in the realizable case and the agnostic case.
Putting aside the practicality of this framework, the theoretical analysis of the advantages of access to this type of oracle is interesting.  This work reveals that the search oracle provides substantial advantages in the context of model selection.  For instance, by making roughly k* search queries (where k* is the index of the model containing the Bayes classifier), Theorem 3 reveals that the number of label queries can be reduced by at least a factor of k* (and sometimes dramatically more) compared to the best known result in the prior literature on model selection in active learning (where only label queries are allowed).  This can be a significant reduction in some settings.  Now, on the issue of practicality, the authors make an effort to motivate each major component of the framework in terms of practical considerations.  However, in many cases, these motivations seem fairly thin.  A good example is the fact that, even in the agnostic case, the search oracle returns the label h*(X) of the optimal classifier h* at the returned point X, rather than the (noisy) label Y associated with that example X.  The motivation given is that we can suppose the oracle will choose an example for which she is supremely confident in the label.  But this motivation implicitly assumes that the only source of noise is the annotator's uncertainty about the label; this is often not the case, as factors such as the feature space representation can also introduce inherent noise (e.g., in NLP, choosing a bag-of-words representation creates noise, since the human annotator sees the words in order, and different word orders may form different valid sentences, and which of these orders is observed may affect her choice of label, so that the conditional distribution of label given the unordered list of words may have nonzero entropy). That said, I can appreciate that it is sometimes desirable for the  annotator to be able to indicate a strong preference for a particular  label, and supposing this strongly-preferred label to be h*(X) might  be a reasonably elegant way to incorporate this extra information  into the formal model, and that in some cases the h*(X) assumption  might not need to strictly hold for the algorithm to have reasonable  behavior in practice.  Overall, I found this to be an interesting piece of work, definitely worth reading for those interested in the theory of interactive learning.  However, I would expect any future works that build on it will probably tweak various aspects of the model, to weaken the  search oracle, or potentially modify the type of input (or the restrictions thereon) to the oracle (e.g., I can imagine these search constraints, based on the region of agreement of a subset of H, could potentially make a minimax lower bound analysis significantly more difficult than it would be for certain natural alternatives, some  of which make more practical sense and still allow for the types of regions used in the algorithms proposed here).   minor comments:  In the related work section, the notation d \nu/\epsilon^2 is used without being defined or explained.  The main theorems in this paper are somewhat difficult to interpret in the abstract, being sample-dependent bounds on the number of labeled samples the algorithm uses.  Is there any hope of obtaining concise abstract a-priori bounds, as were possible for the unions of intervals example?  The wording in the discussion section is confusing to me.  It seems to me that the search oracle would often be more powerful (provides  more information) than the CCQ oracle, since CCQ only needs to return  the noisy label while search returns the h* label.  I guess the discussion is referring to the fact that CCQ can specify any region, not only a region of agreement of a subset of H (and in particular, in the realizable case this means CCQ is at least as powerful as search).  So discussing these two types of queries as being a kind of trade-off in the agnostic case seems more accurate than what is currently written.  Related to this, it might be interesting to discuss teasing apart the contribution of getting h*(X) labels instead of Y labels, vs the inherent power in these kinds of region-constrained mistake queries (i.e., with Y labels returned).  Can one show lower bounds for region-constrained mistake queries where the Y labels are returned instead of h* labels, worse than the upper bounds proven here using the search oracle?  The bibliography is inconsistent in its choice of whether or not to abbreviate author first names.


The Product Cut
We introduce a theoretical and algorithmic framework for multi-way graph partitioning that relies on a multiplicative cut-based objective. We refer to this objective as the Product Cut. We provide a detailed investigation of the mathematical properties of this objective and an effective algorithm for its optimization. The proposed model has strong mathematical underpinnings, and the corresponding algorithm achieves state-of-the-art performance on benchmark data sets.

1. Quick Summary   The authors propose a new objective function for  clustering networks that they call product cut. It is defined based on a personalized pagerank matrix (although it seems like some larger class should be possible and this choice is incidental). The key idea is that we are going to take a complicated product of properties of each partition rather than a simple sum of graph-based metrics of the partition as in  many standard methods. One of the complexities involves using the personalized PageRank matrix in order to define the objective.   This objective is shown to satisfy a simple analytical property that preserves the partitions in some  region around a sample problem with two clear clusters with a third cluster as the perturbation.   Next, in a small surprise, the problem is shown to be solvable by a relaxed convex _maximization_ problem (so this doesn't give any computational efficiency, but  does mean it enables more practical algorithms.) The relaxed problem has integer solutions at any extrema. This motivates a heuristic LP procedure to optimize the function.  There are a set of experimental results that show the standard results on a few test problems in terms of "purity". It'd be nice to see a few more evaluations (NMI or F1) that as well.  Small issues & questions  1. One issue with this paper relates to the convergence of the optimization strategy for convex maximization. The results in the paper do not justify convergence  to a local optima, nor to they justify convergence of the cluster at all; if f(x_{k+1}) >= f(x_k)) then we cannot conclude finding a local maxima (even with an upper-bound) nor does this give convergence of the x_k sequence.  Were you using additional properties in your statement of this conclusion?   2. In your simple example, wouldn't clustering of A, B, C also be appropriate? It seems like you show that if you introduce a good conductance cluster (C), you break the NCut objective for bi-partition.  But this isn't surprising as you are asking for a  bisection of a graph, not the trisection that NCut would more naturally produce.  Can you clarify what your method would do if you asked for a tripartition in this case?   Suggestions   There should be a broader class of matrices than the  personalized PageRank matrices that would support your procedure. (For instance, at least the heat-kernel would seem to work as well...)  Some parameter study about alpha would be nice.   Since this is really a paper about a new objective function it'd be nice to see more motivation for it.   Presentation notes * Given that the objective function is first defined with the most important term missing, it's hard to appreciate it's properties.  * Typo on figure 1a.
This paper has some issues that would need to be addressed related to the use of the final optimization procedure. If this could be corrected, the  rest of the paper makes a small contribution, but in the context of a new objective function. Adding new objective functions to the literature is a lot like adding new standards. It's unclear if they will be useful; but this one seems to have a few nice properties that would merit acceptance.


Adaptive Neural Compilation
This paper proposes an adaptive neural-compilation framework to address the problem of learning efficient programs. Traditional code optimisation strategies used in compilers are based on applying pre-specified set of transformations that make the code faster to execute without changing its semantics. In contrast, our work involves adapting programs to make them more efficient while considering correctness only on a target input distribution. Our approach is inspired by the recent works on differentiable representations of programs. We show that it is possible to compile programs written in a low-level language to a differentiable representation. We also show how programs in this representation can be optimised to make them efficient on a target input distribution. Experimental results demonstrate that our approach enables learning specifically-tuned algorithms for given data distributions with a high success rate.

The paper explores a scheme where a low-level program is compiled into a differentiable representation, and the result is used as a starting point for a learning process optimizing a program for run-time efficiency on a selected distribution of inputs with the correctness constraints also applied only on the same distribution of inputs.  In a number of cases a classical optimized program is recoverable from the differentiable representation found by the learning process, but not always. In some other cases the result of learning gives better performance than the optimal performance possible for a classical program (in those cases, obviously, a classical program is not recoverable from the continuous representation obtained).
Recently we have seen re-emergence of research into neural models of classical software. This line of research is now rapidly accelerating. For many of us, the single paper most associated with the starting point of this re-emergence was "Neural Turing Machines" by Graves et al., and the development was very rapid since then. It is important to explore various angles and viewpoints on this rapidly evolving research topic.  The research presented in the paper under review is of interest on its own, but for many readers the key value will be provided by a new interesting angle of view this paper provides into differentiable models of classical software. This is where the most value from this paper for NIPS readership would probably come from.  I liked the way the paper is written overall, and I benefited from reading it in its current form, but I found some aspects of it confusing and more difficult to the reader than they should be. Some of those confusing aspects are, unfortunately, close to the central part of technical presentation. I hope the authors will be able to improve the paper in those aspects, some of which I list below.  It is not immediately obvious what the convex combination of registers is (formula 1, line 130). The natural interpretation is that it is a convex combination of delta-function probability distributions (distributions on the set of registers where a single register is taken with probability 1). But one also wonders whether it might be a convex combination of the values of registers (resulting in real values, rather than integers, and then using extension of operations listed in Figure 1b onto reals). In any case, if it is a convex combination of delta-function distributions, the paper should say so explicitly. If it is more subtle than that, the need for explanation is even higher.  The formula 2, line 132 is just not interpretable for a reader, even after looking into the supplement. What those double indexes mean for "out" and for "arg" is just too difficult to guess correctly. A verbal explanation of what's going on in this formula is necessary.  The role of soft-write and softmax in all this is not well explained (the reader can guess some of it from looking at the supplement). I think a much crisper explanation of those central aspects I've listed so far is necessary, preferably in the paper itself, rather than in the supplement.  I am going to also list a few less significant aspects where improvement would be desirable.  The term "instruction register (IR)" is somewhat confusing. First the reader might think, is this register going to actually store the machine command? Then the reader eventually realizes that this register is used simply as "program counter (PC)" (acceptable alternatives for this term seem to be "instruction pointer (IP)", "instruction address register (IAR)", or "instruction counter (IC)"). But the wording in line 104-105 does not say that the controller simply uses the "instruction register" to fetch the command from memory using the value of that counter and then parses the resulting command (which is what seems to actually be going on). Instead a wording is more general, and says that the controller computes the components of the command in some unspecified way. So the authors might have meant something else in mind in addition to the straightforward classical schema which they did end up implementing, and "instruction register" terminology might be hinting at those alternatives. In any case, it took some labor to understand that this was used as straightforward program counter, a pointer into the memory tape from which the command is fetched.   In the differentiable version (Section 3.2) the authors seem to rely on the range of accessible interger values being the same as the size of the memory tape M. This range, which is the same as indices into M, should then be from 0 to M-1, line 97 nonwithstanding (because the paper relies on having 0, so indices for m's in line 97 should be adjusted for that). I understand that this is a convenient convention at the early stage of this research (one then can worry less about values of indices which don't point anywhere into the memory, etc), but is it really necessary? It does feel like a very artificial requirement.  There are some minor problems with Figure 2c. The matrix (iv) is missing one row. In the matrix (i) "DEC" is not where it should be according to the order of Figure 1b. Finally the columns look OK on screen, but when I print, the first columns of matrices (1) and (iii) are only half-width, and the last columns of those matrices are one-and-a-half width.  Reference [7] is missing the last name of the second author, Jimmy Ba. Instead, the first word of the paper title, "Adam", has migrated to the position of the second author's last name.  Finally, "efficient program learning" on line 2 is quite ambiguous. The first interpretation which jumps into reader's mind is "efficient process of program learning". Only then one realizes that the authors has meant "learning the efficient programs" instead.   I hope this would help to improve the text.


Synthesis of MCMC and Belief Propagation
Markov Chain Monte Carlo (MCMC) and Belief Propagation (BP) are the most popular algorithms for computational inference in Graphical Models (GM). In principle, MCMC is an exact probabilistic method which, however, often suffers from exponentially slow mixing. In contrast, BP is a deterministic method, which is typically fast, empirically very successful, however in general lacking control of accuracy over loopy graphs. In this paper, we introduce MCMC algorithms correcting the approximation error of BP, i.e., we provide a way to compensate for BP errors via a consecutive BP-aware MCMC. Our framework is based on the Loop Calculus approach which allows to express the BP error as a sum of weighted generalized loops. Although the full series is computationally intractable, it is known that a truncated series, summing up all 2-regular loops, is computable in polynomial-time for planar pair-wise binary GMs and it also provides a highly accurate approximation empirically. Motivated by this, we first propose a polynomial-time approximation MCMC scheme for the truncated series of general (non-planar) pair-wise binary models. Our main idea here is to use the Worm algorithm, known to provide fast mixing in other (related) problems, and then design an appropriate rejection scheme to sample 2-regular loops. Furthermore, we also design an efficient rejection-free MCMC scheme for approximating the full series. The main novelty underlying our design is in utilizing the concept of cycle basis, which provides an efficient decomposition of the generalized loops. In essence, the proposed MCMC schemes run on transformed GM built upon the non-trivial BP solution, and our experiments show that this synthesis of BP and MCMC outperforms both direct MCMC and bare BP schemes.

This article deals with the computation of the partition function (or normalization constant) of graphical models involving discrete variables. The approach used here is based on first, an estimation of the BP partition function (or Bethe-Peierls), which is corrected by a loop-series expansion on the considered system. The main focus of this paper is to design first, an algorithmic method computing the 2-regular generalized loops term of the expansion (which is an approximation of the true loop expansion in most cases) and converging in polynomial time. Second, they extend their algorithm in order to compute the whole loop expansion by a similar algorithm but, in this case, without providing theoretical guarantee.  Finally, their second algorithm is tested on the K_4 graph (a clique of four nodes) and compared to a MCMC estimation and a BP estimation of the log of the partition function for many different types of systems : ferromagnet, antiferromagnet and disordered (random) interactions.
 The authors develop an very interesting method to the hard problem of computing the partition function of GM. In particular, there is no method to compute Z for general graphs accurately and log(Z) is a particularly interesting component in many fields: it is related to the likelihood in Machine Learning, it gives all the statistical information of a system in statistical mechanic, ... I think therefore that any effort toward that direction is an important contribution.  Concerning the work presented here, I have some questions and concerns that I would like the authors to answer:  1) the authors mentioned that BP in general fail to converge on general graph but that more involved convergent alternative exists. I think the author should be more specific, precising if all the convergent algorithms do provide the same estimation of the BP partition function and it they are indeed linear in the system size.  2) I guess that, in line 101/102, the Ising model means the 2D Ising model ?  3) I have some concern about the estimation of the full loop expansion. The 2-reg loop can be proven to converge in polynomial time but the full loop does not have these garranty. Can the author provide at least some empirical estimation of the convergence time for some systems or at least some arguments why this method will not suffer from exponential slowing down when using it on a large system ?  4) Concerning the experimental results:  * it is not specified the number of systems at different temperature that were used for the estimation of the loop series. * it is not clear to me what is the MCMC estimation of log(Z), is it the estimation obtained by integrating the average energy function over \beta ? I guess not, but I think that the authors should specify how they estimate log(Z) using MCMC. * if the estimation of MCMC is not obtained by integrating over many \beta, I would like to know why it is faster to compute Z_2-loop using many different \beta instead of integrating the average energy function over many \beta ?   Finally, maybe the most important point to me that would justify the whole method. The experiments are done over very small systems (4 nodes). I agree with the authors about the importance of being able to compute the true Z, but it is possible with a modest computer to reach systems with at least 20/25 nodes. Therefore I don't understand the choice of the author nor can I be sure that their method work at larger sizes. I therefore strongly advice the author to explain why such a choice. They might want as well to provide some test on larger systems comparing some estimation obtained by integration with MCMC with their method.


Fast recovery from a union of subspaces
We address the problem of recovering a high-dimensional but structured vector from linear observations in a general setting where the vector can come from an arbitrary union of subspaces. This setup includes well-studied problems such as compressive sensing and low-rank matrix recovery. We show how to design more efficient algorithms for the union-of-subspace recovery problem by using approximate projections. Instantiating our general framework for the low-rank matrix recovery problem gives the fastest provable running time for an algorithm with optimal sample complexity. Moreover, we give fast approximate projections for 2D histograms, another well-studied low-dimensional model of data. We complement our theoretical results with experiments demonstrating that our framework also leads to improved time and sample complexity empirically.

In this paper, the authors consider the problem of signal recovery from a union of subspaces (UoS). UoS problems are a generalization of several structurally constrained problems, such as matrix completion and sparse regression. They show that one can obtain fast algorithms for UoS problems by considering an iterative projection scheme, and obtain highly accurate results even when the projections are approximate. A key insight is that they employ TWO approximate projections, onto what they call the “head” and “tail” subspaces. They apply their results to some canonical cases and show that the proposed methods are faster than methods that require exact projections, while not compromising on accuracy of the recovered signals.
The paper is well written, and is very relevant. My only issue is that, while the motivation to propose the methods is that of scalability, the authors do not exlpicitly show that their methods are significantly faster than methods requiring exact projections on very large datasets. However, I recommend an accept since I think this addition in the paper is fairly trivial to do.   MAJOR COMMENTS: equation (2) only applies if the underlying optimization you intend to solve is least squares with a constraint. In the general case, the second term in the parenthesis will be the gradient of a generic loss function.   I expected Definitions 3 and 4 to be symmetric. Is there a reason why c_T = 0 is possible in def. 3 but not in 4?   line 160: recent advances in greedy methods obviate the need for the full SVD. Instead, they only require the top singular value to be computed, getting an order of magnitude improvement. Of course, these methods avoid exact projections but it might be prudent to mention the existence of such methods.  Line 218: How come there is no dependence on \gamma? won’t a simple counting argument yield \gamma even in a lower bound?  Lines 276-278: the explanation here seems to suggest that the order of projections (head first, then tail) is important. Is this correct? What will happen if one reverses the order?  Line 287: The U \otimes U_H \otimes U_T RIP condition is different from the usual (U \otimes U \otimes U) RIP conditions one might assume. Can you comment on this distinction and also compare? Maybe provide an example to indicate whether it is tighter or looser.  Figures 1-3: I suggest a comparison with other, non SVT based methods for low rank matrix recovery as well. greedily adding one SV at a time is significantly faster, and has been shown to perform very well. Also these greedy projections can be approximate (using power methods for example)   A key point you make is that IHT based schemes with approximate projections are faster than SVT based methods. Could you please provide runtimes on larger matrices (order 10K or more? ). More importantly, there should be sizes at which SVT with exact projections becomes infeasible. It might be good to see where that transition occurs.     MINOR COMMENTS: Line 63 : is there a reference for results for sparse PCA?  Definitions 3 and 4 : Do you need the constants to be bounded by 1 and not 0?  In theorem 5, should the \theta in the log terms be \theta^*?  line 157: replace “can be” with “is”. you already mention that r << d_1   line 162: an —> and  Theorems 7 and 8: what are the typical values of epsilon and zeta that one considers? a small epsilon will mean that zeta has to be large. eps = 1e-2 means that zeta has to be close to 10 atleast right? That makes the run time O(d^11) which is prohibitive.   Line 212: Please state the natural structural conjecture here.   Line 293: for the sake of completion, please provide the sample complexities for the cases you consider in the main paper as well.


LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
Recurrent neural networks (RNNs) have achieved state-of-the-art performances in many natural language processing tasks, such as language modeling and machine translation. However, when the vocabulary is large, the RNN model will become very big (e.g., possibly beyond the memory capacity of a GPU device) and its training will become very inefficient. In this work, we propose a novel technique to tackle this challenge. The key idea is to use 2-Component (2C) shared embedding for word representations. We allocate every word in the vocabulary into a table, each row of which is associated with a vector, and each column associated with another vector. Depending on its position in the table, a word is jointly represented by two components: a row vector and a column vector. Since the words in the same row share the row vector and the words in the same column share the column vector, we only need 2 |V | vectors to represent a vocabulary of |V | unique words, which are far less than the |V | vectors required by existing approaches. Based on the 2-Component shared embedding, we design a new RNN algorithm and evaluate it using the language modeling task on several benchmark datasets. The results show that our algorithm significantly reduces the model size and speeds up the training process, without sacrifice of accuracy (it achieves similar, if not better, perplexity as compared to state-of-the-art language models). Remarkably, on the One-Billion-Word benchmark Dataset, our algorithm achieves comparable perplexity to previous language models, whilst reducing the model size by a factor of 40-100, and speeding up the training process by a factor of 2. We name our proposed algorithm LightRNN to reflect its very small model size and very high training speed.

Consider a word level language model. The majority of its parameters lie in the embedding matrix, as each word is typically assigned a 100-1000 dimensional vector. This submission presents a novel factorization of the embedding matrix, that quadratically reduces the memory requiremnts. In turn, this allows the use of larger word embeddings and leads to state-of-the-art performance on ACLW and Billion Words benchmarks.  Traditionally, word embeddings are kept in a single table of size #words x embedding_dim. In this contribution, the authors create two small tables, both of size sqrt(#words) x embedding-dim, that are called the row-embedding matrix and the column-embedding matrix. Each word is assigned to an embedding from the row-embedding matrix and from the column-embedding one. Thus the embeddings of words are shared between words. However, this change substantially reduces the memory requirements.  During model use, the RNN makes two steps for each word - it updates its hidden state first using the row-embedding matrix, then using the column-embedding matrix. Likewise, the probability of sampling a word is factorized into two probabilities: of the probability assigned to the word's position of the word in the row-matrix, and of of the probability assigned to the word's position in the column matrix. This solves another important problem, by reducing the amount of computation needed to compute the probability of an output word.  The key ingredient of the proposed contribution is an algorithm that assigns word and column indexes to words, therefore establishing the sharing pattern for the embeddings. The authors frame it as a matching problem, and elegantly use a heuristic solution to the Minimum Cost Maximum Flow graph problem.
This paper solves two important practical problems of 1. reducing the size of the word embedding matrix and 2. reducing the computations of word probabilities, obviating the need for a hierarchical Softmax or Monte Carlo estimations of the model's training cost. The proposed model makes no assumptions about the structure of the words, which makes it potentially useful outside of NLP. In contrast, character-based word embedding models also reduce the model size, but do need to access the internal structure of words (i.e. their characters).  The presentation is sufficiently clear, though I have the following questions: 1. from equation (1) and Fig. 2 it seems that the model sees the t-th word before predicting it. Still, there is an information bottleneck, as the word is aliased with sqrt(#words) that are assigned the same row-embedding. However, this is still surprising - could the author comment whether this is intentional, and how do the models behave when the word w_t is generated using information about words up to t-1 only?  2. how is the cost (4) computed? I guess that there are separate matrices for input-word embedding and  output-word embedding, that share the word-allocation table. To compute eq. (4) is only the output allocation changed (which is fast to compute, as the hidden states of the RNN do not change and words are conditionally independent given the hidden states), or is the input allocation changed too?   3. how often in practice is the index reallocation (step 3) phase run?


A Forward Model at Purkinje Cell Synapses Facilitates Cerebellar Anticipatory Control
How does our motor system solve the problem of anticipatory control in spite of a wide spectrum of response dynamics from different musculo-skeletal systems, transport delays as well as response latencies throughout the central nervous system? To a great extent, our highly-skilled motor responses are a result of a reactive feedback system, originating in the brain-stem and spinal cord, combined with a feed-forward anticipatory system, that is adaptively fine-tuned by sensory experience and originates in the cerebellum. Based on that interaction we design the counterfactual predictive control (CFPC) architecture, an anticipatory adaptive motor control scheme in which a feed-forward module, based on the cerebellum, steers an error feedback controller with counterfactual error signals. Those are signals that trigger reactions as actual errors would, but that do not code for any current or forthcoming errors. In order to determine the optimal learning strategy, we derive a novel learning rule for the feed-forward module that involves an eligibility trace and operates at the synaptic level. In particular, our eligibility trace provides a mechanism beyond co-incidence detection in that it convolves a history of prior synaptic inputs with error signals. In the context of cerebellar physiology, this solution implies that Purkinje cell synapses should generate eligibility traces using a forward model of the system being controlled. From an engineering perspective, CFPC provides a general-purpose anticipatory control architecture equipped with a learning rule that exploits the full dynamics of the closed-loop system.

The authors extend the classic cerebellar model of Marr-Albus (1969). By doing so they also provide a framework to obtain an optimal eligibility trace. Their contribution is important in understanding the cerebellar function, but also may find practical applications in control-system problems. I would like to make a few comments regarding the exposition of their solution and its context with respect to previous works.
1) As a person from outside the control theory field, I found it difficult to follow some of the derivations and arguments in the manuscript. In particular, the authors might explain in more detail the meaning of matrices A, B, C and D on page 4 (perhaps also by referring back to them in Figure 2). They should also present the form of these matrices used in the simulations in section 3.4. Similarly, the quantities k_p and k_i on page 6 are not defined. They might be trivial for control theorist, but not for readers outside the field.  2) From what I understood, the eligibility trace (h_j on page 5) is computed from the known system transition matrix and cerebellar basis. However, in realistic biological setup this matrix will not be a priori known. How will the eligibility trace be estimated in the cerebellum? Is this matrix hard-coded in the cerebellar architecture? If so, how is it adapted to new tasks? If it is learnt, what would the learning rule be?  3) The framework proposed by the authors has some similarities with predictive coding proposed for vision (Rao and Ballard, Nat Neurosci, 1999). Could the authors expose the similarities and differences? One key difference seems to be that in the authors' approach the error signal e[n] is not used to compute the predicted output o[n], but only to adjust the weights. How would the authors motivate this choice?  4) If the goal of the study is to find a realistic model of cerebellum, the authors should discuss their results in context of experimental findings about cerebellum architecture and function. For example, human study performed by Blakmore  et al. (NeuroReport, 2001) suggest that the cerebellum encodes the error signal. Is this finding consistent with what the authors propose? If not, could their model be used to explain the data?  Minor comments:  line 4: extend -> extent line 164: with relation -> with respect to line 165: get derive -> derive line 170: different -> difference line 177: e[k,n] -> did the authors mean e[k - n]? line 181: "has been computing convolving" - not clear what is meant, please correct line 202: subtracted to the current weight -> subtracted from the current weight line 220: meaning of symbols k_p and k_i is not clear line 226: why these values? what does "spaced by" mean? Is this spacing between tuning curves of different fibers? line 248: are crucial for not only that but also to account -> are not only crucial for [the delays?] but also they account line 304: what predictions precisely do the authors mean?


General Tensor Spectral Co-clustering for Higher-Order Data
Spectral clustering and co-clustering are well-known techniques in data analysis, and recent work has extended spectral clustering to square, symmetric tensors and hypermatrices derived from a network. We develop a new tensor spectral co-clustering method that simultaneously clusters the rows, columns, and slices of a nonnegative three-mode tensor and generalizes to tensors with any number of modes. The algorithm is based on a new random walk model which we call the super-spacey random surfer. We show that our method out-performs state-of-the-art co-clustering methods on several synthetic datasets with ground truth clusters and then use the algorithm to analyze several real-world datasets.

This paper propose a new variant of the super-spacey random walk - a variant of higher-order Markov chain model that has been introduced in previous work for co-clustering tensor data. Experimental results show that the algorithm outperforms several previous works on synthetic dataset. The algorithm is then evaluated on several realistic datasets.
I'm not an expert in the problem of clustering but I have read the proofs in some detail including the supplementary material and did not notice significant errors. Overall, I find the paper well-written and understandable. The supplement is well-prepared and contains more details about experimental results which I appreciate. In terms of technical contributions, the problem of undefined transitions in the probability tensor has been tackled already in previous work [4]. Here the authors consider a different way to solve it. However, since the two models share striking similarities, I think the new model should be justified more in terms of both theoretical motivation and analytical results (see below) so that one can clearly see the advantages/drawbacks of the model compared to previous work. As I'm not an expert in this problem, I would consider most of my comments below as suggestions for improving the paper if it is accepted.  1. Please check carefully the notations & variable's indices between lines 44-45 in the supplement as I believe there are some mistakes.  2. Is there any equivalence form of the newly introduced biased conductance like the standard one as shown in Observation 1. At this point, paper [4] has proposed similar conductance functions - "higher-order conductance" and "higher-order expansion". Please comment on this.  3. The biased conductance has been introduced in the paper as a new quality metric for partitioning data. However, it is never used in the experiments to analyze the algorithm, but instead other evaluation metrics (ARI,FMI,F1) were used to compare all algorithms. Some figures showing the values of different higher-order conductance as functions of |S_k| for different methods (at least GTSC,TSC,SC) could be helpful to readers.  4. It seems that the comparison with previous work is done only on synthetic datasets with quantitative results, while it is rather qualitative on other real-world datasets. In this way, I cannot see why one should use the proposed algorithm in favour of other methods for practical clustering. Moreover, on the synthetic dataset, only 5 trials that has been used to estimate the standard deviations for algorithms might be too small so that one cannot say much about standard deviations.


LazySVD: Even Faster SVD Decomposition Yet Without Agonizing Pain *
We study k-SVD that is to obtain the first k singular vectors of a matrix A. Recently, a few breakthroughs have been discovered on k-SVD: Musco and Musco [19] proved the first gap-free convergence result using the block Krylov method, Shamir [21] discovered the first variance-reduction stochastic method, and Bhojanapalli et al. [7] provided the fastest O(nnz(A) + poly(1/ε))-time algorithm using alternating minimization. In this paper, we put forward a new and simple LazySVD framework to improve the above breakthroughs. This framework leads to a faster gap-free method outperforming [19], and the first accelerated and stochastic method outperforming [21]. In the O(nnz(A) + poly(1/ε)) running-time regime, LazySVD outperforms [7] in certain parameter regimes without even using alternating minimization.

The main result seems to be an improvement of [14] for spectral-error low rank approximation, from time k*nnz(A)/eps^{1/2} + k^2 d / eps + k^3/eps^{3/2} to k nnz(A)/eps^{1/2} + k^2 d / eps^{1/2}. This is interesting in certain regimes, if say, eps is very small and nnz(A) is about d. Note here the time is worse than nnz(A) times one can achieve with Frobenius norm error, though in certain applications one may need spectral error.   My understanding of the savings - and I hope the authors can clarify this - is that [14] was a bit crude with the k/eps^{1/2} vectors in R^d they find via a block Krylov iteration. I think they just compute an orthonormal basis for such vectors which gives k^2 d/eps time (ignoring fast matrix multi optimizations). This paper achieves k^2 d / eps^{1/2} time essentially because they never have to perform such an orthogonalization; they instead iteratively find a very good rank-1 approx, then they deflate off what they just found, and repeat by finding a very good rank-1 approx on what's left, in total k times. They still need O~(1/eps^{1/2}) iterations for each rank-1 approx, and they need to deflate off what they just found, but given that the vectors they find anyway are almost already orthogonal they just pretend that they are and the deflation takes O(kd/eps^{1/2}) per iteration, giving the O(k^2 d/eps^{1/2}) in total. Another key difference with [14] is that the 1/eps^{1/2} is coming from the shift-and-invert algorithm of [7] and [8], which was not directly used in [14].   By instantiating the above framework with different algorithms in different settings, the authors obtain tradeoffs also in a stochastic setting. They also obtain nnz(A) algorithms with a mixed spectral and Frobenius guarantee though I think there is an issue here - see below.
I mentioned a comment in the summary above regarding intuition. I hope the authors can clarify that.   Also, of importance, is for the authors to state what is the relationship with the following work which seems to use similar techniques for a similar problem: http://arxiv.org/abs/1607.02925  I didn't find the nnz(A) results very compelling. The main reason is that the authors seem to miss the earlier work of Cohen et al. "Dimensionality Reduction for k-Means Clustering and Low Rank Approximation" http://arxiv.org/pdf/1410.6801v3.pdf Applying Theorem 27 in that paper, with say, the composition of an OSNAP with an SRHT then a Gaussian, one can in nnz(A) + O~(k^2 d/eps^2) time reduce the dimension to k/eps^2 x d, at which point one can find a good low rank-k approximation of the resulting matrix using any method (this is the definition of a projection-cost preserving sketch). Importantly, the same mixed-Frobenius and spectral error guarantee is obtained. Actually the result in the Cohen et al. paper could be stronger because there is an eps/k factor in front of the Frobenius norm term, and importantly it doesn't have a sigma/sigma_{k+1} term in it, as this submission does, which can be arbitrarily large. The overall comparison of the two works depends on which algorithm one uses to solve the k/eps^2 x d problem.   Just a few presentation comments:  (1) the table in the stochastic setting was a bit confusing, it suggests  kn^{3/4} d/(sigma_k^{1/2} eps^{1/2}) <= kd/(sigma_k^2 eps^2).  which can't be true, so I guess the authors instead mean their bound is never worse than what is in parantheses.   (2) in general I feel the writing style could be toned down a bit.


Swapout: Learning an ensemble of deep architectures
We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout [20], stochastic depth [7] and residual architectures [5,6] as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.
the layers. The theoretical evidence that a depth of 1000 is required for practical problems is thin. Bengio and Dellaleau argue that circuit efficiency constraints suggest increasing depth is important, because there are functions that require exponentially large shallow networks to compute [1]. Less experimental interest has been displayed in the width of the networks (the number of filters in a convolutional layer). We show that increasing the width of our swapout networks leads to significant improvements in their accuracy; an appropriately wide swapout network is competitive with a deep residual network that is 1.5 orders of magnitude deeper and has more parameters.
Contributions: Swapout is a novel stochastic training scheme that can sample from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. Swapout improves the performance of the residual networks for a model of the same depth. Wider but much shallower swapout networks are competitive with very deep residual networks.

This paper proposes a new form of the popular dropout technique for regularization of deep neural networks.  In particular, this approach develops a dropout strategy for a combination of recent developments in deep learning (residual networks, stochastic depth networks).  The algorithm stochastically chooses between dropping out residual connections, feedforward connections and both.  The authors demonstrate that this strategy empirically improves on these aforementioned techniques on a convolutional networks applied to CIFAR-10 and CIFAR-100.
This is a generally well written paper that develops a neat novel idea for regularization of deep networks.  I think the presented idea could be impactful in that it may prove to be very useful in practice to many researchers and practitioners.  The presented empirical evaluation demonstrates that the approach is quite promising.  The idea is, however, not tremendously novel in light of recent work (res-nets and stochastic depth networks) and it would have been nice to see comparison on some more interesting benchmarks.  small typo in the abstract: exiting -> existing  I appreciate the additional analysis of the test time statistics of the approach.  Why can't you estimate the test time statistics empirically on a validation set?    I really appreciate the tidbits on why dropout and swapout interact poorly with batch normalization.     It's useful to know that you don't have to average over very many sampled dropouts (swapouts).  I think this is a neat additional analysis and rather useful to the community.  Why do the authors first do exactly 196 and then 224 epochs before decaying the learning rate?  Normally such specific choices would arouse suspicion except in this case I expect it doesn't make much difference (e.g. between 196 and a round number like 200).  Some additional detail on how the hyperparameters were chosen seems necessary here.  I think it would have significantly improved the paper if experiments were run on more interesting problems.  CIFAR-10 seems to be exhausted at this point.  The current numbers are beyond the human level error of ~6%, which is rather worrying.  Papers that beat this number are either overfitting to the test set or to the original labeler.  Either way, it doesn't seem like a particularly useful benchmark anymore.  One of the major reasons why res-nets were so well received was because they achieved state-of-the-art on the much larger and more challenging imagenet.  A major result of the paper is that we might not need as deep networks as originally thought.  However, it doesn't seem surprising (to me at least) that one wouldn't need an especially deep network for the 32x32 images in the CIFAR dataset (whereas this may be a different story for much larger images with more classes).  As far as I can tell, the authors do not actually compare to the state-of-the-art on CIFAR-10.  The All-convolutional-net paper achieved a lower error.  Could the authors explain why these baselines are not included?  The paper states that it compares to 'fair' baselines but this seems like a somewhat qualified result.  e.g. As far as I can tell, the standard data-augmentation tricks are applied in this work as well.  There is a lot of extra space available in this paper (see e.g. section 3.2).  This leaves lots of space for an additional table of experimental results on e.g. imagenet.


Lifelong Learning with Weighted Majority Votes
Better understanding of the potential benefits of information transfer and representation learning is an important step towards the goal of building intelligent systems that are able to persist in the world and learn over time. In this work, we consider a setting where the learner encounters a stream of tasks but is able to retain only limited information from each encountered task, such as a learned predictor. In contrast to most previous works analyzing this scenario, we do not make any distributional assumptions on the task generating process. Instead, we formulate a complexity measure that captures the diversity of the observed tasks. We provide a lifelong learning algorithm with error guarantees for every observed task (rather than on average). We show sample complexity reductions in comparison to solving every task in isolation in terms of our task complexity measure. Further, our algorithmic framework can naturally be viewed as learning a representation from encountered tasks with a neural network.

The paper presents a learning algorithm that can learn many (different but related) prediction tasks by transferring knowledge obtained from previously solved tasks. The algorithm receives the tasks sequentially and choose to either: - build a majority vote predictor by weighting previously learned predictors; or - learn a new predictor "from scratch". The algorithm is inspired by VC-dimension regret bounds and is designed to obtain theoretical generalization and sample complexity guarantees. However, no empirical experiments are performed.
From my very personal point of view, the lifelong learning paradigm is a vague concept and is sometimes evoked for studying different scenarios that can be named otherwise (like transfer learning). However, I think that the framework studied here (from Balcan et al., 2015) is a very pertinent "lifelong problem". The authors present an honest work in the right direction. The proofs are not trivial but, as I explain below, the contribution appears to me insufficient for NIPS.  The risk bound minimized by the learning algorithm may be very high, as it relies on the VC-dimension of the predictors. Also, the role of the constant C in Theorem 1 is never discussed, and little is said about the way to handle it in the algorithm. Therefore, I am unconvinced that the algorithm will lead to a successful empirical procedure. I consider that the authors should provide empirical evidences that the approach is sound (at least on a synthetic dataset).   It is also likely that a tighter bounding technique (such as Rademacher or PAC-Bayes, both used by some of the cited works) provides better theoretical guarantees and/or empirical results. If there is a particular reason to rely on VC-dimension, the authors should discuss their choice.  In the introduction (Lines 47 to 50, and 80-81), the authors claim that, as opposed to Balcan et al. (2015), their algorithm learns "with weighted majority votes rather than linear combinations over linear predictor" and that "the shift from linear combinations to majority votes introduces a stability to the learned ensemble that allows exploiting it for later tasks." However, this comparison is not explicitly discussed later on. The meaning of the claim is unclear to me; In general, a linear combination of predictors can be expressed as a majority vote with a simple reparameterization.  ***  Post-rebuttal comments:  Considering the authors’ rebuttal, I raised my "technical quality" score a bit.   I still think that the paper would benefit from some experiments, in order to assess if Algorithm 1 would make an empirically good learning algorithm. It would also help to answer practical questions like: - Is the "\gamma-effective dimension" assumption realistic?  - Does the number of required labels for each task will be reasonable? Or should we manually "override' the constant C to make the algorithm work empirically?   The authors argued that we must consider the paper as a theoretical work, and leave these questions for future work. This is defendable. Personally, I feel that a work presenting a learning algorithm without any empirical results (not even on a toy dataset) is a bit incomplete (sure, Balcan et al. did it beforehand, but in COLT, which is strongly theory oriented).   Finally, regarding the linear combinations vs majority votes misunderstanding, I think the distinction must be clearly specified in the paper. From my point of view, a linear *separator* is given by \sign( < v_i, x > ), albeit a linear *regressor* is given by < v_i, x >. Moreover, a majority vote ca be a aggregation of either classifiers or regressors.


Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
We propose a general purpose variational inference algorithm that forms a natural counterpart of gradient descent for optimization. Our method iteratively transports a set of particles to match the target distribution, by applying a form of functional gradient descent that minimizes the KL divergence. Empirical studies are performed on various real world models and datasets, on which our method is competitive with existing state-of-the-art methods. The derivation of our method is based on a new theoretical result that connects the derivative of KL divergence under smooth transforms with Stein's identity and a recently proposed kernelized Stein discrepancy, which is of independent interest.

The authors derive a new inference algorithm for bayesian inference connecting variational inference to recent results  using Kernelized Stein discrepancy. The algorithm is particle based and smoothly interpolates between MAP and true Bayesian posterior (by using more particles).  At a high level, the authors connect the derivative of a small perturbation (x+eps phi(x), which are going to be progressively used to transform an initial distribution into the target distribution) to the stein operator. Then, by restricting the deformation from identity to be a function in a RKHS, they can obtain a closed-form for the optimal deformation function phi, which takes the form of a stein operator on the initial distribution. Distributions are represented by samples and updated according to the simple operator.
Overall, I found the paper interesting; the paper offers new theory as well as numerical results comparable to the state of the art on decently difficult datasets.  Perhaps due to space constraints, an important part of the paper (section 3.2) - the inference algorithm - is poorly explained. In particular, I initially thought that the use of particles meant that the approximating distribution was a sum of Dirac delta functions - but that cannot be the case since, even with many particles, the 'posterior' would degenerate into the MAP (note that in similar work, authors either use particles when p(x) involves discrete x variables, as in Kulkarni et al,  or 'smooth' the particles to approximate a continuous distribution, as in Gershman et al). Instead, it looks like the algorithm works directly on samples of the distribution q0, q1.. (hence the vague 'for whatever distribution q that {xi}ni=1 currently represents'). It is tempting to consider q_i to be a kernel density estimate (mixture of normals with fixed width), and see if we can approximate equation 9 for that representation to be stable. Is this possible / something the authors have considered?  Also, since the algorithm does not care about what precise form the distribution q takes, but the algorithm cannot possibly work for q sum of delta functions, I assume there are some smoothness assumptions required for the theorem to hold (even if it is simple as 'q has a density').   Minor: - In the second paragraph of the introduction, the authors explain that in some work (Kingma and Welling, Rezende and Mohamed, etc.), neural networks are used to approximate a posterior distribution; this is not accurate: in those papers the posterior is typically a simple distribution (often, gaussian mean-field);  the neural network is used to amortize inference and predict the parameters of the posterior as  a function of the data. This is faster, but theoretically worse than regular mean-field inference (optimization issues put aside). Rezende and Mohamed 2015 is, however, truly using a neural network to parametrize a more complex posterior distribution.  - The connection to the reparametrization trick is very promising and would have deserved a few words (even if future work goes into mode details), at least in the appendix, where the reparametrization is key in obtaining 3.1.


Bi-Objective Online Matching and Submodular Allocations
Online allocation problems have been widely studied due to their numerous practical applications (particularly to Internet advertising), as well as considerable theoretical interest. The main challenge in such problems is making assignment decisions in the face of uncertainty about future input; effective algorithms need to predict which constraints are most likely to bind, and learn the balance between short-term gain and the value of long-term resource availability. In many important applications, the algorithm designer is faced with multiple objectives to optimize. In particular, in online advertising it is fairly common to optimize multiple metrics, such as clicks, conversions, and impressions, as well as other metrics which may be largely uncorrelated such as 'share of voice', and 'buyer surplus'. While there has been considerable work on multi-objective offline optimization (when the entire input is known in advance), very little is known about the online case, particularly in the case of adversarial input. In this paper, we give the first results for bi-objective online submodular optimization, providing almost matching upper and lower bounds for allocating items to agents with two submodular value functions. We also study practically relevant special cases of this problem related to Internet advertising, and obtain improved results. All our algorithms are nearly best possible, as well as being efficient and easy to implement in practice.

The paper builds on the following problem.  There is a set of agents.  Each agent a is associated a submodular function f_a. Elements from the universe arrive over time.  When an element arrives, it is to be assigned to an agent. Let S_a be the set of elements assigned to agent a.  The goal is to assign the elements to agents as to maximum sum_a f(S_a).  This problem is known as online submodular welfare maximization.  For this problem, the greedy algorithm which greedily assigns elements to the agent that maximizes the objective is a 1/2 approximation and this is the best possible.  This paper considers a generalization of this problem where each agent has two different submodular functions.  These two functions represent two different objectives each agent may want to optimize.  The goal is to design an an allocation of elements so as to give a (a,b)-approximation where 'a' is the approximation ratio if only the first function is considered in the objective and similarly 'b' is for the second function.  The paper shows what is essentially a (1/3,1/3) approximation for the problem.  This is done simply by flipping a coin when an element arises and choosing to runt the greedy algorithm only considering one of the objectives.  Proving this result is straightforward.  What is nice about this work is they show that this is the best possible algorithm but giving a lower bound.  This I found interesting.  The paper goes on to show better results in some restricted cases.  I think this is a reasonable paper.  The downside of the work is that they did not motivate the two objective setting much and this leaves my feeling that it is a contrived.  On the other hard, I personally think the problem is reasonable despite the lack of motivation given.  I feel the authors add insight into this area.    I feel the paper is a weak accept.
Please give more detailed motivation for the problem if available.


High Dimensional Structured Superposition Models
High dimensional superposition models characterize observations using parameters which can be written as a sum of multiple component parameters, each with its own structure, e.g., sum of low rank and sparse matrices, sum of sparse and rotated sparse vectors, etc. In this paper, we consider general superposition models which allow sum of any number of component parameters, and each component structure can be characterized by any norm. We present a simple estimator for such models, give a geometric condition under which the components can be accurately estimated, characterize sample complexity of the estimator, and give high probability nonasymptotic bounds on the componentwise estimation error. We use tools from empirical processes and generic chaining for the statistical analysis, and our results, which substantially generalize prior work on superposition models, are in terms of Gaussian widths of suitable sets.

The authors study superposition models and their recovery from noisy linear samples. They analyze the scenario where we observe linear samples of linear superpositions of k structured parameters and obtain robust estimation conditions based on standard tools such as restricted strong convexity.
The authors study superposition models and their recovery from noisy linear samples. This topic is very well studied in recent years and there are multiple high impact papers characterizing the behavior of such models. In this sense, the technical contribution is not novel enough, and in particular, many of the results can be derived with already existing tools or are straightforward.  Most importantly, though, the claims of this paper are misleading. Authors first state Eq 3 which would be a great result for the high-dimensional statistics literature. Here, we have $k$ structured parameters, and once we demix them, the total estimation error grows as the worst parameter \max_i omega(C_i). This is great because, what I would expect is " \sum_i omega(C_i)" which is k times worse.  In reality, we can never achieve \max_i omega(C_i) for reasonable problems of interest in the demixing (or superposition) literature. In particular, the proposed SC condition (9) have a additional sqrt(k) constant on the left side for any reasonable demixing assumption. The reason is that as soon as we introduce a mild randomness to one of the parameters, the cones C_i are approximately orthogonal and l2 norm will not linearly add up.   The only interesting problem for which SC condition holds for a constant is when all the parameters are same!  I don't think that the authors are intentionally hiding the impact of k (maybe they didn't notice). However, as it stands, this paper claims to reduce a quantity that grows as k to log(k) which is misleading. I recommend them they emphasize the impact of k cleanly (e.g. at Eq 3 as well as when \rho is first introduced.


Joint quantile regression in vector-valued RKHSs
Addressing the will to give a more complete picture than an average relationship provided by standard regression, a novel framework for estimating and predicting simultaneously several conditional quantiles is introduced. The proposed methodology leverages kernel-based multi-task learning to curb the embarrassing phenomenon of quantile crossing, with a one-step estimation procedure and no postprocessing. Moreover, this framework comes along with theoretical guarantees and an efficient coordinate descent learning algorithm. Numerical experiments on benchmark and real datasets highlight the enhancements of our approach regarding the prediction error, the crossing occurrences and the training time.

The authors consider the problem of quantile regression. That is, given pairs (X,Y), the problem is to estimate specified quantiles of the conditional distributions Y|X=x in contrast to 'normal' regression in which the expectation E(Y|X=x) is to be estimated. Previous papers have phrased this as a regression problem for which functions from an RKHS with norm less than some constant are the hypothesis set.  The main contribution of this paper is to propose a method to tackle the so-called 'crossing problem', an issue that arises when curves for multiple quantiles are estimated simultaneously and the resulting curves cross over. The method the authors propose is to consider the problem as regression in a vector-valued RKHS. The kernel of the RKHS can be chosen in order to penalise the coordinate-wise functions crossing over.   They prove two theoretical results concerning the empirical risk minimiser of their problem. First, that the generalisation error is small with high probability; second, that the error in estimating the quantiles is small with high probability.   They write down the problem as an optimisation problem, and solve this using Primal-Dual Coordinate Descent.   Finally, they test their method on real datasets, showing that their method performs favorably compared to previous methods.
Novelty/Originality: The idea to apply vector-valued RKHSs to the problem of quantile regression appears to be novel. I hadn't encountered vector-valued RKHSs before reading this, but I don't know that much about this area. In any case, I like the idea because it is simple but appears to be effective. Although the authors restrict themselves to considering decomposable kernels, I would be interested to hear some discussion of other matrix-valued kernels and the resulting regression functions they would induce.  Technical Quality: The authors have provided two theoretical results about their method. I would appreciate more explanation of Theorem 4.2 (quantile property) as the statement of the theorem is a bit difficult to parse. I'd also appreciate some more discussion on selection of the parameter gamma - it seems like it should be interpretable in some way. At the end of section 6.1, it is stated that the pinball loss embraces the crossing loss as a subcriterion. I can see intuitively that this makes sense, but I'm confused as to why the crossing loss would then sometimes be so bad with the previous methods, even though they were also cross-validated using the pinball loss.  The following view might be unfashionable, but I would like to see the method being tested on synthetic data for which we know the conditional distributions Y|X=x and so can compare the estimated quantiles to the true quantiles. in addition to the real data experiments.  Potential impact: While I am not aware of any particular uses of quantile regression, it seems like it ought to be something that is practically used in the application areas listen in the introduction.    Clarity and presentation: In general, I thought that the section layout of the paper was good and the logic was clear. However, the style of writing was at times quite casual and there were a few grammatical errors.


A Non-generative Framework and Convex Relaxations for Unsupervised Learning
We give a novel formal theoretical framework for unsupervised learning with two distinctive characteristics. First, it does not assume any generative model and based on a worst-case performance metric. Second, it is comparative, namely performance is measured with respect to a given hypothesis class. This allows to avoid known computational hardness results and improper algorithms based on convex relaxations. We show how several families of unsupervised learning models, which were previously only analyzed under probabilistic assumptions and are otherwise provably intractable, can be efficiently learned in our framework by convex optimization.

This paper proposes a PAC-style learning framework for unsupervised learning which the authors call CONUS (Comparative Non-generative UnSupervised) and also proposes learning methodology. In contrast to PAC-learning, the CONUS framework is distribution-dependent hence suitable to analysis of unsupervised learning. To further support such analysis, it is non-generative, i.e. makes no assumption on joint distributions but instead evaluates learning by minimizing reconstruction error. As an illustrative example the authors show how Principle Component Analysis can be formally learned in their setting. They also give a general class of hypothesis classes for unsupervised learning which they call polynomial spectral decoding, and they show how they can be efficiently learned under their definitions via convex optimization. Their main contribution is a convex optimization-based methodology for improper learning of a class of hypothesis classes, which includes dictionary learning. For analysis of this they introduce an averaged variant on their original CONUS-learnable definition. Their new class of algorithms are polynomial time for a broad class of dictionary models, thus significantly advancing state of the art.
* I need to clarify: my low score on question 8 is only because the paper must be revised to fix very frequent typos and English issues (i.e., in its current form the paper's presentation is not NIPS-acceptable just because of these issues). From a technical point of view however the paper is clear and very well-written and I feel it makes an important contribution. My score on question 8 would have been a 4 if the English issues were not present. I therefore strongly vote for acceptance after thorough revision. * It is good that sufficient care is taken to provide background for non-experts (see some positive comments on this later on) but in some places at the very start the level of discussion is aimed a bit low, for example in lines 14-19. * It would be essential that the authors have the paper proofread for style and grammar issues. Currently, there are too many errors. Some examples (likely careless typos): a missing "is" in the sentence "it does not assume any generative model and based on..." in the abstract; should be "learner" or "learning algorithm"in the sentence "In PAC learning, the learning can access.."; the sentence "Another approach from the information theory literature studies with online lossless compression." does not make sense as stated but I'm not sure what the authors intended; "crucial" twice in a row around line 90; should be "if there exists" in line 96; "the function that compute" should be "... computes", "can efficiently solvable" in line 130, "matrice" should be "matrix" etc... * Another (relatively minor) presentation issue: the chosen typesetting/formatting style italicizes the words Definition and Theorem etc.. but leaves the body of these environments un-italicized. This does not seem standard in NIPS; more common and clearer in my opinion is boldface for Def/Thm and italics for body (also italics for the word Proof not boldface). Was the submitted version generated with the NIPS style file? Please verify. Also, when referring to a specific definition or theorem, I believe more common NIPS practice is to capitalize the word Definition/Theorem, e.g. "from Theorem X we see..." * Line 111, there is an errant period: "Notations: . " * "We keep a list of notations that we are using." - > change to for example "The following is a listing of notation that we use." * Line 126 "pretended to the new data points" should be changed. Also, state what \ell is here, or perhaps avoid this letter altogether since it is also used for the loss function. * The actual specification of the framework and discussion around it in Section 2 is well-done (in particular, here the authors assume just enough familiarity with standard theory and give a solid high-level explanation of why Theorem 2.1 holds based on this). Also high-level explanation for the proof of Theorem 3.1 is appreciated and convincing. The framework is very promising.  * The example and variant of framework in Section 4 on optimization encoding and efficient dictionary learning is nice. It is well-described. The authors also take care to explain its element of novelty on lines 231ff. The clearly presented proof overview which follows is greatly appreciated. It seems to correct to me, though I did not have time to read the detailed proof in the Appendix. * I am curious if the authors have considered auto encoders in the context of the CONUS framework. * Throughout the paper, the authors need to refer to a class of hypothesis class or a type of hypothesis class. I think it would be good to fix handling/wording for this at the start and consistently use it also in defining the framework in Section 2 + later discussions. As it is, one sees the wording "hypothesis class" in the framework definition, but "set of hypothesis" to refer to a type of hypothesis class on line 134. * The example and discussion in line 144ff is nice.