Imagination-Augmented Agents for Deep Reinforcement Learning
We introduce Imagination-Augmented Agents (I2As), a novel architecture for deep reinforcement learning combining model-free and model-based aspects. In contrast to most existing model-based reinforcement learning and planning methods, which prescribe how a model should be used to arrive at a policy, I2As learn to interpret predictions from a learned environment model to construct implicit plans in arbitrary ways, by using the predictions as additional context in deep policy networks. I2As show improved data efficiency, performance, and robustness to model misspecification compared to several baselines.

This paper presents an approach to model-based reinforcement learning where, instead of directly estimating the value of actions in a learned model, a neural network processes the model's predictions, combining with model-free features, to produce a policy and/or value function. The idea is that since the model is likely to be flawed, the network may be able to extract useful information from the model's predictions while ignoring unreliable information. The approach is studied in procedurally generated Sokoban puzzles and a synthetic Pac-Man-like environment and is shown to outperform purely model-free learning as well as MCTS on the learned model.
---Quality---
I feel the authors have done an exceptional job of presenting and evaluating their claims. The experiments are thorough and carefully designed to tease issues apart and to clearly answer well-stated questions about the approach. I found the experiments to provide convincing evidence that I2A is taking advantage of the learned model, is robust to model flaws, and can leverage the learned model for multiple tasks.
I had one question regarding the experiments comparing I2A and MCTS. I2A was trained for millions of steps to obtain a good policy. Was a comparable amount of data used to train the MCTS leaf evaluation function? I looked in the appendix, but didn't see that detail. Also, out of curiosity, if you have access to a perfect simulator, isn't it a bit funny to make I2A plan with the learned model? Why not skip the model learning part and train I2A to interpret rollouts from the simulator?
---Clarity---
I felt the paper was exceptionally well-written. The authors did a good job of motivating the problem and their approach and clearly explained the big ideas. As is often the case with deep learning papers, it would be *extremely* difficult to reproduce these results based purely on the paper itself -- there are always lots of details regarding architecture, meta-parameters, etc. However, the appendix does a good job of narrowing this gap. I hope the authors will consider publishing source code as well.
---Originality---
As far as I am aware, this is a highly novel approach to MBRL. I have never seen anything like it, and it is fundamentally, conceptually different than existing MBRL approaches.
The authors did a good job of citing relevant work (the references section is prodigious). Nevertheless, there has been some other recent work outside the context of deep nets studying the problem of planning with approximate, learned models that the authors may consider discussing (recognizing of course the limitations of space). For instance, Talvitie UAI 2014 and AAAI 2016, Jiang et al. AAMAS 2015, Venkatraman et al. ISER 2016. The presented approach is unquestionably distinct from these, but there may be connections or contrasts worth making. Also, the I2A architecture reminds me of the Dyna architecture, which similarly combines model-based and model-free updates into a single value function/policy (the difference being that it directly uses reward/transition predictions from the model). Again, I2A is definitely new, but the connection may be valuable to discuss.
---Significance---
I think this is an important paper. It is addressing an important problem in a creative and novel way, with demonstrably promising results. It seems highly likely that this paper will inspire follow-up work, since it effectively proposes a new category of MBRL algorithms that seems well-worth exploring.
***After author response***
I have read the authors' response and thank them for the clarifications.


Extracting low-dimensional dynamics from multiple large-scale neural population recordings by learning to predict correlations
A powerful approach for understanding neural population dynamics is to extract low-dimensional trajectories from population recordings using dimensionality reduction methods. Current approaches for dimensionality reduction on neural data are limited to single population recordings, and can not identify dynamics embedded across multiple measurements. We propose an approach for extracting low-dimensional dynamics from multiple, sequential recordings. Our algorithm scales to data comprising millions of observed dimensions, making it possible to access dynamics distributed across large populations or multiple brain areas. Building on subspace-identification approaches for dynamical systems, we perform parameter estimation by minimizing a moment-matching objective using a scalable stochastic gradient descent algorithm: The model is optimized to predict temporal covariations across neurons and across time. We show how this approach naturally handles missing data and multiple partial recordings, and can identify dynamics and predict correlations even in the presence of severe subsampling and small overlap between recordings. We demonstrate the effectiveness of the approach both on simulated data and a whole-brain larval zebrafish imaging dataset.

The authors suggest a method to estimate latent low dimensional dynamics from sequential partial observations of a population. Using estimates of lagged covariances, the method is able to reconstruct unobserved covariances quite well. The extra constraints introduced by looking at several lags allow for very small overlaps between the observed subsets.
This is an important work, as many imaging techniques have an inherent tradeoff between their sampling rate and the size of the population. A recent relevant work considered this from the perspective of inferring connectivity in a recurrent neural network (parameters of a specific nonlinear model with observations of the entire subset) [1].
The results seem impressive, with an ability to infer the dynamics even for very low overlaps, and the spiking network and zebrafish data indicate that the method works for certain nonlinear dynamics as well.
A few comments:
1.	In Figure 3b the error increases eventually, but this is not mentioned.
2.	Spiking simulation – the paper assumes a uniform sampling across clusters. How sensitive is it to under-sampling of some clusters?
3.	The zebrafish reconstruction of covariance. What is the estimated dimensionality of the data? The overlap is small in terms of percentage (1/41), but large in number of voxels (200K). Is this a case where the overlap is much larger than the dimensionality, and the constraints from lagged covariances do not add much? This example clearly shows the computational scaling of the algorithm, but I’m not sure how challenging it is in terms of the actual data.
[1] Soudry, Daniel, et al. "Efficient" shotgun" inference of neural connectivity from highly sub-sampled activity data." PLoS computational biology 11.10 (2015): e1004464.


Sparse convolutional coding for neuronal assembly detection
Cell assemblies, originally proposed by Donald Hebb (1949), are subsets of neurons firing in a temporally coordinated way that gives rise to repeated motifs supposed to underly neural representations and information processing. Although Hebb's original proposal dates back many decades, the detection of assemblies and their role in coding is still an open and current research topic, partly because simultaneous recordings from large populations of neurons became feasible only relatively recently. Most current and easy-to-apply computational techniques focus on the identification of strictly synchronously spiking neurons. In this paper we propose a new algorithm, based on sparse convolutional coding, for detecting recurrent motifs of arbitrary structure up to a given length. Testing of our algorithm on synthetically generated datasets shows that it outperforms established methods and accurately identifies the temporal structure of embedded assemblies, even when these contain overlapping neurons or when strong background noise is present. Moreover, exploratory analysis of experimental datasets from hippocampal slices and cortical neuron cultures have provided promising results. * Both authors contributed equally. † Majority of this work was done while co-author was at 3 . ‡ Majority of this work was done while co-author was at 1 .
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

The paper proposes a new method for the identification of spiking motifs in neuronal population data.  The authors’ approach is to use a sparse convolutional coding model that is learned using an alternation between LASSO and matching pursuit.  The authors address the problem of false positives and compare their algorithm to 3 competing methods in both simulation experiments and real neural data.
It appears that the basic model is not original, but that the regularization has been improved, leading to a method that performs well beyond the state of the art.
I believe that the method could be of some use to the neuroscience community as it may be powerful enough to determine if population codes fire in coordinated assemblies and if so, what is the nature of the assemblies?  This is a step towards addressing these long-standing questions in the neuroscience community.
Overall, the paper is clearly written, the method is reasonably conceived and described, and the simulation studies were sufficiently conceived for a paper of this length. Some of the methods for evaluating performance were unclear.  I am not totally certain how to assess Figure 4, for example since lines 169-175 where the “functional association” metric is described is too impoverished to determine what was actually done and what is being compared across detection thresholds.
I also am of the opinion that the authors are over-interpreting their data analysis results when they claim that the motif appearance is related to re-activation of former assemblies (do they mean replay?).
Also, it appears that few of the motif activations in Figure 7 include all of the cells that are a part of the motif.  This is concerning as it is not a part of the model description and makes it appear as though there is a poor model fit to the data.


Filtering Variational Objectives
When used as a surrogate objective for maximum likelihood estimation in latent variable models, the evidence lower bound (ELBO) produces state-of-the-art results. Inspired by this, we consider the extension of the ELBO to a family of lower bounds defined by a particle filter's estimator of the marginal likelihood, the filtering variational objectives (FIVOs). FIVOs take the same arguments as the ELBO, but can exploit a model's sequential structure to form tighter bounds. We present results that relate the tightness of FIVO's bound to the variance of the particle filter's estimator by considering the generic case of bounds defined as log-transformed likelihood estimators. Experimentally, we show that training with FIVO results in substantial improvements over training the same model architecture with the ELBO on sequential data.

This paper introduces Monte Carlo objective to improve the scaling of the IWAE and provides a tighter lower bound to the log likelihood than traditional ELBO. Various theoretical properties of Monte Carlo objectives are discussed. An example of Monte Carlo objective is provided to use particle filters for modeling sequential data. The paper is generally well written and I enjoyed reading it.
* For the FIVO algorithm, it'd be more self-contained if including a brief description of the resampling criteria or its exact procedure used in the experiment, so that the reference doesn't need to be consulted.
* Although not required, it'd be helpful to provide extra examples of Monte Carlo objectives other than applications of sequential data as FIVOs.
* I feel it's fairer to stay with a batch of 4 instead of 4N for ELBO to reduce the impact of different batch configurations, even if the computational budget does not match up (line 259). The authors could also consider reporting the running time of each method, and normalize it somehow when interpreting the results. This could also give readers some idea about the overhead of resampling.
* I'm a bit worried about the use of the biased gradients to train FIVO. Is it possible that part of the superior performance of FIVO over IWAE somehow benefits from training using the incorrect gradients? It'd be useful to also report the performance of FIVO trained with the unbiased gradients so that the readers will have the idea about the impact of the large variance coming from the resampling term.
* It's not clear to me why checking on the KL divergence between the inference model q(z|x) to the prior p(z) addresses the need to make use of the uncertainty (line 279 to 283). What's the connection between "the degree to which each model uses the latent states" and the KL divergence? How to interpret ELBO's KL collapses during training? Doesn't the inference model collapses onto the prior suggest that it carries less information about the data?
* Typo: it should be z ~ q(z|x) in line 111-112.


Uprooting and Rerooting Higher-Order Graphical Models
The idea of uprooting and rerooting graphical models was introduced specifically for binary pairwise models by Weller [19]  as a way to transform a model to any of a whole equivalence class of related models, such that inference on any one model yields inference results for all others. This is very helpful since inference, or relevant bounds, may be much easier to obtain or more accurate for some model in the class. Here we introduce methods to extend the approach to models with higher-order potentials and develop theoretical insights. In particular, we show that the triplet-consistent polytope TRI is unique in being 'universally rooted'. We demonstrate empirically that rerooting can significantly improve accuracy of methods of inference for higher-order models at negligible computational cost.

This paper presents a reparametrization method for inference in undirected graphical models. The method generalizes the uprooting and rerooting approach from binary pairwise graphical models to binary graphical models with factors of arbitrary arity. At the heart of the method is the observation that all non-unary potentials can be made symmetric, and once this has been done, the unary potentials can be changed to pairwise potentials to obtain a completely symmetric probability distribution, one where x and its complement have the exact same probability. This introduces one extra variable, making the inference problem harder. However, we can now "clamp" a different variable from the original distribution, and if we choose the right variable, approximate inference might perform better in the new model than in the original. Inference results in the reparametrized model easily translate to inference results in the original model.
The idea is supported by a number of theorems. I believe the results, but I did not study the proofs in the supplementary material. In addition to the main result, which is that inference results in the new model can be mapped back to inference results in the original model, the authors also discuss how the reparametrized problems relate to each other in the marginal polytope and how factors can be decomposed into sums of "even" factors -- basically, parity over k variables.
There are also some basic experiments on small, synthetic datasets which demonstrate that rerooting can yield better approximations in some cases.
Given that the negligible cost of reparametrization and mapping the results, this seems like a promising way to improve inference. There's no guarantee that the results will be better, of course -- in general, rerooting could lead to better or worse results. Furthermore, we can clamp variables without rerooting, at the cost of doubling the inference time for each clamped variable. (Just run inference twice, once with each value clamped.) If rerooting is only done once, then you're getting at best a factor-of-two speedup compared to clamping. A factor of two is a small constant, and I didn't see any discussion of how rerooting could be extended to yield better results. So the practical benefit from the current work may be marginal, but the ideas are interesting and could be worth developing further in future work.
Quality: I believe the quality is high, though as I mentioned, I have not studied the proofs in the supplementary material. The limitations are not discussed in detail, but this is typical for conference papers.
Clarity: The writing was fairly clear. Figure 2 has very small text and is uninterpretable in a black and white printout.
Originality: I believe this is original, but it's a natural evolution of previous work on rerooting. This is a meaningful increment, supported by theoretical results and token empirical results, but not transformative.
Significance: As stated earlier, it's not clear that these methods will yield any improvements on real-world inference tasks (yet). And if they do, it's at best a factor-of-two speedup. Thus, this work will mainly be relevant to people working in the area of graphical models and probabilistic inference. It's still significant to this group, though, since the results are interesting and worthy of further exploration.


Quantifying how much sensory information in a neural code is relevant for behavior
Determining how much of the sensory information carried by a neural code contributes to behavioral performance is key to understand sensory function and neural information flow. However, there are as yet no analytical tools to compute this information that lies at the intersection between sensory coding and behavioral readout. Here we develop a novel measure, termed the information-theoretic intersection information I II (S; R; C), that quantifies how much of the sensory information carried by a neural response R is used for behavior during perceptual discrimination tasks. Building on the Partial Information Decomposition framework, we define I II (S; R; C) as the part of the mutual information between the stimulus S and the response R that also informs the consequent behavioral choice C. We compute I II (S; R; C) in the analysis of two experimental cortical datasets, to show how this measure can be used to compare quantitatively the contributions of spike timing and spike rates to task performance, and to identify brain areas or neural populations that specifically transform sensory information into choice.

This manuscript proposes a new information-theoretic measure for quantifying the amount of information about the stimuli carried in the neural response, which is also used for behavior for a perceptual discrimination tasks. This work builds on the previous literature of Partial Information Decomposition. The information-theoretic measure is tested using a simulation and two published experimental datasets.
Overall, I find that this is a potentially interesting contribution. The manuscript is generally well-written, except it misses the definitions of a few key quantities. The general approach the authors use is principled and potentially lead to better way to examine the relation between the neural response, behavior and the stimuli.
I have a few general as well as technical concerns about the applicability of this measure in typical experiments which I will detail below.
Major concerns:
Eq.(1) is a key equation, however, the terms at the r.h.s are not defined, and only intuitions are given. For example, how to compute SI(C:{S,R})? I think it would be useful to have that spell out explicitly. Minor comment- there is a typo in the last two terms, and now they appear to be identical.
Eq. (2) gives the definition of the proposed new measure. However, what is the difference between this measure, and the bivariate redundancy measure proposed in Harder et al. (PRE, 2013; reference (21) in the submission)? In Harder et al., a measure with similar structure is proposed, see their Eq. (13). It is not clear to me how the "new definition" the authors proposed here is different from the Eq. (13) there.  Many of the properties of the measure proposed here seems to be shared with the measure defined there. However, since the definition of SI(C:{S,R}) is not given in the current paper, I might be missing something important here. In any case, I'd like to understand whether the measure defined in Eq. (2) is indeed different from Eq. (13) of Harder et al (2013), as this seems to be critical for judging the novelty of the current paper.
Regardless of whether the measure the authors proposed is indeed novel, there is a crucial question in terms of  practically how useful this method is. I'd think one of the main challenges for using this method is estimating the joint probably of the stimulus, activity and the choice. But in the Supplementary Information, the authors only explained how to proceed after assuming this joint probability has been estimated. Do the authors make any important assumptions to simplify the computations here, in particular for the neural response R? For example, is the independence across neurons assumed? Furthermore, in many tasks with complex stimuli such as in many vision experiments, I'd think it is very difficult to estimate even the prior distribution on the stimulus, not to say the joint probability of the stimulus, the activity and the behavior. I understand that the authors here only focus on simple choice tasks in which the stimuli are modeled as binary variables, but the limitations of this approach should be properly discussed.
In the experimental section, it seems that the number information is a bit small, e.g., ~10^{-3} bit. Can these number map to the values of choice probability and are they compatible? In general, it might be useful to discuss the relation of this approach to the approach of using choice probability, both in terms of the pros and the cons, as well as in which cases they relate to each other.
In summary, my opinion is that the theoretical aspects of the paper seems to be incremental. However, I still find the paper to be of potential interests to the neurophysiologists. I have some doubts on the general applicability of the methods, as well as the robustness of the conclusions that can be drawn from this method, but still this paper may represent an interesting direction and may prove useful for studying the neural code of choice behavior.
Minor:
In eq. (2), the notation for I_{II}(R) is confusing, because it doesn't contain (R,C), but nonetheless it depends on them.  This further leads to the confusion for I_{II}(R1,R2) at the top of line 113. What does it mean?


Temporal Coherency based Criteria for Predicting Video Frames using Deep Multi-stage Generative Adversarial Networks
Predicting the future from a sequence of video frames has been recently a sought after yet challenging task in the field of computer vision and machine learning. Although there have been efforts for tracking using motion trajectories and flow features, the complex problem of generating unseen frames has not been studied extensively. In this paper, we deal with this problem using convolutional models within a multi-stage Generative Adversarial Networks (GAN) framework. The proposed method uses two stages of GANs to generate crisp and clear set of future frames. Although GANs have been used in the past for predicting the future, none of the works consider the relation between subsequent frames in the temporal dimension. Our main contribution lies in formulating two objective functions based on the Normalized Cross Correlation (NCC) and the Pairwise Contrastive Divergence (PCD) for solving this problem. This method, coupled with the traditional L1 loss, has been experimented with three real-world video datasets viz. Sports-1M, UCF-101 and the KITTI. Performance analysis reveals superior results over the recent state-of-the-art methods.

This method provides 2 contributions for next frame prediction from video sequences.
The first is the introduction of a normalized cross correlation loss, which provide a better similarity score to judge if the predicted frame is close to the true future. The second is the pairwise contrastive divergence loss, based on the idea of similarity of the image features. Results are presented on the UCF101 and Kitti datasets, and a numerical comparison using image similarity metrics (PSNR, SSIM) with Mathieu et al ICLR16 is performed.
Comments:
The newly proposed losses are interesting, but I suspect a problem in the evaluation.
I fear that the evaluation protocol comparing the present approach to previous work [13] is different to [13]: In the table, the authors don't mention evaluating their method on 10% of the test set and only in the moving area as Mathieu et al. did.
Did the authors performed the sanity check of comparing the obtained PSNR/SSIM score of frame copy to see if they match the ones reported in [13]?
Please be more specific about the details of the evaluation (full test set? Full images? -> in this case, what are the scores for a frame copy baseline?)
The authors answered my concerns in the rebuttal.
- Providing generated videos for such paper submission would be appreciated.
In the displayed examples, including the ones from the supplementary material, the movements are barely visible.
- Predicting several frames at a time versus one?
In Mathieu et al. work, results showed that predicting only one frame at the time worked better. Have you tried such experiment?
l.138: I don't understand the h+4?
Minor
The first sentence, "Video frame prediction has always been one of the fundamental problems in computer vision as it caters to a wide range of applications [...]" is somehow controversial in my opinion: The task was, in the best of my knowledge, introduced only in 2014. As for the applications, even if i believe they will exist soon, I am not aware of systems using next frames predictions yet.
l. 61 uses -> use
l. 65 statc -> statistical ?
l 85. [5]is -> [5] is
Eq 4: use subscripts instead of exponents for indexes
l 119 (b)P -> (b) P
l 123-125 weird sentence
l. 142 ,u -> , u
l. 153 tries to -> is trained to
l. 163 that, -> that
l. 164 where, -> where
[13] ICLR 16
[17] ICLR 16
please check other references


On Frank-Wolfe and Equilibrium Computation
We consider the Frank-Wolfe (FW) method for constrained convex optimization, and we show that this classical technique can be interpreted from a different perspective: FW emerges as the computation of an equilibrium (saddle point) of a special convex-concave zero sum game. This saddle-point trick relies on the existence of no-regret online learning to both generate a sequence of iterates but also to provide a proof of convergence through vanishing regret. We show that our stated equivalence has several nice properties, as it exhibits a modularity that gives rise to various old and new algorithms. We explore a few such resulting methods, and provide experimental results to demonstrate correctness and efficiency.

The paper draws a connection between the classical Frank-Wolfe algorithm for constrained smooth & convex optimization (aka conditional gradient method) and using online learning algorithms to solve zero-sum games. This connection is made by casting the constrained convex optimization problem as a convex-concave saddle point problem between a player that takes actions in the feasible set and another player that takes actions in the gradient space of the objective function. This saddle point problem is derived using the  Flenchel conjugate of the objective. Once this is achieved, a known and well explored paradigm of using online learning algorithms can be applied to solving this saddle point problem (where each player applies its own online algorithm to either minimize or maximize), and the average regret bounds obtained by the algorithms translate back to the approximation error with respect to the objective on the original offline convex optimization problem.
The authors show that by applying this paradigam with different kinds of online learning algorithms, they can recover the original Frank-Wolfe algorithm (though with a slightly different step size and rate worse by a factor of log(T)) and several other variants, including one that uses the averaged gradient, using stochastic smoothing for non-smooth objectives and even a new variant that converges for non-smooth objectives (without smoothing), when the feasible set is strongly convex.
***Criticism***
Pros:
I think this is a really nice paper, and I enjoyed reading it. While, in hindsight, the connection between offline optimization and zero-sum games via Fenchel conjugates is not surprising, I find this perspective very refreshing and original, in particular in light of the connection with online learning. I think NIPS is the best place for works making such a connection between offline optimizaiton and online learning. I truly believe this might open a window to obtain new and improved projection-free algorithms via the lens of online learning, and this is the main and worthy contribution of this paper, in my opinion. This view already allows the authors to give a FW-like algorithm for non-smooth convex optimization over strongly convex sets which overcomes the difficulties of the original method. There is an easy example of minimizing a piece-wise linear function over the euclidean unit ball for which FW fails, but this new variant works (the trick is to indeed use the averaged gradient). Unfortunately, as I discuss next, the authors do a somewhat sloppy work presenting this result.
Cons:
- I find Section 4.3 to be very misleading. Algorithm 5 is a first-order method for convex non-smooth optimization. We know that in this setting the best possible (worst-case) rate is 1/\eps^2. The construction is simply minimization of a piece-wise linear function over a unit Euclidean ball (which is certainly strongly convex with a constant strong convexity parameter). Hence, claiming Algorithm 5 obtains a log(T)/T rate is simply not true and completely misleading. Indeed, when examining the details of the above ``bad" construction, we have that L_T will be of order \eps when aiming for \eps error. The authors should do a better job in correctly placing the result in the context of offline optimization, and not simply doing plug&play from the online world. Again, stating the rate as log(T)/T is simply misleading. In the same vain, I find the claims on lines 288-289 to be completely false.
It is also disappointing and in a sense unsatisfactory that the algorithm completely fails when gradient=0 is achievable for the problem at hand. I do hope the authors will be able to squeeze more out of this very nice result (as I said, it overcomes a standard lower bound for original FW I had in mind).
Minor things:
- When referring to previous works [13,15] the authors wrongly claim that they rely on a stronger oracle. They also require the standard linear oracle but use it to construct a stronger oracle. Hence, they work for every polytope.
- In Definition 2 is better to state it with gradient instead of subgradient since we assume the function is smooth.
-Line 12: replace an with a
- on several occasions when using ``...=argmin", note in general the argmin is a set, not a singleton. Same goes for the subdifferential.
-Line 103: I believe ``often times" should be a single word
- Theorem 3: should state that there exists a choice for the sequence \gamma_t such that...
- Theorem 5: should also state the gamma_t in the corresponding Algorithm 2
- Supp material Definition on line 82: centered equation is missing >=0. Also, ``properly contains" in line 84 not defined.
- Supp material: Section 4 is missing (though would love to hear the details (: )
*** post rebuttal ***
I've read the rebuttal and overall satisfied with the authors response.


Style Transfer from Non-Parallel Text by Cross-Alignment
This paper focuses on style transfer on the basis of non-parallel text. This is an instance of a broad family of problems including machine translation, decipherment, and sentiment modification. The key challenge is to separate the content from other aspects such as style. We assume a shared latent content distribution across different text corpora, and propose a method that leverages refined alignment of latent representations to perform style transfer. The transferred sentences from one style should match example sentences from the other style as a population. We demonstrate the effectiveness of this cross-alignment method on three tasks: sentiment modification, decipherment of word substitution ciphers, and recovery of word order.

This paper presents a method for learning style transfer models based on non-parallel corpora. The premise of the work is that it is possible to disentangle the style from the content and that when there are two different corpora on the same content but in distinctly different styles, then it is possible to induce the content and the style components. While part of me is somewhat skeptical whether it is truly possible to separate out the style from the content of natural language text, and that I tend to think sentiment and word-reordering presented in this work as applications correspond more to the content of an article than the style, I do believe that this paper presents a very creative and interesting exploration that makes both theoretical and empirical contributions.
— (relatively minor) questions:
- In the aligned auto-encoder setting, the discriminator is based on a simple feedforward NN, while in the cross-aligned auto-encoder setting, the discriminators are based on convolution NNs. I imagine ConvNets make stronger discriminators, thus it’d be helpful if the paper can shed lights on how much the quality of the discriminators influence the overall performance of the generators.
- Some details on the implementation and experiments are missing, which might matter for reproducibility. For example, what kind of RNNs are used for the encoder and the generator? What kind of convolution networks are used for the discriminators? What are the vocabulary sizes (and the corresponding UNK scheme) for different datasets? The description for the word reordering dataset is missing. If the authors will share their code and the dataset, it’d help resolving most these questions, but still it’d be good to have some of these detailed specified in the final manuscript.


QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to its excellent scalability properties. A fundamental barrier when parallelizing SGD is the high bandwidth cost of communicating gradient updates between nodes; consequently, several lossy compresion heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always converge. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes with convergence guarantees and good practical performance. QSGD allows the user to smoothly trade off communication bandwidth and convergence time: nodes can adjust the number of bits sent per iteration, at the cost of possibly higher variance. We show that this trade-off is inherent, in the sense that improving it past some threshold would violate information-theoretic lower bounds. QSGD guarantees convergence for convex and non-convex objectives, under asynchrony, and can be extended to stochastic variance-reduced techniques. When applied to training deep neural networks for image classification and automated speech recognition, QSGD leads to significant reductions in end-to-end training time. For instance, on 16GPUs, we can train the ResNet-152 network to full accuracy on ImageNet 1.8× faster than the full-precision variant.

Update: I decrease slightly the grade due to the mismatch between theoretical and practical results that could be better covered. Still this paper has strong experimental results and some theoretical results. I would encourage the authors to improve on the gap between the two.
In this paper the author introduce Quantized SGD (QSGD), a scheme for reducing the communication cost of SGD when performing distributed optimization. The quantization scheme is useful as soon as one has to transmit gradients between different machines. The scheme is inspired by 1 BitSGD where 32 bits are used to transfer the norm of the gradient and then 1 bit per coordinate to encode the sign. The author extends this approach by allowing to send extra bits per coordinate to encode the scale of the coordinate. They obtain an unbiaised stochastic quantized version of the gradient. Therefore, one can still see QSGD as an SGD scheme, but with extra variance due to the quantization. Because of the quantization of the scale of each coordinate, small coordinates will be approximated by 0 and therefore the quantized gradient will be sparse, unlike 1BitSGD which gives a dense gradient. The authors use this property to further compress the gradient by encoding the non zero coordinates and only transfering information for those. In order to encode the position of those coordinates efficiently, the authors uses the Elias code, a variable length integer encoding scheme.
The authors then applies this approach to deep learning model training.
The authors also present how to quantize SVRG, a variance reduced SGD method with linear rate of convergence that can be used for convex optimization.
Strengths
=========
- The introduction is very clear and the authors do a great job at presenting the source of inspiration for this work, 1BitSGD. They also explain the limits of 1BitSGD (namely that there is no convergence guarentees) and clearly state the contribution of their paper (the ability to choose the number of bits and therefore the trade off between precision and communication cost, as well as having convergence guarantees).
- the authors reintroduce the main results on SGD in order to properly understand the convergence of SGD in relation with the variance or second order moment of the gradients. This allow the reader to quickly understand how the quantization will impact convergence without requiring in depth technical knowledge of SGD.
- As a consequence of the above points, this paper is very well self contained and can be read even with little knowledge about quantization or SGD training.
- The authors did a massive investigation work, regarding both synchronous and asynchronous SGD, convex and non convex, as well as variance reduced SGD. Despite all this work, the authors did a good job at keeping the reading complexity low by separating properly each concern, sticking to simple case in the main paper (synchronous SGD) and keeping more complex details for the supplementary materials.
- This paper contains a large amount of experiments, on different datasets, different architecture and comparing with existing methods such as 1bit SGD.
- This paper offers a non trivial extension of 1bit SGD. For instance this method introduce sparsity in the gradient update while keeping convergence guarantees.
Improvements
============
- The result on sparsity in Lemma 3.1, whose proof is in Lemma A.5 in the supplementary material seems wrong to me. Line 514 in the supplementary materials, first `u_i` should be replaced by `|u_i|` but more importantly, according to the definition on line 193 in the main paper, the probability is not `|u_i|` but `s |u_i|`. Thus, the sparsity bound becomes `s^2 + sqrt(n) s`. I would ask the authors to confirm this and update any conclusion that would be invalidated. This will have an impact on Lemma A.2. I would also encourage the authors to move this proof with the rest of the proof of Lemma 3.1.
- Line 204, the description of Elias(k) is not super clear to me, maybe giving a few examples would make it better. The same applies to the description in the supplementary materials. Besides, the authors should cite the original Elias paper [1].
- The authors could comment on how far is the synchronous data-parallel SGD from current state of the art practice in distributed deep learning optimization. My own experience with mini-batches is that although theoretically and asymptotically using a batch size of B can make convergence take B times less iterations, this is not always true especially when far from the optimum, see [2] for instance, in practice the `L R^2/T` can dominate training for a significant amount of time. Other approaches than mini-batch exist, such as EASGD. I am not that familiar with the state of the art in distributed deep learning but I believe this paper could benefit from giving more details on the state of the art techniques and how QSGD can improve them (for instance for EASGD I am guessing that QSGD can allow for more frequent synchronization between the workers). Besides the model presented in this paper requires O(W^2) communication at every iteration where W is the number of worker, while only O(W) could be used with a central parameter server, thus this paper setup is particularly beneficial to QSGD. Such an approach is shortly described in the supplementary materials but as far as I could see there was no experiment with it.
Overall this papers contains novel ideas, is serious and well written and therefore I believe it belongs at NIPS. It has detailed theoretical and experimental arguments in favor of QSGD. As noted above, I would have enjoyed more context on the impact on state of art distributed methods and there is a mistake in one of the proof that needs to be corrected.
References
==========
[1] Elias, P. (1975). Universal codeword sets and representations of the integers. IEEE transactions on information theory.
[2] Sutskever, Ilya, et al. On the importance of initialization and momentum in deep learning. ICML 2013


Modulating early visual processing by language
It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic inputs are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the entire visual processing by a linguistic input. Specifically, we introduce Conditional Batch Normalization (CBN) as an efficient mechanism to modulate convolutional feature maps by a linguistic embedding. We apply CBN to a pre-trained Residual Network (ResNet), leading to the MODulatEd ResNet (MODERN) architecture, and show that this significantly improves strong baselines on two visual question answering tasks. Our ablation study confirms that modulating from the early stages of the visual processing is beneficial.

The paper proposes a novel method called conditional batch normalization (CBN) to be applied on top of existing visual question answering models in order to modulate the visual processing with language information from the question in the early stages. In the proposed method, only the parameters of the batch norm layer of a pre-trained CNN are updated with the VQA loss by conditioning them on the LSTM embedding of the input question.
The paper evaluates the effectiveness of CBN on two VQA datasets – the VQA dataset from Antol et al., ICCV15 and the GuessWhat?! dataset from Vries et al., CVPR17. The experimental results show that CBN helps improve the performance on VQA by significant amount. The paper also studies the effectiveness of adding CBN to different layers and shows that adding CBN to last (top) 2 layers of CNN helps the most. The paper also shows quantitatively that the improvements in VQA performance are not merely due to fine-tuning of CNN by showing that the proposed model performs better than a model in which the Batch Norm parameters are fine-tuned but without conditioning on the language. Hence demonstrating that modulating with language helps.
Strengths:
1.	The paper is well-motivated and the idea of modulating early visual processing by language is novel and interesting for VQA task.
2.	The proposed contribution (CBN) can be added on top of any existing VQA model, hence making it widely applicable.
3.	The ablation studies are meaningful and are informative about how much of early modulation by language helps.
4.	The paper provides the details of the hyper-parameters, hence making the work reproducible.
Weaknesses:
1.	The main contribution of the paper is CBN. But the experimental results in the paper are not advancing the state-of-art in VQA (on the VQA dataset which has been out for a while and a lot of advancement has been made on this dataset), perhaps because the VQA model used in the paper on top of which CBN is applied is not the best one out there. But in order to claim that CBN should help even the more powerful VQA models, I would like the authors to conduct experiments on more than one VQA model – favorably the ones which are closer to state-of-art (and whose codes are publicly available) such as MCB (Fukui et al., EMNLP16), HieCoAtt (Lu et al., NIPS16). It could be the case that these more powerful VQA models are already so powerful that the proposed early modulating does not help. So, it is good to know if the proposed conditional batch norm can advance the state-of-art in VQA or not.
2.	L170: it would be good to know how much of performance difference this (using different image sizes and different variations of ResNets) can lead to?
3.	In table 1, the results on the VQA dataset are reported on the test-dev split. However, as mentioned in the guidelines from the VQA dataset authors (http://www.visualqa.org/vqa_v1_challenge.html), numbers should be reported on test-standard split because one can overfit to test-dev split by uploading multiple entries.
4.	Table 2, applying Conditional Batch Norm to layer 2 in addition to layers 3 and 4 deteriorates performance for GuessWhat?! compared to when CBN is applied to layers 4 and 3 only. Could authors please throw some light on this? Why do they think this might be happening?
5.	Figure 4 visualization: the visualization in figure (a) is from ResNet which is not finetuned at all. So, it is not very surprising to see that there are not clear clusters for answer types. However, the visualization in figure (b) is using ResNet whose batch norm parameters have been finetuned with question information. So, I think a more meaningful comparison of figure (b) would be with the visualization from Ft BN ResNet in figure (a).
6.	The first two bullets about contributions (at the end of the intro) can be combined together.
7.	Other errors/typos:
a.	L14 and 15: repetition of word “imagine”
b.	L42: missing reference
c.	L56: impact -> impacts
Post-rebuttal comments:
The new results of applying CBN on the MRN model are interesting and convincing that CBN helps fairly developed VQA models as well (the results have not been reported on state-of-art VQA model). So, I would like to recommend acceptance of the paper.
However I still have few comments --
1. It seems that there is still some confusion about test-standard and test-dev splits of the VQA dataset. In the rebuttal, the authors report the performance of the MCB model to be 62.5% on test-standard split. However, 62.5% seems to be the performance of the MCB model on the test-dev split as per table 1 in the MCB paper (https://arxiv.org/pdf/1606.01847.pdf).
2. The reproduced performance reported on MRN model seems close to that reported in the MRN paper when the model is trained using VQA train + val data. I would like the authors to clarify in the final version if they used train + val or just train to train the MRN and MRN + CBN models. And if train + val is being used, the performance can't be compared with 62.5% of MCB because that is when MCB is trained on train only. When MCB is trained on train + val, the performance is around 64% (table 4 in MCB paper).
3. The citation for the MRN model (in the rebuttal) is incorrect. It should be --
@inproceedings{kim2016multimodal,
  title={Multimodal residual learning for visual qa},
  author={Kim, Jin-Hwa and Lee, Sang-Woo and Kwak, Donghyun and Heo, Min-Oh and Kim, Jeonghee and Ha, Jung-Woo and Zhang, Byoung-Tak},
  booktitle={Advances in Neural Information Processing Systems},
  pages={361--369},
  year={2016}
}
4. As AR2 and AR3, I would be interested in seeing if the findings from ResNet carry over to other CNN architectures such as VGGNet as well.


Scalable Planning with Tensorflow for Hybrid Nonlinear Domains
Given recent deep learning results that demonstrate the ability to effectively optimize high-dimensional non-convex functions with gradient descent optimization on GPUs, we ask in this paper whether symbolic gradient optimization tools such as Tensorflow can be effective for planning in hybrid (mixed discrete and continuous) nonlinear domains with high dimensional state and action spaces? To this end, we demonstrate that hybrid planning with Tensorflow and RMSProp gradient descent is competitive with mixed integer linear program (MILP) based optimization on piecewise linear planning domains (where we can compute optimal solutions) and substantially outperforms state-of-the-art interior point methods for nonlinear planning domains. Furthermore, we remark that Tensorflow is highly scalable, converging to a strong plan on a large-scale concurrent domain with a total of 576,000 continuous action parameters distributed over a horizon of 96 time steps and 100 parallel instances in only 4 minutes. We provide a number of insights that clarify such strong performance including observations that despite long horizons, RMSProp avoids both the vanishing and exploding gradient problems. Together these results suggest a new frontier for highly scalable planning in nonlinear hybrid domains by leveraging GPUs and the power of recent advances in gradient descent with highly optimized toolkits like Tensorflow.

This paper describes a gradient ascent approach to action selection for planning domains with both continuous and discrete actions. The paper shows that backpropagation can be used to optimize the actions while avoiding long horizon problems such as exploding or vanishing gradients. Experimental results are given for 2D planar navigation, reservoir control and HVAC control and the proposed approach is shown to outperform existing hybrid optimization techniques.
Planning in hybrid domains is an important problem and existing methods do not scale to real world domains. This paper proposes a scalable solution to these problems and demonstrates its usefulness on several domains including two real world domains. As such, I believe it makes an important contribution to addressing real world planning problems. The paper is well written except for some minor comments (see below).
Evidence that the proposed solution works comes in the form of several experiments that demonstrate scalability and solution quality.
I have two concerns about the experiments that I would like to see the author's comments on:
1. The paper lacks any statistical significance results, error bars, or other statistics of variation.
2. Are the experimental baselines the state-of-the-art of hybrid domain planning problems? The introduction mentions other methods but justification is not given for not making an empirical comparison.
This paper needs a distinct related work section. The current treatment of related work is limited and is just in the introduction.
My last major question is whether or not Tensorflow is a critical part of the proposed method? It seems that what is important is the use of backpropagation and Tensorflow is just an implementation detail. I think the impact of the work might be more if the method was presented more generally. Or is there something I'm missing that makes Tensorflow essential?
Overall, the work presents a useful method for planning in hybrid domain problems. Its main limitations are the handling of related literature and statistical significance. Clarifying these issues and addressing the minor comments given below would help improve the impact of the work.
I have read the author's response and other reviews.
Minor comments:
Line 35: I.e., -> i.e.,
Line 35: Should cite Watkins for Q-learning
Line 106: Do you mean optimizing V directly?
Line 117: What do you mean by transfer function? Did you mean activation function?
Line 134: This statement seems too strong? Why are there always constraints on the actions? Is it more appropriate to say, "planning problems frequently have constraints on the actions that can be taken"?
Line 164 - 167: The top line (d_t = ||s_t - s||) is too far over.
Line 170: It is not clear what the dynamics are for linear and bilinear. It would be useful to write them explicitly as done for the nonlinear dynamics.
Figure 3: This figure would be more clear if the y-axis was in the correct direction. It would also be useful to have a statement such as "lower is better" so the reader can quickly compare TF to the other baselines.
Figure 3: Since the MILP solution is optimal it would be more useful to see relative error values for TF and Heuristic. Absolute reward isn't as informative since its sensitive to reward scaling.
All figures: It would be nice to see error bars on all applicable plots
Line 235: "to for" -> for
Figure 6: space before (b)
All figures: check for black and white printing, especially figure 6
Line 262: RMSProb -> RMSProp


Boltzmann Exploration Done Right
Boltzmann exploration is a classic strategy for sequential decision-making under uncertainty, and is one of the most standard tools in Reinforcement Learning (RL). Despite its widespread use, there is virtually no theoretical understanding about the limitations or the actual benefits of this exploration scheme. Does it drive exploration in a meaningful way? Is it prone to misidentifying the optimal actions or spending too much time exploring the suboptimal ones? What is the right tuning for the learning rate? In this paper, we address several of these questions for the classic setup of stochastic multi-armed bandits. One of our main results is showing that the Boltzmann exploration strategy with any monotone learning-rate sequence will induce suboptimal behavior. As a remedy, we offer a simple non-monotone schedule that guarantees near-optimal performance, albeit only when given prior access to key problem parameters that are typically not available in practical situations (like the time horizon T and the suboptimality gap ∆). More importantly, we propose a novel variant that uses different learning rates for different arms, and achieves a distribution-dependent regret bound of order K log 2 T ∆ and a distributionindependent bound of order √ KT log K without requiring such prior knowledge. To demonstrate the flexibility of our technique, we also propose a variant that guarantees the same performance bounds even if the rewards are heavy-tailed.
in reinforcement learning [23,25,14,26,16,18]. In the multiarmed bandit literature, exponentialweights algorithms are also widespread, but they typically use importance-weighted estimators for the rewards -see, e.g., [6,8] (for the nonstochastic setting), [12] (for the stochastic setting), and [20] (for both stochastic and nonstochastic regimes). The theoretical behavior of these algorithms is generally well understood. For example, in the stochastic bandit setting Seldin and Slivkins [20] show a regret bound of order
, where ∆ is the suboptimality gap (i.e., the smallest difference between the mean reward of the optimal arm and the mean reward of any other arm).
In this paper, we aim to achieve a better theoretical understanding of the basic variant of the Boltzmann exploration policy that relies on the empirical mean rewards. We first show that any monotone learning-rate schedule will inevitably force the policy to either spend too much time drawing suboptimal arms or completely fail to identify the optimal arm. Then, we show that a specific non-monotone schedule of the learning rates can lead to regret bound of order K log T ∆ 2 . However, the learning schedule has to rely on full knowledge of the gap ∆ and the number of rounds T . Moreover, our negative result helps us to identify a crucial shortcoming of the Boltzmann exploration policy: it does not reason about the uncertainty of the empirical reward estimates. To alleviate this issue, we propose a variant that takes this uncertainty into account by using separate learning rates for each arm, where the learning rates account for the uncertainty of each reward estimate. We show that the resulting algorithm guarantees a distribution-dependent regret bound of order
, and a distribution-independent bound of order √ KT log K.
Our algorithm and analysis is based on the so-called Gumbel-softmax trick that connects the exponential-weights distribution with the maximum of independent random variables from the Gumbel distribution.

Pros:
      	- A systematic study on the classical Boltzmann exploration heuristic in the context of multi-armed bandit. The results provide useful insights to the understanding of Boltzmann exploration and multi-armed bandits
      	- The paper is clearly written
      	Cons:
      	- The technique is incremental, and the technical contribution to multi-armed bandit research is small.
      	The paper studiee Boltzmann exploration heuristic for reinforcement learning, namely use empirical means and exponential weight to probabilistically select actions (arms) in the context of multi-armed bandit. The purpose of the paper is to achieve property theoretical understanding of the Boltzmann exploration heuristic. I view that the paper achieves this goal by several useful results. First, the authors show that the standard Boltzmann heuristic may not achieve good learning result, in fact, the regret could be linear, when using monotone learning rates. Second, the authors show that, if the learning rate remain constant for a logarithmic number of steps and then increase, the regret is close to the optimal one. This learning strategy is essentially explore-then-commit strategy, but the catch is that it needs to know the critical problem parameter \Delta and T, which are typically unknown. Third, the authors propose to generalize the Boltzmann exploration by allowing individual learning rates for different arms based on their certainty, and show that this leads to good regret bounds.
      	The above serious of results provide good understanding of Boltzmann exploration. In particular, it provides the theoretical insight that the naive Boltzmann exploration lacks control on the uncertainty of arms, so may not preform well. This insight may be useful in the more general setting of reinforcement learning.
      	The paper is in general well written and easy to follow.
      	The technical novelty of the paper is incremental. The analysis are based on existing techniques. The new technical contribution to the multi-armed bandit research is likely to be small, since there are already a number of solutions achieving optimal or new optimal regret bounds.
      	Minor comments:
      	- line 243, log_+(\cdot) = min{0, \cdot}. Should it be max instead of min?


Learning Mixture of Gaussians with Streaming Data
In this paper, we study the problem of learning a mixture of Gaussians with streaming data: given a stream of N points in d dimensions generated by an unknown mixture of k spherical Gaussians, the goal is to estimate the model parameters using a single pass over the data stream. We analyze a streaming version of the popular Lloyd's heuristic and show that the algorithm estimates all the unknown centers of the component Gaussians accurately if they are sufficiently separated. Assuming each pair of centers are Cσ distant with C = Ω((k log k) 1/4 σ) and where σ 2 is the maximum variance of any Gaussian component, we show that asymptotically the algorithm estimates the centers optimally (up to certain constants); our center separation requirement matches the best known result for spherical Gaussians [18]. For finite samples, we show that a bias term based on the initial estimate decreases at O(1/poly(N )) rate while variance decreases at nearly optimal rate of σ 2 d/N . Our analysis requires seeding the algorithm with a good initial estimate of the true cluster centers for which we provide an online PCA based clustering algorithm. Indeed, the asymptotic per-step time complexity of our algorithm is the optimal d · k while space complexity of our algorithm is O(dk log k). In addition to the bias and variance terms which tend to 0, the hard-thresholding based updates of streaming Lloyd's algorithm is agnostic to the data distribution and hence incurs an approximation error that cannot be avoided. However, by using a streaming version of the classical (soft-thresholding-based) EM method that exploits the Gaussian distribution explicitly, we show that for a mixture of two Gaussians the true means can be estimated consistently, with estimation error decreasing at nearly optimal rate, and tending to 0 for N → ∞.

This paper proposes a LLoyd type method with PCA initialization to estimate means of Gaussians in a streaming setting. References seem to be severely lacking as the domain is wide. The key point of the algorithm seems to be the initialization and there is no discussion on this part (comparison with the literature on k-means initialization). While the technical results might be interesting I have difficulties commenting on them without the proper background on the subject. I find as well that the technical exposition of proofs is not very clear (notations, chaining of arguments).
Other concerns
1. Line 116 : What about the estimation of variance and weights?
2. Algorithm 1 : How do you know N in a streaming setting?
3. Theorem 1 : The error bound assumes that estimated centers ans true centers are matched. The bound is a k-means type bound. Hence, the title of the paper is misleading as it should talk about k-means clustering of Gaussian mixtures.  The estimation of sigma is vaguely discussed in Section 6.
4. Section 6 looks like a very partial result with no appropriate discussion on why it is interesting.


Context Selection for Embedding Models
Word embeddings are an effective tool to analyze language. They have been recently extended to model other types of data beyond text, such as items in recommendation systems. Embedding models consider the probability of a target observation (a word or an item) conditioned on the elements in the context (other words or items). In this paper, we show that conditioning on all the elements in the context is not optimal. Instead, we model the probability of the target conditioned on a learned subset of the elements in the context. We use amortized variational inference to automatically choose this subset. Compared to standard embedding models, this method improves predictions and the quality of the embeddings.

The authors propose an extension to the Exponential Family Embeddings (EFE) model for producing low dimensional representations of graph data based on its context (EFE extends word2vec-style word embedding models to other data types such as counts or real number by using embedding-context scores to produce the natural parameters of various exponential family distributions). They note that while context-based embedding models have been extensively researched, some contexts are more relevant than others for predicting a given target and informing its embedding.
This observation has been made for word embeddings in prior work, with [1] using a learned attention mechanism to form a weighted average of predictive token contexts and [2] learning part-of-speech-specific classifiers to produce context weights. Citations to this related work should be added to the paper. There has also been prior work that learns fixed position-dependent weights for each word embedding context, but I am not able to recall the exact citation.
The authors propose a significantly different model to this prior work, however, and use a Bayesian model with latent binary masks to choose the relevant context vectors, which they infer with an amortized neural variational inference method. Several technical novelties are introduced to deal with the sparsity of relevant contexts and the difficulties with discrete latent variables and variably-sized element selection.
They show that their method gives significant improvements in held-out pseudolikelihood on several exponential family embedding tasks, as well as small improvements in unsupervised discovery of movie genres. They also demonstrate qualitatively that the learned variational posterior over relevant contexts makes sensible predictions with an examination of chosen contexts on a dataset of Safeway food products.
The idea of the paper is a straightforward, intuitively appealing, clear improvement over the standard EFE model, and the technical challenges presented by inference and their solutions will be interesting to practitioners using large-scale variational models.
One interesting technical contribution is the architecture of the amortized inference network, which must deal with variable-sized contexts to predict a posterior over each latent Bernoulli masking variable, and solves the problem of variable-sized contexts using soft binning with Gaussian kernels.
Another is the use of posterior regularization to enforce a sparsity penalty on the selected variables. Presumably, because the Bernoulli priors on the mask variables are independent, they are insufficient to enforce the sort of "spiky" sparsity that we want in context selection, and they show significant improvements to held out likelihood by varying the posterior regularization on effectively the average number of selected contexts. Perhaps the authors would like to more explicitly comment on the trade-off between posterior regularization and prior tuning to obtain the desired sparsity.
Overall, while the evaluation in the paper is fairly limited, the paper presents a clear improvement to the Exponential Family Embeddings model with a sound probabilistic model and inference techniques that will be of interest to practitioners.
1. Ling et al. Not All Contexts Are Created Equal: Better Word Representations with Variable Attention. 2015
2. Liu et al. Part-of-Speech Relevance Weights for Learning Word Embeddings. 2016


Deep Supervised Discrete Hashing
With the rapid growth of image and video data on the web, hashing has been extensively studied for image or video search in recent years. Benefiting from recent advances in deep learning, deep hashing methods have achieved promising results for image retrieval. However, there are some limitations of previous deep hashing methods (e.g., the semantic information is not fully exploited). In this paper, we develop a deep supervised discrete hashing algorithm based on the assumption that the learned binary codes should be ideal for classification. Both the pairwise label information and the classification information are used to learn the hash codes within one stream framework. We constrain the outputs of the last layer to be binary codes directly, which is rarely investigated in deep hashing algorithm. Because of the discrete nature of hash codes, an alternating minimization method is used to optimize the objective function. Experimental results have shown that our method outperforms current state-of-the-art methods on benchmark datasets.

The paper proposes a supervised hashing algorithm based on a neural network. This network aims to output the binary codes while optimizing the loss for the classification error. To jointly optimize two losses, the proposed method adopts an alternating strategy to minimize the objectives. Experiments on two datasets show good performance compared to other hashing approaches, including the ones use deep learning frameworks.
Pros:
The paper is written well and easy to follow. Generally, the idea is well-motivated, where the proposed algorithm and optimization strategy are sound and effective.
Cons:
Some references are missing as shown below, especially that [1] shares similar ideas. It is necessary to discuss [1] and carry out in-depth performance evaluation to clearly demonstrate the merits of this work.
[1] Zhang et al., Efficient Training of Very Deep Neural Networks for Supervised Hashing,
CVPR16
[2] Liu et al., Deep Supervised Hashing for Fast Image Retrieval, CVPR16
There are missing details for training the network (e.g., hyper-parameters), so it may not be able to reproduce the results accurately. It would be better if the authors will release the code if the paper is accepted.
Since the method adopts an alternating optimization approach, it would be better to show the curve of losses and demonstrate the effectiveness of the convergence.
The study in Figure 1 is interesting. However, the results and explanations (Ln 229-233) are not consistent. For example, DSDH-B is not always better than DSDH-A. In addition, the reason that DSDH-C is only slightly better than DSDH-A does not sound right (Ln 231-232). It may be the cause of different binarization strategies, and would require more experiments to validate it.
Why is DSDH-B better than DSDH in Figure 1(a) when using more bits?


EEG-GRAPH: A Factor-Graph-Based Model for Capturing Spatial, Temporal, and Observational Relationships in Electroencephalograms
This paper presents a probabilistic-graphical model that can be used to infer characteristics of instantaneous brain activity by jointly analyzing spatial and temporal dependencies observed in electroencephalograms (EEG). Specifically, we describe a factor-graph-based model with customized factor-functions defined based on domain knowledge, to infer pathologic brain activity with the goal of identifying seizure-generating brain regions in epilepsy patients. We utilize an inference technique based on the graph-cut algorithm to exactly solve graph inference in polynomial time. We validate the model by using clinically collected intracranial EEG data from 29 epilepsy patients to show that the model correctly identifies seizure-generating brain regions. Our results indicate that our model outperforms two conventional approaches used for seizure-onset localization (5-7% better AUC: 0.72, 0.67, 0.65) and that the proposed inference technique provides 3-10% gain in AUC (0.72, 0.62, 0.69) compared to sampling-based alternatives.

SUMMARY:
========
The authors propose a probabilistic model and MAP inference for localizing seizure onset zones (SOZ) using intracranial EEG data.  The proposed model captures spatial correlations across EEG channels as well as temporal correlations within a channel.  The authors claim that modeling these correlations leads to improved predictions when compared to detection methods that ignore temporal and spatial dependency.
PROS:
=====
This is a fairly solid applications paper, well-written, well-motivated, and an interesting application.
CONS:
=====
The proof of Prop. 1 is not totally clear, for example the energy in Eq. (4) includes a penalty for label disagreement across channels, which is absent in the the graph cut energy provided by the proof. The relationship between min-cut/max-flow and submodular pairwise energies is well established, and the authors should cite this literature (e.g. Kolmogorov and Zabih, PAMI 2004).  Note that the higher-order temporal consistency term can be decomposed into pairwise terms for every pair of temporal states.  It is unclear why this approach is not a valid one for showing optimality of min-cut and the authors should include an explanation.
What do the authors mean by "dynamic graph" (L:103)?  Also, the edge set $E_n$ is indexed by epoch, suggesting graph structure is adjusted over time.  It is not discussed anywhere how these edges are determined and whether they in fact change across epochs.
This estimate of SOZ probability is motivated in Eq. (5) as an MLE.  It isn't clear that (5) is a likelihood as it is not a function of the data, only the latent states.  The estimated probability of a location being an SOZ is given as an average over states across epochs, which is a valid estimator, and connections to MLE are unclear beyond that.
Generalizability claims of the approach (L:92-93) are weak.  The extent to which this is a general model is simply that the model incorporates spatio-temporal dependencies.  Specifying the factors encoding such dependencies in any factor graph will always require domain knowledge.
Some detailed comments:
  * Are there really no free parameters in the model to scale energy weights?  This seems surprising since the distance parameter in the spatial energy would need to be rescaled depending on the units used.
  * Authors claim no other work incorporates spatio-temporal EEG constraints but a recent paper suggests otherwise: Martinez-Vargas et al., "Improved localization of seizure onset zones using spatiotemporal constraints and time-varying source connectivity.", Frontiers in Neuroscience (2017).  Please cite relevant material.
  * Lines in plots of Fig. 3 are too thin and do not appear on a printed version


ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events
Then detection and identification of extreme weather events in large-scale climate simulations is an important problem for risk management, informing governmental policy decisions and advancing our basic understanding of the climate system. Recent work has shown that fully supervised convolutional neural networks (CNNs) can yield acceptable accuracy for classifying well-known types of extreme weather events when large amounts of labeled data are available. However, many different types of spatially localized climate patterns are of interest including hurricanes, extra-tropical cyclones, weather fronts, and blocking events among others. Existing labeled data for these patterns can be incomplete in various ways, such as covering only certain years or geographic areas and having false negatives. This type of climate data therefore poses a number of interesting machine learning challenges. We present a multichannel spatiotemporal CNN architecture for semi-supervised bounding box prediction and exploratory data analysis. We demonstrate that our approach is able to leverage temporal information and unlabeled data to improve the localization of extreme weather events. Further, we explore the representations learned by our model in order to better understand this important data. We present a dataset, ExtremeWeather, to encourage machine learning research in this area and to help facilitate further work in understanding and mitigating the effects of climate change. The dataset is available at extremeweatherdataset.github.io and the code is available at https://github.com/eracah/hur-detect.
Deep neural networks, especially deep convolutional neural networks, have enjoyed breakthrough success in recent recent years, achieving state-of-the-art results on many benchmark datasets (Krizhevsky et al., 2012; and also compelling results on many practical tasks such as disease diagnosis (Hosseini-Asl et al., 2016), facial recognition (Parkhi et al., 2015), autonomous driving , and many others. Furthermore, deep neural networks have also been very effective in the context of unsupervised and semi-supervised learning; some recent examples include variational autoencoders (Kingma & Welling, 2013), adversarial networks (Goodfellow et al., 2014;Makhzani et al., 2015;Salimans et al., 2016;Springenberg, 2015), ladder networks (Rasmus et al., 2015) and stacked what-where autoencoders (Zhao et al., 2015).
There is a recent trend towards video datasets aimed at better understanding spatiotemporal relations and multimodal inputs (Kay et al., 2017;Gu et al., 2017;Goyal et al., 2017). The task of finding extreme weather events in climate data is similar to the task of detecting objects and activities in video -a popular application for deep learning techniques. An important difference is that in the case of climate data, the 'video' has 16 or more 'channels' of information (such as water vapour, pressure and temperature), while conventional video only has 3 (RGB). In addition, climate simulations do not share the same statistics as natural images. As a result, unlike many popular techniques for video, we hypothesize that we cannot build off successes from the computer vision community such as using pretrained weights from CNNs (Simonyan & Zisserman, 2014;Krizhevsky et al., 2012) pretrained on ImageNet (Russakovsky et al., 2015).
Climate data thus poses a number of interesting machine learning problems: multi-class classification with unbalanced classes; partial annotation; anomaly detection; distributional shift and bias correction; spatial, temporal, and spatiotemporal relationships at widely varying scales; relationships between variables that are not fully understood; issues of data and computational efficiency; opportunities for semi-supervised and generative models; and more. Here, we address multi-class detection and localization of four extreme weather phenomena: tropical cyclones, extra-tropical cyclones, tropical depressions, and atmospheric rivers. We implement a 3D (height, width, time) convolutional encoderdecoder, with a novel single-pass bounding-box regression loss applied at the bottleneck. To our knowledge, this is the first use of a deep autoencoding architecture for bounding-box regression. This architectural choice allows us to do semi-supervised learning in a very natural way (simply training the autoencoder with reconstruction for unlabelled data), while providing relatively interpretable features at the bottleneck. This is appealing for use in the climate community, as current engineered heuristics do not perform as well as human experts for identifying extreme weather events.
Our main contributions are (1) a baseline bounding-box loss formulation; (2) our architecture, a first step away from engineered heuristics for extreme weather events, towards semi-supervised learned features; (3) the ExtremeWeather dataset, which we make available in three benchmarking splits: one small, for model exploration, one medium, and one comprising the full 27 years of climate simulation output.
2 Related work 2.1 Deep learning for climate and weather data Climate scientists do use basic machine learning techniques, for example PCA analysis for dimensionality reduction (Monahan et al., 2009), and k-means analysis for clusterings Steinhaeuser et al. (2011). However, the climate science community primarily relies on expert engineered systems and ad-hoc rules for characterizing climate and weather patterns. Of particular relevance is the TECA (Toolkit for Extreme Climate Analysis) Prabhat et al. (2012Prabhat et al. ( , 2015, an application of large scale pattern detection on climate data using heuristic methods. A more detailed explanation of how TECA works is described in section 3. Using the output of TECA analysis (centers of storms and bounding boxes around these centers) as ground truth, (Liu et al., 2016) demonstrated for the first time that convolutional architectures could be successfully applied to predict the class label for two extreme weather event types. Their work considered the binary classification task on centered, cropped patches from 2D (single-timestep) multi-channel images. Like (Liu et al., 2016) we use TECA's output (centers and bounding boxes) as ground truth, but we build on the work of Liu et al. (2016) by: 1) using uncropped images, 2) considering the temporal axis of the data 3) doing multi-class bounding box detection and 4) taking a semi-supervised approach with a hybrid predictive and reconstructive model. Some recent work has applied deep learning methods to weather forecasting. Xingjian et al. (2015) have explored a convolutional LSTM architecture (described in 2.2 for predicting future precipitation on a local scale (i.e. the size of a city) using radar echo data. In contrast, we focus on extreme event detection on planetary-scale data. Our aim is to capture patterns which are very local in time (e.g. a hurricane may be present in half a dozen sequential frames), compared to the scale of our underlying climate data, consisting of global simulations over many years. As such, 3D CNNs seemed to make more sense for our detection application, compared to LSTMs whose strength is in capturing long-term dependencies.

This paper presents a new dataset, a model and experimental results on this dataset to address the task of extreme weather events detection and localization. The dataset is 27 year weather simulation sampled 8 times per day for 16 channels only the surface atmospheric level. The proposed model is based on 3D convolutional layers with an autoencoder architecture. The technique is semi-supervised, thus training with a loss that combines reconstruction error of the autoencoder and detection and localization from the middle code layer.
In general the paper is very well written and quite clear on most details. The experimental results only use a small part of the data and are a bit preliminary, but still they do show the potential that this data has for future research.
Some comments/concerns/suggestions are the following.
- Since the paper presents a new dataset it is important to include a link to where the data will be available.
- In line 141 it says that only surface quantities are considered without any explanation why. Is it because the simulation would have been significantly more expensive so only surface computed? Is it a decision of the authors to make the dataset size more manageable? Are there any future plans to make available the full 30 atmospheric levels?
- The caption of table 1 says "Note test statistics are omitted to preserve anonymity of the test data." I assume this means that the ground truth labels are not being made available with the rest of the data. If this is the case, in the paper it should be explained what is the plan for other researchers to evaluate the techniques they develop. Will there be some web server where results are submitted and the results are returned?
- From what I understand the ground truth labels for the four weather events considered were generated fully automatically. If so, why was only half of the data labeled? I agree that exploring semi-supervised methods is important, but this can be studied by ignoring the labels of part of the data and analyze the behavior of the techniques when they have available more or less labeled samples.
- Is there any plan to extend the data with weather events that cannot be labeled by TECA?
- VGG is mentioned in line 51 without explaining what it is. Maybe add a reference for readers that are not familiar with it.
- In figure 3 the the graphs have very tiny numbers. These seem useless so maybe remove them saving a bit of space to make the images slightly larger.
- Line 172 final full stop missing.
- Be consistent on the use of t-SNE, some places you have it with a dash and other places without a dash.


SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods). We deploy this tool to measure the intrinsic dimensionality of layers, showing in some cases needless over-parameterization; to probe learning dynamics throughout training, finding that networks converge to final representations from the bottom up; to show where class-specific information in networks is formed; and to suggest new training regimes that simultaneously save computation and overfit less.

The paper presents an analysis of deep networks based on a combination of SVD and CCA to test similarity between representations at different layers. The authors use this technique to analyze aspects of the deep neural network training procedure, for example, how the representation builds bottom-up throughout training, and suggest an improved training procedure based on this analysis where lower-layers are frozen for most the training.
The paper is well-written and contributions are clearly stated.
The authors characterize the representation of a neuron by the vector of its activations for each data point, and then represent each layer as a matrix composed of the activation vectors for each neuron. Abstracting each layer as a kernel, and applying kernel SVD / CCA (with the linear kernel as a special case) would seem a more natural way to describe the same problem.
In section 2.1, it is unclear why the authors perform CCA on two networks instead of PCA on a single network, except for the sake of expressing the analysis as a CCA problem. It would be good to have PCA as a baseline.
While the bottom-up emergence of representation is an interesting effect, it is unclear whether the effect is general or is a consequence of the convnet structure of the model, in particular, the smaller number of parameters in the early layers.
In Figure 4, it seems that the representations have relatively low correlation until the very end where they jump to 1. It suggests noise in the training procedure for the top layers. It would be interesting to test whether the frozen layers training strategy lead to higher correlations also for the higher layers.


Affinity Clustering: Hierarchical Clustering at Scale
Graph clustering is a fundamental task in many data-mining and machine-learning pipelines. In particular, identifying a good hierarchical structure is at the same time a fundamental and challenging problem for several applications. The amount of data to analyze is increasing at an astonishing rate each day. Hence there is a need for new solutions to efficiently compute effective hierarchical clusterings on such huge data. The main focus of this paper is on minimum spanning tree (MST) based clusterings. In particular, we propose affinity, a novel hierarchical clustering based on Borůvka's MST algorithm. We prove certain theoretical guarantees for affinity (as well as some other classic algorithms) and show that in practice it is superior to several other state-of-the-art clustering algorithms. Furthermore, we present two MapReduce implementations for affinity. The first one works for the case where the input graph is dense and takes constant rounds. It is based on a Massively Parallel MST algorithm for dense graphs that improves upon the state-of-the-art algorithm of Lattanzi et al. [34]. Our second algorithm has no assumption on the density of the input graph and finds the affinity clustering in O(log n) rounds using Distributed Hash Tables (DHTs). We show experimentally that our algorithms are scalable for huge data sets, e.g., for graphs with trillions of edges.

The paper focuses on the development of the field of distributed hierarchical clustering. The authors propose a novel class of algorithms tagged 'affinity clustering' that operate on the basis of Boruvka's seminal work on minimal spanning trees and contrast those to linkage clustering algorithms (which are based on Kruskal's work).
The authors systematically introduce the theoretical underpinnings of affinity clustering, before proposing 'certificates' as a metric to characterise clustering algorithm solutions more generally by assessing the clustered edge weights (cost).
Following the theoretical analysis and operationalisation of MapReduce variants of affinity clustering for distributed operation, the quality is assessed empirically using standard datasets with variants of linkage- and affinity-based algorithms, as well as k-means. In addition to the Rand index (as metric for clustering accuracy) the quality of algorithms is assessed based on the ratio of the detected clusters (with balanced cluster sizes considered favourable).
Affinity-based algorithms emerge favourably for nearly all datasets, with in parts significant improvements (with the flat clustering algorithm k-means as closest contestant).
Finally, the scalability of affinity clustering is assessed using private and public corpi, with near-linear scalability for the best performing case.
Overall, the paper proposes a wide range of concepts that extend the field of hierarchical clustering (affinity-based clustering, certificates as QA metric). As far as I could retrace, the contributions are systematically developed and analysed, which warrants visibility.
One (minor) comment includes the experimental evaluation. In cases where the affinity-based algorithms did not perform as well (an example is the cluster size ratio for the Digits dataset), it would have been great to elaborate (or even hypothesise) why this is the case. This would potentially give the reader a better understanding of the applicability of the algorithm and potential limitations.
Minor typos/omissions:
- Page 8, Table 1: "Numbered for ImageGraph are approximate."
- Page 8, Subsection Scalability: Missing reference for public graphs.


Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model
With the goal of making high-resolution forecasts of regional rainfall, precipitation nowcasting has become an important and fundamental technology underlying various public services ranging from rainstorm warnings to flight safety. Recently, the Convolutional LSTM (ConvLSTM) model has been shown to outperform traditional optical flow based methods for precipitation nowcasting, suggesting that deep learning models have a huge potential for solving the problem. However, the convolutional recurrence structure in ConvLSTM-based models is location-invariant while natural motion and transformation (e.g., rotation) are location-variant in general. Furthermore, since deep-learning-based precipitation nowcasting is a newly emerging area, clear evaluation protocols have not yet been established. To address these problems, we propose both a new model and a benchmark for precipitation nowcasting. Specifically, we go beyond ConvLSTM and propose the Trajectory GRU (TrajGRU) model that can actively learn the location-variant structure for recurrent connections. Besides, we provide a benchmark that includes a real-world large-scale dataset from the Hong Kong Observatory, a new training loss, and a comprehensive evaluation protocol to facilitate future research and gauge the state of the art.

Summary
This paper describes a new GRU-based architecture for precipitation nowcasting, a task which can be seen as a video prediction problem with a fixed camera position. The authors also describe a new benchmarking package for precipitation nowcasting and evaluate their model on both this benchmark and an altered version of moving MNIST.
Technical quality
I think this is a good application paper while the proposed architecture is interesting as well. For the sake of completeness I would have liked to see some comparisons with LSTM versions of the networks as well and with fully connected RNNs but the authors already did look at a nice selection of baselines.
Clarity
The paper is generally well-written and coherent in structure. While I found it easy to follow the definition of the ConvGRU, I found the description of the proposed TrajGRU much harder to understand. The most important change in the new model is the replacement of the state-to-state convolutions/multiplications with a transformation that seems to be a specific version of the type of module described by Jaderberg et al. (2015) (reference 10 in the paper). It is not clear to me how the structure generating network gamma produces the flow fields U and V exactly though. It would help if this was made more explicit and if there would be a more clear emphasis on the differences with the transformation module from Jaderberg et al. (2015). At least the goal of the trajectory selection mechanism itself is clear enough for the rest of the narrative to make sense.
Novelty
While the general architecture is the same as the ConvLSTM and the paper is somewhat application oriented, the idea of learning trajectories with spatial transformer inspired modules is quite original to me.
Significance
The new data set seems to be a useful benchmark but I know too little about precipitation nowcasting to judge this. I also don't know enough about the state-of-the-art in that field to judge the performance gains very well and since the benchmark is a new one I suppose time will need to tell. The work would have been stronger if there was also a comparison on another existing precipitation nowcasting benchmark. I think that the idea to learn sparsely connected temporal trajectories may have more widespread applications in other domains where the distributions to be modelled are of high dimensionality but display certain independence properties and redundance.
pros:
Interesting model
Nice baselines and practical evaluation
The paper is generally well-written.
cons:
The description and motivation of the model is a bit short and hard to follow.
The novelty is somewhat limited compared to ConvGRU/LSTM
EDIT: The rebuttal addressed some of the concerns I had about the paper and I think that the additional experiments also add to the overall quality so I increased my score.


Learning Hierarchical Information Flow with Recurrent Neural Modules
We propose ThalNet, a deep learning model inspired by neocortical communication via the thalamus. Our model consists of recurrent neural modules that send features through a routing center, endowing the modules with the flexibility to share features over multiple time steps. We show that our model learns to route information hierarchically, processing input data by a chain of modules. We observe common architectures, such as feed forward neural networks and skip connections, emerging as special cases of our architecture, while novel connectivity patterns are learned for the text8 compression task. Our model outperforms standard recurrent neural networks on several sequential benchmarks.

The paper introduces a modular architecture that has a central routing module and other in/out modules, akin to the role of thalamus/neocortex regions in the brain. Initial experiments show promises.
I like the idea of the paper, which was nicely written. Some module details are missing (the read function r, merging function m and the module function f), possibly because the space limitation, but it would be very useful to have in a supplement.
The main cons are (i) the experiments are not super-strong; (ii) the search for the best module/routing functions is quite preliminary. However I believe that the direction is fruitful and worth exploring further.
In related work, there is a contemporary architecture with similar intention (although the specific targets were different, i.e., multi-X learning, where X = instance, view, task, label): "On Size Fit Many: Column Bundle for Multi-X Learning", Trang Pham, Truyen Tran, Svetha Venkatesh. arXiv preprint arXiv: 1702.07021.