The Multi-fidelity Multi-armed Bandit
We study a variant of the classical stochastic K-armed bandit where observing the outcome of each arm is expensive, but cheap approximations to this outcome are available. For example, in online advertising the performance of an ad can be approximated by displaying it for shorter time periods or to narrower audiences. We formalise this task as a multi-fidelity bandit, where, at each time step, the forecaster may choose to play an arm at any one of M fidelities. The highest fidelity (desired outcome) expends cost λ (M )  . The m th fidelity (an approximation) expends λ (m) < λ (M ) and returns a biased estimate of the highest fidelity. We develop MF-UCB, a novel upper confidence bound procedure for this setting and prove that it naturally adapts to the sequence of available approximations and costs thus attaining better regret than naive strategies which ignore the approximations. For instance, in the above online advertising example, MF-UCB would use the lower fidelities to quickly eliminate suboptimal ads and reserve the larger expensive experiments on a small set of promising candidates. We complement this result with a lower bound and show that MF-UCB is nearly optimal under certain conditions.

The paper models multi-armed bandit problems where there is an 'fidelity' component to be chosen by the learner (from among a finite set of fidelities) for the arm that is being pulled. An arm pulled at a certain fidelity yields a stochastic iid reward with mean within zeta_m of the true mean, where the zeta_m parameters (errors) are known a priori, and where the highest fidelity level corresponds to zeta = 0. In addition, smaller fidelity pulls are cheaper than higher fidelity pulls (i.e., cost less). The authors frame the problem of sequentially pulling arms with fidelities so as to maximize a notion of net reward, give a new algorithm for the problem, and argue that it can outperform UCB operating at only high fidelity levels. The paper also gives a lower bound for the regret of the proposed algorithm.
The paper in my opinion studies an interesting and relevant problem - one of modelling the tradeoff between information, cost and reward (whether to choose low information that is cheap or high information that is expensive) - in online learning , specifically stochastic bandits. In this sense it may be useful as a benchmark to improve upon.   Though the paper seems technically solid, a key shortcoming is the lack of adequate explanation about its results and assumptions. The regret definition adopted seems unnatural at least from one angle - why not penalize resource consumption (or 'cost') additively instead of multiplicatively as done here? The authors' example of ad-display motivates their definition, but may not be the most general. What happens when the cost paid for suitable fidelity observations is additively factored in total reward (or regret) is perhaps a more natural question, and would be good to know.   The introduction of Assumption 1 seems rather artificial/opaque, involving constraints on the fidelity parameters given to the learner. Why would the authors expect this to hold in a general learning problem? More importantly, if the zeta_m's don't decay as fast as required by Assumption 1, is there a general way of selecting a subsequence of fidelities to satisfy the assumption and hence the results (this would significantly help strengthen the paper's message)? Though the authors mention that the assumption is not crucial, it seems to be used in the proofs, and my main concern is whether the assumption influences the algorithm's design and performance in a critical fashion. It appears that the algorithm's design is tailored towards performing well for some kinds of zeta configurations, but the justification of why these are the 'right' zeta sequences is missing. Again, the gap in the lower bound also demands more investigation and perhaps a re-design of the proposed strategy.   Interpreting the main result (Thm. 2, the regret bound for MF-UCB) and contrasting with UCB (at highest fidelity? (this is not explicitly stated in the discussion)) is not easy to grasp due to the highly technical nature of the presentation. The relative gain/loss of MF-UCB seems to depend on Delta and [[k]] which in turn depend on the rather complicated sequence of sets \cal{K}^(m). A more explicit explanation would greatly help the authors' case.   Overall, I feel the paper gets off to a good start in terms of considering an interesting learning model, but rapidly loses clarity when it comes to presenting the results.   Minor typos:   l 56: "near-optimal" l 126: period unnecessary before "for all m < M" l 179: set notation (curly braces) would be better to define [[k]]


Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
Deep neural networks (DNNs) have demonstrated state-of-the-art results on many pattern recognition tasks, especially vision classification problems. Understanding the inner workings of such computational brains is both fascinating basic science that is interesting in its own right-similar to why we study the human brain-and will enable researchers to further improve DNNs. One path to understanding how a neural network functions internally is to study what each of its neurons has learned to detect. One such method is called activation maximization (AM), which synthesizes an input (e.g. an image) that highly activates a neuron. Here we dramatically improve the qualitative state of the art of activation maximization by harnessing a powerful, learned prior: a deep generator network (DGN). The algorithm (1) generates qualitatively state-of-the-art synthetic images that look almost real, (2) reveals the features learned by each neuron in an interpretable way, (3) generalizes well to new datasets and somewhat well to different network architectures without requiring the prior to be relearned, and (4) can be considered as a high-quality generative method (in this case, by generating novel, creative, interesting, recognizable images).

The authors adopted a pretrained GAN network as prior for visualizing preferred inputs for deep neurons.
Visual quality of preferred inputs obtained using the proposed method is significantly improved compared to existing methods. Generalizability of the method is also supported by extra experiments. Overall the paper should be interesting to the deep learning community. However, there're also many key questions left untouched. For example, is visual interpretability a sufficient and necessary condition for good network performance? How exactly can we use visual interpretability to improve a network? Also, what are the differences in neural responses induced by natural images vs. preferred inputs generated by existing/proposed methods? The main text also cites supplementary materials very frequently, which might not be very convenient for readers. The authors may consider rearrange their manuscript (e.g. moving details of methods into supplementary materials).  %% After Rebuttal %% (No score change.) As most of the reviewers (including myself) agreed that this paper is at least poster worthy for its comprehensive experiments & visually impressive results, I do however want to discuss about the general direction/motivation of this work, corresponding to the authors' response -- "we found no correlation between the activation of a neuron and the recognizability of its visualization" (also in main text):  A deep CNN is essentially a nonlinear function that maps an input image to a real value (a target neuron being studied), and activation maximization (AM) is generally considered useful since it tries to discover what images drive a target neuron the most, and those images can be interpreted as "the meaning(s)" of the neuron. However, if like the authors said, recognizability has no correlation against the functional value (activation) of the function (DCNN) being studied, how can we be convinced that better visual recognizability (this work) is a substantial improvement to existing AM methods in helping us better understand these functions? Put another way, how much more understanding about DCNNs have we gained using the proposed method, compared to using methods from e.g. Simonyan et al., whose results are described as uninterpretable (but may actually induce higher activations, since less regularization on the input space is imposed -- unfortunately one of my questions not answered by the authors)? My arguments are in no way trying to reject this paper, but hopefully to help the authors reevaluate their view on AM/preferred inputs and on toward understanding DCNNs. That is, preferred inputs (alone) does not describe/characterize a neuron very well. For example, (Machens et al.) gave a nice example of the efficient coding hypothesis (Barlow; Simoncelli) that the (most) preferred inputs of a grasshopper's auditory neuron would be something completely out of its natural environment, and what really matters should be how well the neuron's tuning curve matches the statistics of natural inputs. Similarly, preferred inputs of DCNN neurons may not be revealing very much about those neurons, even though we feel visually they are. I hope this example can help the authors further improve their paper, especially the discussion/future direction section (but up to the authors' discretion).  Machens et al., "Testing the efficiency of sensory coding with optimal stimulus ensembles," Neuron, 2005 Barlow, "Possible principles underlying the transformations of sensory messages," Sensory Communication, 1961 Simoncelli, "Vision and the statistics of the visual environment," Current Opinion in Neurobiology, 2003


Bootstrap Model Aggregation for Distributed Statistical Learning
In distributed, or privacy-preserving learning, we are often given a set of probabilistic models estimated from different local repositories, and asked to combine them into a single model that gives efficient statistical estimation. A simple method is to linearly average the parameters of the local models, which, however, tends to be degenerate or not applicable on non-convex models, or models with different parameter dimensions. One more practical strategy is to generate bootstrap samples from the local models, and then learn a joint model based on the combined bootstrap set. Unfortunately, the bootstrap procedure introduces additional noise and can significantly deteriorate the performance. In this work, we propose two variance reduction methods to correct the bootstrap noise, including a weighted M-estimator that is both statistically efficient and practically powerful. Both theoretical and empirical analysis is provided to demonstrate our methods.

This article proposes two new distributed algorithms to ameliorate the original issue with the bootstrapped KL averaging method. The article shows that a naive bootstrapped KL averaging would result in the same computational burden in combining the estimation even the data is first partitioned for local estimation. The main reason is due to the variance contained in the bootstrap sample. The paper then proposed a variance reduction technique to fix this issue. In particular, the algorithm tries to quantify the bootstrap variance by measuring the difference between the local estimator and local bootstrapped estimator, and correct the global KL estimator based on the quantification. The theory shows that by adding this correction, the convergence rate can be improved to O(N^{-1} + (dn^2)^{-1}).
This is a successive work correcting previous research on using KL averaging combining subset estimators. I think the most appealing point for using KL averaging, despite the computational issue, is its power in dealing with latent variable models. There is another line of work in using geometric median to combine subset estimators the authors might want to compare to, for example, Minsker (2013) and Hsu and Sabato (2013). These algorithms are simple and efficient in most cases, but might not be doing well for latent variable models. The variance reduction technique used in this article is very similar to the de-bias technique used in Javanmard and Montanari (2015) and Lee et a. (2015), so the theoretical contribution is kind of limited.  There are two caveats for the proposed approach. The first lies in the resampling. If the original model is complicated in sampling, then the additional resample requirement from the KL averaging could ruin the whole framework. Second, the variance reduction requires to compute the Fishier information matrix and the inverse, which will adds an O(p^3) computational burden. This will also become huge when p is large.  ========================  after reading feedback ====================  Previously I thought the advantage of this paper over the geometric median or the simple averaging approach is that it can handle latent variable models, such as estimating mixture of Gaussians. It is well known that the latent variable model has this "label switching" problem (the parameters are equivalent up to some permutation), so a direct averaging of the parameters obtained from different subsets might result in bad outcomes due to the permutation. The KL averaging method solves this problem by avoiding directly averaging the sub-estimators. Instead, it resamples data from each subset and pools the new data together to estimate the final result, so as to avoid the "label switching" issue.  However, as now I double checked the paper, I found that the variance reduction technique introduced in this paper actually ruined this nice property of the original KL averaging. If you take a look at (7), which is the main formula in this article, you will realize that (7) also suffers from the "label switching" problem if \theta's are up to some permutation. Then we arrive at some tricky scenario. On the one hand, the KL averaging can handle the latent variable model, but the efficiency is bad (according to this article) compared to the direct averaging. On the other hand, the enhanced KL averaging with variance reduction can improve the efficiency to the same level as direct averaging, but loses the ability to hand latent variable models (essentially the same as direct averaging). So I kind of doubt how much can be gained practically by using this method, which involves much heavier computation compared to averaging.  It is a pity there is only one round of author feedback and it is my fault not to pose this question to the author in the previous round. I would love to know how the authors address this problem. But given this is my problem not to ask, I will raise my score of technical quality from 2 --> 3


Generative Adversarial Imitation Learning
Consider learning a policy from example expert behavior, without interaction with the expert or access to a reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.

The paper is concerned with methods trying to imitate expert behavior via imitation learning. It presents both a theoretical investigation into connections between existing imitation learning (or more concretely, apprenticeship learning) algorithms and a new algorithm for imitation learning via a method akin to the training of generative adversarial networks. Extensive experiments are conducted on a variety of benchmarks showing the efficacy of the derived algorithm.
First off, this is a clearly written and well thought through paper that absolutely deserves to be published at NIPS.  The derivation of inverse reinforcement learning as the dual of learning an occupancy measure in the state-action space is clear, concise, and allows the authors to highlight connections between different, existing, IRL algorithms. These were interesting to discover for me as a reader and the proofs all seem sound.  The newly derived adversarial imitation learning method, which draws on a connection to generative adversarial networks (GANs) (formulating the problem of learning an occupancy measure as a classification problem), is interesting and seems less limited than the previously existing state of the art (in terms of applicability to a wide range of different tasks). I suspect that we will see several new advancements based on this new algorithm as it naturally invites for further application of existing deep learning machinery to the problem.  One argument that could be made against the paper is that it is fairly dense, packing a lot of material into 8 pages, and thus fails to be completely self-contained (i.e. the steps required to optimize the policy via TRPO are not well explained). While true, I believe this generally to be tolerable in this case as the paper is well written and explains all necessary concepts in enough detail for the reader to follow. However, I do believe that an explanation of the exact algorithm, including the TRPO steps would be useful for reproducing the paper, and aiding the interested reading in following each step. I thus would encourage the authors to add such a description to the supplementary material.  The only real issue I have with the paper is the fact that the authors seem to gloss over the problems that using an GAN like procedure can introduce. It is well known that GANs can be difficult to train and I suspect that the usage in the presented algorithm makes no exception to that rule. To that end it would be interesting to understand which (if any) countermeasures the authors took to stabilize training. Touching on the same subject, the objective maximized by the discriminator (Equation (19)) results in rewards going towards -inf for the policy (in cases where it does not currently match the expert behaviour), this could result in very large variances in the likelihood ratio estimator used for optimizing the policy parameters with TRPO. I suspect the authors thus implemented some counter measures in their actual implementation, ensuring numerical stability, that would be nice to know. Furthermore, while the evaluation is extensive, covering a large amount of different domains, the effect of random initialization of the networks seems to be neglected. To that end it would be valuable to see error bars for Figures 1-2 and a note on how stable the presented procedure is with respect to initialization and hyperparameter choices.  Apart from that I only have a few minor comments: - While the paper is well written articles are sometimes missing or wrongly placed, it would be a shame if these small mistakes make it into the paper, here are the ones I found:   Abstract:   - "without interaction with expert or access to reinforcement signal" -> "without interaction with expert or access to a reinforcement signal"   Introduction:   - "Given that learner’s true goal often is" -> "Given that the learner’s true goal often is"   - "a technique from deep learning community that has" -> "a technique from the deep learning community that has"   - "will in fact find that \psi plays a central role in our discussion, not as a nuisance in our analysis." -> "we will in fact find that \psi plays a central role in our discussion, and not only plays the role of a nuisance in our analysis."   Section 5:   - "We are now treat the causal entropy" -> "We now treat the causal entropy"   - "in contrast with linear apprenticeship" -> "in contrast to linear apprenticeship"   - "adaptation of generative adversarial training principle" -> "adaptation of the generative adversarial training principle"   - "addressed the capability of \psi_{GA} of exactly matching occupancy measures" -> "addressed the capability of \psi_{GA} to exactly match occupancy measures"   Section 5.1:   - "denote an empirical expectations over" -> "denote an empirical expectation over"   Section 6:   - "evaluate the imitation performance respect to sample" -> "evaluate the imitation performance with respect to sample" - In Section 6 (and I believe in other places as well) there is a forward reference to the supplementary (as Section 9). It would be good to explicitely write "Section 9 in the supplementary" (and perhaps name the Sections in the supplementary alphabetically) - In Section 5.1 you write that you use the ADAM learning rule but in fact never explicitely mention that you train on mini batches using stochastic gradient descent. Of course using ADAM implies this but it would be better to add it to the sentence.


Value Iteration Networks
We introduce the value iteration network (VIN): a fully differentiable neural network with a 'planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiable approximation of the value-iteration algorithm, which can be represented as a convolutional neural network, and trained end-to-end using standard backpropagation. We evaluate VIN based policies on discrete and continuous path-planning domains, and on a natural-language based search task. We show that by learning an explicit planning computation, VIN policies generalize better to new, unseen domains.

The paper presents a view of value iteration that links it to a convolutional layer in a neural network. This observation is exploited in a NN architecture that has multiple layers that effectively perform depth-limited VI, the parameters of which specify the MDP on which the planning will occur. In this way, the network learns to construct MDPs for short-term planning based on the current state of the real system. Experiments demonstrate that this approach allows the agent to generalize more effectively over many instances of navigation problems.
***After Author Response***  I was literally just complaining about reviewers that say "Add the kitchen sink" without regard for the research cost or for the impact on clarity in the paper, and here I am being that reviewer! The authors are right to call me on that. I agree that the paper has plenty going on as it is (and said as much in my original review). At the time I was erroneously thinking one could simply take the same architecture as the VIN, but train the model using prediction error rather than the policy gradient. I see now that I hadn't thought it all the way through -- the VIN model does not have the same state space as the world, so it's not clear what the error signal would even be. Rather than suggesting an additional impractical empirical comparison, I should have instead suggested that the paper back off a bit from its dismissal of model-based RL. In general, I agree that MBRL is a challenge, but it may be that the domains that are ideal for VIN are also viable for MBRL. I think that's an open question, and could be presented as such.  I have to admit that I did not follow the "constant domain observation" explanation. I might have missed something. In any case, I think runtime bears at least a little bit of discussion. If there's really no room for it, maybe it can go in the appendix.  I'm genuinely surprised that VIN helps early in learning, but that's really cool, and worth reporting.  ---Technical Quality---  The claims of the paper seemed to be technically sound and the experiments were well-designed.  I will say that I am not sure I am totally on board with the phrase "learns to plan." I think one could argue that the actual planning process is still fixed (it is the VI-like process, defined by the K convolution+max layers). What the network learns are model parameters for which the planning process yields useful results. "Learns to plan" almost implies that it has to learn about value iteration itself, which is not the case.  There was also a claim that I felt was a little under-supported. The paper waves off model-based learning because it can be hard to learn accurate dynamics. I think this is fair to say. *However*, in the particular domains the paper considers, I'm not convinced that a model would be harder to learn than the VIN. The navigation tasks all have the same flavor where the local dynamics are very simple. Presumably this is why the VIN gets any traction at all. I would imagine that a conv-net would be ideally suited to learning in these problems. I recognize that space is limited and I think there is plenty in this paper to chew on. Still, I did wish that this approach had been compared to the clear alternative approach of simply training a convolutional dynamics model and performing value iteration on that to obtain a policy. This would answer the question of whether a VIN is actually easier to learn than a dynamics model. If this approach could be demonstrated to perform robustly in domains where the more traditional model-based approach fails, that would significantly strengthen this case.  I have one other small technical quibble, and it only seems to be directly relevant to the supplementary material. When describing the grid world experiment, the text seems to suggest that the VIN statespace \bar{S} and the real statespace S are similar. The implication seems to be that S is a 2D grid (i.e. only the agent's position). Actually, of course, a state in S contains *both* the agent's position *and* the arrangement of the obstacles. \bar{S} is actually far simpler than S, which is important for the success of the VIN, I suspect.   ---Novelty---  I found the idea of the VIN to be highly novel and interesting. The idea that a planner might generate a temporary, locally useful model is not in itself new, but I don't believe I have seen an approach that learns *how* to construct such a model from experience. I also found the link between convolution and value iteration on a locally connected MDP to be a fresh view of familiar constructs.  The paper did connect to some earlier work that the authors might consider mentioning. Both pieces of work I'm thinking of use policy gradient to directly adapt model parameters rather than trying to learn the most accurate model. Sorg et al. (NIPS 2010) does this with the reward model and Joseph et al. (ICRA 2013) does it with the dynamics model. There's been follow-up at least to the Sorg work (Guo et al., IJCAI 2016). Like this work, they treat the model-based architecture as just a complicated parameterization of the policy and adapt the whole thing with model-free learning. This work is definitely distinct in several ways, but I think there is a kinship with these existing ideas.  ---Impact---  I think this paper has substantial potential for impact. I can easily see this particular algorithm being further explored/applied in other domains. I also believe that the core idea of *learning* to map complicated observations into simple models for the sake of short-horizon planning is potentially very insightful, and one that seems likely to inspire follow-up work.  I will say that I wish the paper had a little more room to discuss the weaknesses/challenges of this approach. For instance... - The paper does not discuss runtime, but I assume that the VIN module adds a *lot* of computational expense. - Though f_R and f_P can be adapted over time, the experiments performed here did incorporate a great deal of domain knowledge into their structure. A less informed f_R/f_P might require an impractical amount of data to learn. - The results are only reported after a bunch of training has occurred, but in RL we are often also interested in how the agent behaves *while* learning. I presume that early in training the model parameters are essentially garbage and the planning component of the network might actually *hurt* more than it helps. This is pure speculation, but I wonder if the CNN is able to perform reasonably well with less data. - I wonder whether more could be said about when this approach is likely to be most effective. The navigation domains all have a similar property where the *dynamics* follow relatively simple, locally comprehensible rules, and the state is only complicated due to the combinatorial number of arrangements of those local dynamics. WebNav is less clear, but then the benefit of this approach is also more modest. In what kinds of problems would this approach be inappropriate to apply?  ---Clarity---  I found the paper to be clear and highly readable. I thought it did a good job of motivating the approach and also clearly explaining the work at both a high level and a technical level.  I thought the results presented in the main text were sufficient to make the paper's case, and the additional details and results presented in the supplementary materials were a good compliment. This is a small point, but as a reader I personally don't like the supplementary appendix to be an entire long version of the paper; it makes it harder to simply flip to the information I want to look up. I would suggest simply taking the appendices from that document and putting them up on their own.  ---Summary of Review---  I think this paper presents a clever, thought-provoking idea that has the potential for practical impact. I think it would be of significant interest to a substantial portion of the NIPS audience and I recommend that it be accepted.


Learning to Communicate with Deep Multi-Agent Reinforcement Learning
We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate endto-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.
The tasks that we consider are fully cooperative, partially observable, sequential multi-agent decision making problems. All the agents share the goal of maximising the same discounted sum of rewards. While no agent can observe the underlying Markov state, each agent receives a private observation correlated with that state. In addition to taking actions that affect the environment, each agent can also communicate with its fellow agents via a discrete limited-bandwidth channel. Due to the partial observability and limited channel capacity, the agents must discover a communication protocol that enables them to coordinate their behaviour and solve the task.
We focus on settings with centralised learning but decentralised execution. In other words, communication between agents is not restricted during learning, which is performed by a centralised algorithm; however, during execution of the learned policies, the agents can communicate only via the limited-bandwidth channel. While not all real-world problems can be solved in this way, a great many can, e.g., when training a group of robots on a simulator. Centralised planning and decentralised execution is also a standard paradigm for multi-agent planning [1,2].
To address this setting, we formulate two approaches. The first, reinforced inter-agent learning (RIAL), uses deep Q-learning [3] with a recurrent network to address partial observability. In one variant of this approach, which we refer to as independent Q-learning, the agents each learn their own network parameters, treating the other agents as part of the environment. Another variant trains a single network whose parameters are shared among all agents. Execution remains decentralised, at which point they receive different observations leading to different behaviour.
The second approach, differentiable inter-agent learning (DIAL), is based on the insight that centralised learning affords more opportunities to improve learning than just parameter sharing. In particular, while RIAL is end-to-end trainable within an agent, it is not end-to-end trainable across agents, i.e., no gradients are passed between agents. The second approach allows real-valued messages to pass between agents during centralised learning, thereby treating communication actions as bottleneck connections between agents. As a result, gradients can be pushed through the communication channel, yielding a system that is end-to-end trainable even across agents. During decentralised execution, real-valued messages are discretised and mapped to the discrete set of communication actions allowed by the task. Because DIAL passes gradients from agent to agent, it is an inherently deep learning approach.
Experiments on two benchmark tasks, based on the MNIST dataset and a well known riddle, show, not only can these methods solve these tasks, they often discover elegant communication protocols along the way. To our knowledge, this is the first time that either differentiable communication or reinforcement learning with deep neural networks has succeeded in learning communication protocols in complex environments involving sequences and raw images. The results also show that deep learning, by better exploiting the opportunities of centralised learning, is a uniquely powerful tool for learning such protocols. Finally, this study advances several engineering innovations that are essential for learning communication protocols in our proposed benchmarks.
Given the topic of our paper, we focus on related work that deals with the cooperative learning of communication protocols. Out of the plethora of work on multi-agent RL with communication, e.g., [4][5][6][7], only a few fall into this category. Most assume a pre-defined communication protocol, rather than trying to learn protocols. One exception is the work of Kasai et al. [7], in which tabular Q-learning agents have to learn the content of a message to solve a predator-prey task with communication.
Another example of open-ended communication learning in a multi-agent task is given in [8]. Here evolutionary methods are used for learning the protocols which are evaluated on a similar predator-prey task. Their approach uses a fitness function that is carefully designed to accelerate learning. In general, heuristics and handcrafted rules have prevailed widely in this line of research. Moreover, typical tasks have been necessarily small so that global optimisation methods, such as evolutionary algorithms, can be applied. The use of deep representations and gradient-based optimisation as advocated in this paper is an important departure, essential for scalability and further progress. A similar rationale is provided in [9], another example of making an RL problem end-to-end differentiable.
Unlike the recent work in [10], we consider discrete communication channels. One of the key components of our methods is the signal binarisation during the decentralised execution. This is related to recent research on fitting neural networks in low-powered devices with memory and computational limitations using binary weights, e.g. [11], and previous works on discovering binary codes for documents [12].

The paper considers problems involving multiple agents in an environment with a shared goal, and attempts to learn strategies to solve such collaborative tasks. The novelty of the work is to introduce differentiable real-valued message-passing between agents, so that their communication can be trained end-to-end in order to solve the collaborative tasks.  The resulting Differential Inter-Agent Learning (DIAL) technique has been thoroughly evaluated on complex games like the switch riddle and MNIST digit recognition games to evaluate its utility and convergence to optimal policy. A good empirical study has been carried out to compare DIAL and RIAL in the presence and absence of parameter sharing and the study shows encouraging results for DIAL.
Though the paper contains a very thorough experimental evaluation of the suggested DIAL technique for multi-agent settings and the reviewer understands that it would have taken a lot of time and effort to set up and evaluate the experiments, the paper does not make a very novel contribution.  It is clear that idea of shared memory and passing message gradients between agents would speed up learning and help to find the optimal policy faster, but this might not be a very natural way to do it. For instance, humans working in teams do not have shared memories. Also, for humans, messages from other humans are a part of their observation at each time step, rather than separate signals which are treated specially as messages and optimized differently than the rest of the observation.  The idea of passing message gradients is certainly useful to have trainable message protocols while training a set of agents to perform a repetitive task, but doesn't offer much insight or useful interpretation as to how humans perform tasks in teams.  Furthermore, experimental evaluation has been done for very small instances of the games involved, e.g. only for n = 3 and 4 for the switch riddle. It is probably due to the size of the policy space involved. But this is exactly what is needed from a good reinforcement learning algorithm - combatting huge state spaces with intelligent exploration instead of random exploration. Even while going from n=3 to n=4, there is significant performance degradation involved in the sense that DIAL (with parameter sharing) fails to find the optimal policy despite the increase in number of epochs from ~5000 to ~50000. This somewhat reflects the non-scalability of the algorithm for many agents. Moreover since the paper relies highly on experimental evaluation, it might be useful to try more challenging and bigger tasks to evaluate the performance and scalability of the DIAL technique.  Apart from this, a language and grammar check can improve the presentation somewhat. Also avoid repetitions, for example in the last paragraph of section 5.2. Lastly, at some places relaxing the notations a bit can enhance readability. For instance, in the color-digit MNIST in section 6.3, the reward r(a) was quite hard to interpret. It is also not clear what a_2^a is, since it hasn't been described in that section (perhaps the authors meant u_2^a).  EDIT AFTER REBUTTAL: The authors have clarified some of my raised concerns, regarding scalability of the algorithm and its contrast to how groups of humans accomplish such tasks. I am convinced with the response and consequently I am improving my ratings slightly to reflect the same.


Deconvolving Feedback Loops in Recommender Systems
Collaborative filtering is a popular technique to infer users' preferences on new content based on the collective information of all users preferences. Recommender systems then use this information to make personalized suggestions to users. When users accept these recommendations it creates a feedback loop in the recommender system, and these loops iteratively influence the collaborative filtering algorithm's predictions over time. We investigate whether it is possible to identify items affected by these feedback loops. We state sufficient assumptions to deconvolve the feedback loops while keeping the inverse solution tractable. We furthermore develop a metric to unravel the recommender system's influence on the entire user-item rating matrix. We use this metric on synthetic and real-world datasets to (1) identify the extent to which the recommender system affects the final rating matrix, (2) rank frequently recommended items, and (3) distinguish whether a user's rated item was recommended or an intrinsic preference. Our results indicate that it is possible to recover the ratings matrix of intrinsic user preferences using a single snapshot of the ratings matrix without any temporal information.

The paper presents a methodology and evaluation for finding out which ratings in a dataset are likely to have been generated due to the influence of a recommender system. A simple model is proposed, under which users deviate from their "true" ratings over a series of steps. In each step (a) item similarity is used to make recommendations, and (b) users pick a recommendation and "adopt" it, adding it to their set of ratings, with a certain probability. The observed rating matrix is the limit of this process. The authors show that, under certain assumptions (e.g., knowledge of the adoption probability), the "true" rating matrix can be recovered from the "observed" ratings, i.e., its inference is tractable. The difference of the two is used by the authors as indication that certain items were recommended, rather than inherent to users.  This intuition is applied and evaluated on a synthetic dataset, illustrating that this methodology can  succesfuly discern "true" from "recommended" ratings. The same methodology is turned next to real datasets, where its validity is reinforced through anecdotal evidence: e.g., no recommended ratings are discovered in a dataset collected in the absence of a recommender system, while in a dataset including both movies and TV shows, tv shows were less likely to be recommended.
I appreciated the model and methodology proposed by the authors, as well as the difficulty of the task at hand. The authors do a very good job motivating their assumptions, and I have no objection with the idea that certain somewhat strong assumptions are needed to perform this.   That said, I would have like to see a discussion on how this information (discerning "true" from  "recommended" ratings) could actually be put to use. Why is this an important problem to solve? User preferences may be affected by a variety of exogenous factors; does accounting for this one lead to better recommendations? If this is the case, is a recommender, who has full view of the system, constrained by the same absence of information as assumed here?   A technical question: the relationship between true and observed ratings ends up being determined by the singular value relationship (8) (or the cleaner (18) in the supplement). In short, it seems to be "boosting" items with high sigma^true_i. Is there some intuition to be gained from this? How is the algorithm identifying recommended items affected by the gap between sigma^true and sigma^obs, and can perhaps the inference occur on the singular value space, rather than by observing how ratings change?  Section 3.2. is a bit abrupt: when stating "We develop an algorithm" it is not clear that his is an algorithm to classify items rated as "true" or  "recommended".  "validating that the Jester..." mention the fact that Jester did not use an RS earlier


Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images
In this paper, we focus on training and evaluating effective word embeddings with both text and visual information. More specifically, we introduce a large-scale dataset with 300 million sentences describing over 40 million images crawled and downloaded from publicly available Pins (i.e. an image with sentence descriptions uploaded by users) on Pinterest [2]. This dataset is more than 200 times larger than MS COCO [22], the standard large-scale image dataset with sentence descriptions. In addition, we construct an evaluation dataset to directly assess the effectiveness of word embeddings in terms of finding semantically similar or related words and phrases. The word/phrase pairs in this evaluation dataset are collected from the click data with millions of users in an image search system, thus contain rich semantic relationships. Based on these datasets, we propose and compare several Recurrent Neural Networks (RNNs) based multimodal (text and image) models. Experiments show that our model benefits from incorporating the visual information into the word embeddings, and a weight sharing strategy is crucial for learning such multimodal embeddings. The project page is: http://www.stat. ucla.edu/~junhua.mao/multimodal_embedding.html 1 .

This paper presents a new multi-million image-caption style dataset and use it for training multimodal embedding models. They also present a new user click based similarity dataset for evaluation. Finally, they try 2-3 CNN-RNN style models for multimodal phrase embedding similarity. The datasets are decent contributions (but with some issues). However, the evaluation has multiple issues in its current form and the models are also borrowed from previous work.
After author response: Thanks for answering some of my questions -- I updated some of my scores accordingly. I still encourage the authors to answer the rest of the questions, esp. eval on better downstream tasks like captioning and visual zero-shot learning.  --------------------- Evaluation issues:   -- No existing datasets have been used to evaluate, e.g., WordSim353, SimLex-999 or the new visual similarity datasets VisSim and SemSim from Lazaridou et al., 2015. The authors have not compared to any existing paper/model on these datasets to show the advantage of either their larger/better training dataset or their models, hence leaving no takeaways.  -- Why only work on word similarity? Why not show other image-language tasks in addition, e.g., captioning, visual zero-shot learning? This will better demonstrate the advantages of the training dataset.  -- For word embedding baseline, please use stronger latest models such as GloVe and Skip-thought and paragram.  Evaluation dataset issues:  -- The dataset mixes multiple types of similarity, e.g., related versus paraphrase/synonym, and this has been discussed to be a big issue for evaluating embedding models.  -- the data uses triplets instead of the standard ranking+correlation method, and then they also choose random negative phrases for these triplets (which will be easy to detect), which makes the task much easier.  -- In the CrowdFlower experiment, the turkers will assume they need to choose at least one of the two phrases most of the time, and even if they choose the correct phrase, it might be only because it is more related to the query phrase as compared to the other random negative phrase, but overall the positive phrase might still be only very slightly related to the query phrase (because of the initial recommendation system based retrieval). To verify this, the authors should get turkers to also rate how much these 'positive' phrases are related to the query phrase in absolute terms/ratings.  -- they use 3-5 annotators and then choose phrases where >50% of the annotators agree, but this will mean agreement between just two people in many cases. For such a noisy initial dataset, the filtering should be more strict than 50%, or the #annotators should be higher.  Training dataset:  -- Since the data is from Pinterest and the 'captions' are just user comments, I am worried that some of these captions might not be standard description style sentences but instead might contain some non-image/visual story or information -- the authors should investigate and present this, and also verify if this corpus can be used for training captioning systems.  Model:  -- main model figure is very unclear and should be expanded and maybe separated.  -- all models are from previous work; would have been good to also suggest some new model variants for this task/datasets.  -- why use average pooling at test time for a phrase representation and not run the trained RNN model?  Other:   -- Lots of typo's throughout the paper, e.g.,  Line15: "crutial" --> "crucial" Line34: "commemted" --> "commented" Table 3: "weigh" --> "weight" Line 232: "to Model A, We" --> "to Model A, we"


Completely random measures for modelling block-structured sparse networks
Statistical methods for network data often parameterize the edge-probability by attributing latent traits such as block structure to the vertices and assume exchangeability in the sense of the Aldous-Hoover representation theorem. These assumptions are however incompatible with traits found in real-world networks such as a power-law degree-distribution. Recently, Caron & Fox (2014) proposed the use of a different notion of exchangeability after Kallenberg (2005) and obtained a network model which permits edge-inhomogeneity, such as a power-law degree-distribution whilst retaining desirable statistical properties. However, this model does not capture latent vertex traits such as block-structure. In this work we re-introduce the use of block-structure for network models obeying Kallenberg's notion of exchangeability and thereby obtain a collapsed model which both admits the inference of block-structure and edge inhomogeneity. We derive a simple expression for the likelihood and an efficient sampling method. The obtained model is not significantly more difficult to implement than existing approaches to block-modelling and performs well on real network datasets.

Past research had introduced the concept of exchangeability in network models, whereby two isomorphic graphs have the same distribution. In particular, Caron & Fox (2014) incorporated power-law degree inhomogeneity to exchangeability of graph models. The present paper proposes a technique to model exchangeability and edge-inhomogeneity (power-law degree distribution) but, in this case, the model incorporates also block-structure in networks.
      While the idea of modeling exchangeability for block-structure is not new in itself (see for instance Nowicki and Snijders 2001 for statistical perspective of the same problem), the perspective the authors follow seems novel because they extend Kallenberg exchangeability to model block-structure. While this approach can help elucidate exchangeability in network models I have a few concerns about the empirical evidence shown in the paper and the presentation of sections 2.2-2.5.  Some drawbacks:  Technical:       The experiments state that the choice of parameters for the competing models is "similar" to the choice for the proposed model without providing the experimental setup for the alternative models. However, this should be explicitly explained for replicability purposes.         The results themselves seem not very convincing. For instance, in figure 4b the performance of the proposed model DCSBM seem not very much better than that of competing alternatives, except for the first 3 datasets. However, and more importantly, not much can be said of the advantage of DCSBM because the results are for a single run (with thousands of iterations of the algorithm). Hence, without an statistical assessment of the results it is hard to tell what is the advantage of the proposed model with respect to the other alternatives (for instance Nowicki and Snijders (2001)).         Lastly, the conclusion could be improved by explicitly stating how much reduction of the number of parameters (line 284) is attained with the proposed model.        Presentation:        In general Sections 2.2-2.5 specialy 2.4 and 2.5 are hard to follow. I would kindly recommend to review these sections. The reason for this is that some notational information is not explicitly explained, or has been omitted. For instance:              All symbols used in the paper must be explicitly defined including $\delta$ (Dirac measure?). This also applies to operations, e.g. $o$ or $\otimes$ in line 113 or $dw$ and $d\theta$.              Please, kindly explicitly state what basic properties of completely random measures (CRM) make the thinned measures in line 172 also CRM (Don't leave this to the reader).              The authors do not define what the equality in line 40 means (equality in distribution?, I assume).              The paper leaves many details (e.g paragraph 190-195) to outside sources which makes it not fully self-contained.              Some minor typos: The sentence 32-33 "will to refer to". Many openning/closing brackets are of the type [a,b [ (this may be a type unless the authors meant open intervals, which is not stated in the paper): e.g. line 126, line 180. Possible should replace possibly in line 110, and vertex should replace vertice in line 93


An Efficient Streaming Algorithm for the Submodular Cover Problem
We initiate the study of the classical Submodular Cover (SC) problem in the data streaming model which we refer to as the Streaming Submodular Cover (SSC). We show that any single pass streaming algorithm using sublinear memory in the size of the stream will fail to provide any non-trivial approximation guarantees for SSC. Hence, we consider a relaxed version of SSC, where we only seek to find a partial cover. We design the first Efficient bicriteria Submodular Cover Streaming (ESCStreaming) algorithm for this problem, and provide theoretical guarantees for its performance supported by numerical evidence. Our algorithm finds solutions that are competitive with the near-optimal offline greedy algorithm despite requiring only a single pass over the data stream. In our numerical experiments, we evaluate the performance of ESC-Streaming on active set selection and large-scale graph cover problems.

This paper studies the streaming version of set cover.  In the model proposed here, the input consists of sets that cover a universe that arrive online in an arbitrary order, and an algorithm must find an approximately optimal solution using a single pass on the input while using sublinear memory.  The main result shows that if one is willing to compromise for a 1-\epsilon coverage of the universe, there is an algorithm that achieves (log n/\epsilon) approximation of the offline optimal solution.  Thus, the algorithm is 2/\epsilon factor of optimal, as the offline set cover problem cannot be approximated within a factor strictly better than log n.  In addition, the authors justify the relaxed version of set cover with a nice lower bound.  They show that obtaining a non-trivial approximation guarantee for the online set cover problem (i.e. without the 1-\epsilon relaxation) requires linear memory.
The problem studied is the min set cover where there is a family of sets that cover some universe and the goal is to find a family of sets of minimal size that covers at least Q elements in the universe.  It is well known that this problem is NP hard and that no polynomial time algorithm can obtain an approximation guarantee better than log n, under reasonable computational complexity assumptions.    In the streaming setting, defined in this paper for this problem for the first time, the goal is to solve the set cover problem using a single pass on the data using sublinear memory when the assumption is that sets arrive in an adversarial order.    Motivated by impossibility results for this problem, shown in this paper, the authors define the Streaming Bicriteria Submodular Cover (SBSC) problem where the goal is to obtain (1-\epsilon,delta) bi-criteria approximations, i.e. cover a (1-\epsilon) fraction of the universe using \delta factor of the minimum sets of the optimal solution.  The main result is a streaming algorithm in the model defined which covers a (1-\epsilon) fraction of universe using 2/\epsilon sets of the optimal solution.  The idea of the analysis is a refinement of the classic analysis of the greedy algorithm for set cover by Wolsey.  In particular, as the authors point out, if the greedy algorithm is stopped after covering 1-\epsilon of the universe, this implies that at most k*ln(1/\epsilon) sets are picked which gives a tight (1-\epsilon,ln(1/\epsilon)) bi criteria  offlne apx algorithm.    The authors essentially take this idea and turn it into a streaming algorithm by maintaining log n different solutions, each with a different threshold (in multiples of 2).  An element is then added to all the solutions in which its coverage passed the threshold of the solution.  At a high level, the idea of the algorithm is very similar to previous streaming algorithms for the dual max-k-cover problem.  The analysis however, introduces new ideas as discussed above.     In terms of applications, the authors give two nice applications of active set selection and graph set cover.  Both are problems studied in the past where it is quite reasonable to use a streaming algorithm.  The experiments are quite comprehensive, and are preformed on very large data sets (graphs with nearly a billion nodes and ~50 billon edges).  The algorithm was run against strong benchmarks, and seems to be competitive with the offline greedy algorithm which is optimal for this problem.  Overall, this is a nice paper. The paper introduces a new problem which is quite natural and does a great job in modeling given that the strict version of the problem does not yield non-trivial solutions.  The algorithms are similar to previous ones for related problems, but the ideas guiding the analysis are all new and interesting.  The applications are interesting and the experiments are solid.  In addition, the paper is well written and easy to follow.       Minor comments:  - In line 47, shouldn't the main result be 2/\epsilon (rather than 1/\epsilon) of the minimal number of sets?


Bayesian Optimization with Robust Bayesian Neural Networks
Bayesian optimization is a prominent method for optimizing expensive-to-evaluate black-box functions that is widely applied to tuning the hyperparameters of machine learning algorithms. Despite its successes, the prototypical Bayesian optimization approach -using Gaussian process models -does not scale well to either many hyperparameters or many function evaluations. Attacking this lack of scalability and flexibility is thus one of the key challenges of the field. We present a general approach for using flexible parametric models (neural networks) for Bayesian optimization, staying as close to a truly Bayesian treatment as possible. We obtain scalability through stochastic gradient Hamiltonian Monte Carlo, whose robustness we improve via a scale adaptation. Experiments including multi-task Bayesian optimization with 21 tasks, parallel optimization of deep neural networks and deep reinforcement learning show the power and flexibility of this approach.

This paper develops a novel approach to Bayesian optimization using Bayesian deep neural networks.  In particular, the authors combine recent advances in approximating Bayesian neural networks (stochastic gradient MCMC and Hamiltonian Monte Carlo) to be effective at Bayesian optimization.  Bayesian optimization is traditionally performed using Gaussian processes, but this does not scale well and thus becomes computationally intractable for many observations (or few observations across many different tasks).  The authors show that their approach usually outperforms the one existing formulation for Bayesian optimization with neural networks on a variety of optimization tasks.
It is very difficult to get most of the existing approaches to approximate Bayesian neural networks to work robustly without a significant amount of problem specific tuning (which makes them a poor choice for a task such as Bayesian optimization).  I really appreciate the authors' commitment to making the methodology generalize across different problems.  In general, I think this is an excellent paper.  One glaring omission in this paper is any evaluation of the wall-clock runtime of the approach.  How long does it really take in practice to perform 2000 burn-in steps and then 50*50 = 2500 steps for thinning and sampling?  I'm curious how many data points one could reasonably accommodate while performing a parallel Bayesian optimization before the optimization procedure becomes the bottleneck.  Having a latent vector representation for each experiment in the multitask setting seems quite sensible.  However, I don't see much detail about how these latent vectors are learned.  Are they learned in the same way as the model weights? What are the priors on these vectors?  The choice of prior could certainly have a non-trivial impact on the performance in Bayesian optimization.  e.g. setting the the prior to zero-mean Gaussian encodes a notion that all tasks are similar (i.e. all the zero-vector) in the absence of observations to the contrary.  That seems like a sensible prior for this, but it would be nice to have a discussion about this.  I disagree that a one-hot encoding would be less efficient; in fact I think it would be exactly the same.  With a one hot coding you are essentially just treating the weights attached to each input as the latent vector which the one hot code just indexes into.  This can be done as efficiently as this latent vector approach.  The supplementaries contain some really interesting additional information.  I appreciated the plots showing the sinc fit of various other approximations for Bayesian neural nets.  I have done very similar experiments myself and definitely agree that the uncertainty estimates from these is not of high enough quality to use in Bayesian optimization.  However, in the fit of the sinc function shown in Figure 1, you can see that the posterior mean of the function dips down as it gets farther from the data on the left side but up on the right side.  This seems like a result of the highly non-stationary model and an undesirable quality for Bayesian optimization.  For example, the optimization routine (assuming we are minimizing) would arbitrarily explore on the left side instead of the right side.  This doesn't seem to be as much of an issue for the other approaches, as their means seem to asymptote towards the same value on either side.  It would be nice to have an understanding of why this happens and why it might not be a problem.  In the supplementaries it looks like DNGO outperforms SGHMCBO on at least one of the benchmarks (Hartmann 3).  I'm curious why it seems to perform better on this problem.  Does SGHMCBO perhaps do better as the dimensionality of the problem increases (and thus there are more model parameters)?  Nit picky comments: Line 49: It seems weird to indicate the number of experiments using #i since i is a scalar.  Why not follow convention and just use n, i.e. i=1...n


Learning feed-forward one-shot learners
One-shot learning is usually tackled by using generative models or discriminative embeddings. Discriminative methods based on deep learning, which are very effective in other learning scenarios, are ill-suited for one-shot learning as they need large amounts of training data. In this paper, we propose a method to learn the parameters of a deep model in one shot. We construct the learner as a second deep network, called a learnet, which predicts the parameters of a pupil network from a single exemplar. In this manner we obtain an efficient feed-forward one-shot learner, trained end-to-end by minimizing a one-shot classification objective in a learning to learn formulation. In order to make the construction feasible, we propose a number of factorizations of the parameters of the pupil network. We demonstrate encouraging results by learning characters from single exemplars in Omniglot, and by tracking visual objects from a single initial exemplar in the Visual Object Tracking benchmark.

This paper focuses on the problem of one-shot learning, where one wants to build a new classifier given just a single training image. The obvious complication in this problem is that with just one image it is very hard to predict whether the resulting network can generalize well in new categories, as the danger of overfitting is very high.  To deal with this problem the authors propose to learn a second "meta"-network, whose outputs will be the parameters of the classification network. As the potential number of parameters to be returned is very high, especially given the number of available training images, the authors propose to factorize the matrix-vector multiplications of a fully connected in a form (M * diag(W) * M') and learn only a diagonal matrix W, while having M and M' pre-trained. Similarly, for the convolutional layers a filter basis is learnt and at test time it is decided which filter should be applied on which channel.  Three different architectures are presented: a) a siamese architecture casting the one-shot classification as an image comparison problem, b) a siamese learnet, which predicts the shared parameters for some layers in a siamese network that compares images, and c) a learnet, which directly returns the parameters for the (not siamese) classification network. The method is evaluated on character recognition and on tracking.
The paper overall looks interesting with some nice ideas. Below I make a list of positive and negative elements and some remarks.  Positives --------------- + The proposed method and the factorization makes sense and is interesting. Could one explain the matrices M and M' as learning some latent, topic models and then using w(z) to select the right proportions of these topic models for the novel category? Alternatively, could there be a connection with sparce coding, where at test time one needs to recover the right set of basis vectors?  + The results support the method. In two benchmarks the obtained accuracies improve over reasonable (but not neceessarily the latest state-of-the-art) baselines.  + The writing is mostly clear, except for section 2.3 (see below).   Negatives & Remarks ----------------------  - The paper seems to overclaim and fails to cite some relevant papers. First and most importantly, there are some previous works, which also learn parameters dynamically, such as "A Dynamic Convolutional Layer for Short Range Weather Prediction" presented in CVPR 2015 and "DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction" in arXiv, 2015. Although the first paper is not for one-shot learning, for image question answering the model also needs to learn in one-shot. Also, regarding object tracking the proposed method seems very close to "Siamese Instance Search for Tracking", CVPR 2016. The paper needs to make more precise statements and incorporate missing references.  - Given that one wants discriminative classifiers, wouldn't it make more sense to consider triplet siamese networks that consider positive and negative pairs at the same time? How would such a network work compare to the standalone learnet?  - Some clearer explanation would be welcome is in Section 2.3, where the factorized convolutional layers are explained. What is it exactly meant by saying that "M and M' disentangle the feature channels, allowing w(z) to choose which filter to apply to each channel"? Is there an over-redundant number of channels, which is then reshuffled and the right set of channels are selected to be convolved with the right filters? Also, it is not clear why the factorization of the convolutional layers should be convolutional by itself? Namely, the convolution operates as normal between filters F and the input x. Why does F need to also rely on some convolutional operation as well? Could one directly predict the parameters of the convolutional filters?  - How is the weighted l1 distance in Table 1 computed?


Unifying Count-Based Exploration and Intrinsic Motivation
We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across states. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into exploration bonuses and obtain significantly improved exploration in a number of hard games, including the infamously difficult MONTEZUMA'S REVENGE.

This paper has multiple contributions: 1. an extension to counting-based exploration based on sequential density model; 2. a simple but effective modification to DQN; 3. impressive empirical results incl. progress towards solving Montezuma's Revenge; 4. a connection to intrinsic motivation (more comments on this topic in detailed review below)
Overall this paper is a good paper that should be accepted. In particular, the formulation of pseudo-count seems novel, and Montezuma's Revenge experiment demonstrated convincingly the method can be effectively applied.  I have however 2 concerns that I'd like the authors to address/respond to:  1. As stressed in title and in line 206, the claimed main result is the relationship between pseudo-count and information gain (IG), which is a classic concept in intrinsic motivation literature. The stated relationship, though technically sound, doesn't seem to relate to existing intrinsic motivation literature. I'd be happy to be corrected but as far as I can tell, the term "Information gain" here is used to define a quantity that's both technically and philosophically different from the usual Information gain.  Specifically IG in this paper is defined over information gain over a mixture sequential density models of state visitations whereas IG is usually defined as information gain of "model of the environment" [1,2]. Philosophically, IG is introduced to create a "knowledge-seeking agent ... to gain as much information about its environment as possible" [2] and this is a characteristic that the "IG" definition in this paper fundamentally lacks. The reason is that if the behavior policy changes, there will always be information gain of state density model because state visitation frequencies change as the policy changes yet this specific "IG" doesn't need to reveal anything new about the environment, hence making the "IG" used in this paper distinctly different from the classic notion.  The connection between "IG", PG, and pseudo-count is an interesting one but I'm wondering if the authors can justify how it "connects to intrinsic motivation", and "unifies count-based exploration and intrinsic motivation"?  2. Can the authors discuss more why pseudo-count is preferable to PG? From figure 2 in appendix it seems PG performs very competitively without extra tweaking on selecting what function to use to transform it but the inverse pseudo-count performs poorly with the /sqrt transformation. I suspect that if similar PG can be tried with different transformations it's going to outperform the tweaked pseudo-count bonus. Proof of Theorem 2 also indicates that 1/pseudo-count >= e^PG - 1. Since 1/psudo-count is exponentially larger than PG, it'd not be surprising that PG is better-behaved than 1/pseudo-count. I'm wondering if the authors can clarify if pseudo-count has distinctive advantages over PG or they are more of less interchangeable?   [1]: Sun, Yi, Faustino Gomez, and Jürgen Schmidhuber. "Planning to be surprised: Optimal bayesian exploration in dynamic environments." International Conference on Artificial General Intelligence. Springer Berlin Heidelberg, 2011. [2]: Orseau, Laurent, Tor Lattimore, and Marcus Hutter. "Universal knowledge-seeking agents for stochastic environments." International Conference on Algorithmic Learning Theory. Springer Berlin Heidelberg, 2013.


Large Margin Discriminant Dimensionality Reduction in Prediction Space
In this paper we establish a duality between boosting and SVM, and use this to derive a novel discriminant dimensionality reduction algorithm. In particular, using the multiclass formulation of boosting and SVM we note that both use a combination of mapping and linear classification to maximize the multiclass margin. In SVM this is implemented using a pre-defined mapping (induced by the kernel) and optimizing the linear classifiers. In boosting the linear classifiers are pre-defined and the mapping (predictor) is learned through a combination of weak learners. We argue that the intermediate mapping, i.e. boosting predictor, is preserving the discriminant aspects of the data and that by controlling the dimension of this mapping it is possible to obtain discriminant low dimensional representations for the data. We use the aforementioned duality and propose a new method, Large Margin Discriminant Dimensionality Reduction (LADDER) that jointly learns the mapping and the linear classifiers in an efficient manner. This leads to a data-driven mapping which can embed data into any number of dimensions. Experimental results show that this embedding can significantly improve performance on tasks such as hashing and image/scene classification.

This paper deals with two major themes. First it describes a duality between multi-class SVMs and multiclass adaboost. Secondly it uses this duality to derive a novel dimensionality reduction method that preserves discriminability between different classes.
The authors modify the MCBoost criterion, in order to allow for multi-class boosting that is based on arbitrary number of dimensions (compared to a previous formulation that limits the number of dimensions to the number of classes). This lift of the limits in terms of dimensionality allows for a boosting-like framework that is comprised of controllable amount of boosting functions, and thus can be used as.  + The connection between MC-Boost and MV-SVM is interesting, and the discussion is good. The discussion of the differences in terms of the predictor and the codewords (e.g. table 1) is also quite readable. Is the fact that both MC-SVM and MC-Boost try to maximise the margin well known?  + The authors present improved results in terms of error rate, and in terms of mAP. More specifically, LADDER outperforms MCBoost by 2% in terms of classification accuracy. In addition, LADDER outperforms MCBoost for across all number of dimensions in terms of error rate (Figure 3).  - More details could be given in terms of dimensionality reduction and CNN. This is a promising line of work, and has been shown to outperform all the other techniques even for cases where the size training data is not huge as the authors suggest. (e.g. DrLim Hadsell et al 2006). This holds for the literature review.  - Results could be compared with the new results based on CNNs. The representations learned on CNNs can be quite powerful, since both the features and the discriminant projection is learned. In addition, altough CNN's are highly non-convex, in practice they give better results than convex methods. The only comparison with CNNs is given in terms of using the features extracted from conv5 (Table 2) in order to compare with the BoW features. However, the real power of deep learning comes from simultaneous learning of both the features and the discriminative projection functions.  - Another convex method presented by Simonyan,Vedaldi and Zisserman (PAMI 2014) is not compared. In fact the authors only compare with lower accuracy methods such as PCA or LDA and their kernel equivalents.  - In general the experimental evaluation is not convincing, and there are many more experiments needed to show the merits of the proposed method. Convergence characteristics and more sub-analyses of the proposed method are needed.


Proximal Deep Structured Models
Many problems in real-world applications involve predicting continuous-valued random variables that are statistically related. In this paper, we propose a powerful deep structured model that is able to learn complex non-linear functions which encode the dependencies between continuous output variables. We show that inference in our model using proximal methods can be efficiently solved as a feedfoward pass of a special type of deep recurrent neural network. We demonstrate the effectiveness of our approach in the tasks of image denoising, depth refinement and optical flow estimation.

The paper presents a way to train a certain class of energy-based models commonly used in computer vision tasks with potential functions based on deep learning outputs.  The proposed method is a backpropagation-through-inference (BPTI) approach, where an iterative algorithm for optimizing the energy is unrolled for a finite number of steps, and each step is treated as a regular computation layer in a deep neural net.  Although BPTI has been applied in many previous works, the present work is novel principally in that it applies to a general class of energy-based models over continuous-valued states, whereas previous works either applied only to discrete problems, or more restricted classes of continuous problems.  Experiments show state-of-the art results on several applications.
The work addresses a problem of significant current interest, since it addresses the issue of how to combine state-of-the-art variational methods for computer vision tasks with deep learning methods, which have rapidly become the de-facto choice for nearly all applications.  The proposed solution is practical, elegant, computationally efficient, and applicable to a wide variety of problems.  The experimental results are also state-of-the-art for several tasks.  Therefore, I believe that this work could have a significant practical impact on the field.  My only, relatively minor, reservation with the work is its novelty.  If you are a specialist in state-of-the-art convex optimization methods for computer vision tasks, and you know about structured prediction models for deep learning, then you could argue that this is a fairly obvious thing to do, since it is basically backpropagation-through-inference (BPTI) applied to primal-dual methods for convex variational problems.  On the other hand, this would be novel to anyone who is not a specialist in both those fields.  So, one could argue that the novelty of this method is mainly of the type of cross-pollination between two fields that should have more interaction, but there are no fundamentally new methods here.  Delving a little deeper into this issue, one might consider what the potential alternatives to this method may be, and how obvious each of those alternatives may be.  Prior work has mostly approached similar problems from a graphical models perspective, in which applying BPTI to belief propagation is the obvious solution.  Relative to that, the main innovation of this work is in defining a BP-like inference algorithm that works for an important class of graphical models over continuous variables.  Although graphical models people tend to shy away from continuous variables, BP (in this case, the max-product variant) still works conceptually in the continuous case.  The main difficulty is that the messages become functions, which are generally hard to represent.  However, if one assumes that the local minimizations have stable closed-form solutions, then max-product BP may still be applicable.  It would be interesting to see whether this is feasible, and if so, what relationship this method would have to the proposed method.  Another potential baseline would be block coordinate descent: i.e., instead of using the primal-dual method, alternatively fix blocks of variables and optimize them jointly.  For example, given “4-connected” cliques with variables on a regular grid, one could perform a method analogous to red-black Gauss-Seidel iteration, alternatively fixing ‘red’ and ‘black’ pixels.  This would reduce to independent, pointwise optimizations over the grid.  If one again assumes that these pointwise optimizations have closed-form solutions, then BPTI should be applicable to this method as well.  Experimentally, the work seems to be strong, showing state-of-the-art results compared to other methods, although I am unfortunately not that familiar with these benchmarks and many of the other methods.  One small caveat is that the image denoising dataset seems to be fairly small, and the Gaussian noise assumption a bit too simple, although it has apparently been used as a benchmark for other methods as well.  It would also be nice to have an idea of the computational performance for the depth refinement and optical flow tasks, although I assume they would be similar to the denoising case.  A summary of my review along each of the evaluation metrics is provided below.  Technical quality: The method is sound, and the experimental results are thorough and state-of-the-art.  Novelty: Although no fundamentally new methods are developed here, this work constitutes a clever way to combine state-of-the-art techniques from different fields (high-dimensional convex optimization and deep learning).  Potential impact: Due to the relative simplicity of the method, its computational efficiency, and state-of-the-art results, I think the method could (or at least should) be adopted widely for practical applications.  Clarity: The paper is written in a straightforward way and is easy to read.  ============ REVISION POST-REBUTTAL:  After the review period, I happened to find some uncited work (see [A] below) that, at least superficially, seems similar.  That work also involves doing backprop through a proximal method. I also mentioned in my review the possibility of doing backprop through coordinate descent, and it seems that this idea was also tried in [A].   So, I am now a little more worried about the technical novelty of this method.  A future revision of the paper should definitely cite [A] and discuss the differences between this work and that.  [A] K. Gregor and Y. Lecun. Learning fast approximations of sparse coding. ICML 2010.


Scaling Factorial Hidden Markov Models: Stochastic Variational Inference without Messages
Factorial Hidden Markov Models (FHMMs) are powerful models for sequential data but they do not scale well with long sequences. We propose a scalable inference and learning algorithm for FHMMs that draws on ideas from the stochastic variational inference, neural network and copula literatures. Unlike existing approaches, the proposed algorithm requires no message passing procedure among latent variables and can be distributed to a network of computers to speed up learning. Our experiments corroborate that the proposed algorithm does not introduce further approximation bias compared to the proven structured mean-field algorithm, and achieves better performance with long sequences and large FHMMs.

This paper develops more scalable variational methods for factorial hidden Markov models by combining ideas from structured mean-field optimization, recognition networks and stochastic gradient descent.   In the proposed approximating posterior based on a Gaussian bivariate copula, each timestep t within chain m has two variational parameters: marginal probability \theta_tm and correlation \rho_tm. These two parameters, together with parameters of neighboring timesteps, can represent any possible pairwise distribution for neighboring binary states. For a single binary chain of length T, there are in total 2T required parameters. The choice of the marginal and correlation values, as opposed to other representations, is done to make each value predictable from a separate recognition network. The proposed recognition network approach reduces the total number of parameters per chain from 2T to 2H, where H is the total number of parameters in a feed-forward network that predicts the scalar correlation or marginal probabilty value for timestep t given a fixed-width window of data around timestep t. The length of the fixed-width window and the internal structure of the recognition network are in general application dependent and likely must be chosen carefully.  The whole approach results in a variational objective function parameterized by global parameters like emission and transition probabilities, and the recognition network weight vectors which control the local state sequence posterior. The number of parameters per sequence is much less than the structured mean field approach, leading to gains in training and evaluation speed. This function can be optimized via stochastic gradient descent, where we repeatedly sample small windows of small, fixed length from the long sequence and take gradient steps. Automatic differentiation packages are used here.  Experiments compare the proposed approach against a full-batch SMF baseline on several datasets including Bach chorales and household power consumption.
Overall I think this work is interesting and the proposed union of recognition networks and variational inference is of growing interest in the community. This could make a good poster presentation at NIPS, but I could also understand arguments for borderline rejection. My primary concerns are the method requires lots of tuning to find the right recognition network structure with little guidance on this front, and that the experiments are a bit weak in that they only compare the proposed method to a very basic method (full-batch structured mean field) without evaluating recent alternatives like Foti et al [4] which might be more scalable by using small windows of the long sequence without the added complexity of recognition networks.  Technical feedback ================== Overall the proposed method requires several important decisions on the part of the end user: recognition network structure, fixed-window length, how to handle the edge case timesteps at the beginning and end of the sequence where the network's window does not fit, etc. These are casually mentioned, but no experiment really provides guidance about how to deal with these in a principled way, or even compares tradeoffs in runtime vs accuracy under different possibilities. I think this lack of careful consideration will make this method difficult to apply in practice, and I'd like to see more discussion in a revision.   The question of local optima is definitely relevant here, and little guidance is provided on this front. Without showing results from many independent initializations, it is difficult to know whether the proposed approach might be more or less vulnerable to local optima than the standard SMF approach.  Is code available? Perhaps sharing code would mitigate some concerns, especially when for dealing with the edge cases.  I'd like to see some scalability experiments that directly compare two methods for a sequence of length T and a trained model with a single binary chain: (1) using a recognition network of width W to predict the entire state sequence, including edge cases, and (2) using the standard forward/backward algorithm to estimate the state sequence. I think it would be useful to directly quantify how much easier this kind of test-time inference gets under the proposed approach... is it 2x? 10x better? Hard to say right now.  Experiment feedback =================== Looking at Table 1 for the Bach dataset, the baseline structured mean field (SMF) approach produces log likelihood on training (LLtrain) values that continue to improve with more hidden chains all the way through 8 chains, as we would expect because the model capacity is improving and can fit the data better. However, for the proposed algorithm, the LLtrain with 7 chains (-5.75) is noticeably better than for 8 chains (-5.836), which I would not expect. I would guess this is a local optima issue. I'd like to hear more about this in rebuttal. Should we be concerned that for large model capacity the performance of the proposed approach might suffer? This doesn't really reassure me that the performance is always indistinguishable from SMF.  Overall, I'd much rather see Table 1 turned into a line plot of performance vs # chains. Tables are not ideal for helping your audience understand relative performance trends.  When evaluating scalability in Sec. 5, the comparison between the proposed approach (which can stochastically use subsequences *and* take advantage of the cheap recognition network) and the SMF-EM algorithm, which must process a whole sequence before each global parameter update, might not be the fairest possible comparison. I'd be curious how the Foti et al method [4], which can process small subsequences from the long sequence but does *not* use the recognition network, stacks up in this comparison. This may be too much to ask, but otherwise this experiment isn't especially convincing that each piece in the proposed pipeline lead to scalability wins. It's hard to know whether the effort for the recognition network is really "worth it", or whether just doing stochastic updates after a few short windows would give similar performance.  Household power dataset questions: how long (wallclock time) did training take here? How many SGD iterations were done? How much data does one iteration use?  Instead of just showing the predictions of the method on the power dataset, I'd suggest a within-method comparison, where you choose some useful tweak to your model (5 vs 10 chains, or rolling window size of 5 vs 10, etc) and report how the resulting performance vs speed tradeoffs.  Typos, etc ========== line 160 typo: "ramdom variables in different chains"  Sec 5.2 suggestion: Say "this experiment consists of two parts" instead of "two components".


Improved Deep Metric Learning with Multi-class N-pair Loss Objective
Deep metric learning has gained much popularity in recent years, following the success of deep learning. However, existing frameworks of deep metric learning based on contrastive loss and triplet loss often suffer from slow convergence, partially because they employ only one negative example while not interacting with the other negative classes in each update. In this paper, we propose to address this problem with a new metric learning objective called multi-class N -pair loss. The proposed objective function firstly generalizes triplet loss by allowing joint comparison among more than one negative examples -more specifically, N -1 negative examples -and secondly reduces the computational burden of evaluating deep embedding vectors via an efficient batch construction strategy using only N pairs of examples, instead of (N +1)×N . We demonstrate the superiority of our proposed loss to the triplet loss as well as other competing loss functions for a variety of tasks on several visual recognition benchmark, including fine-grained object recognition and verification, image clustering and retrieval, and face verification and identification.

Update: My review remains the same after rebuttal. The rebuttal did help to clear some questions. I think the paper can be accepted, provided the authors include the comments and points discussed in the revised manuscript.   Summary of the paper: The paper proposed a new multi-class pair/triplet loss for metric learning with CNN. The novelty of their work lies in the proposed loss function and the efficient construction of the mini-batches for it.  Their method improves over the standard triplet loss, which pushes the impostor of  one offending class at a time. In contrast their loss pushes impostors of multiple classes for each update. They claim, that learning while pushing multiple impostor classes leads to a better metric. They verify their claim experimentally and benchmark their methods against relevant baselines. Overall the paper is well written, well-presented and provides clear experimental results supporting their claim.  Advantages: - Novel loss function for metric learning, which works well over standard approaches - Easy to implement and understand - Simple method for efficient mini-batch construction   Disadvantages:  - Method might not be extensible to cases where we only have similar and dissimilar constraints as it needs labels for constructing the tuple in an efficient manner.
Novelty/Originality: The novelty of the paper is two fold. First, they analyze the widely used triplet loss and its variants. Doing so, they propose a modification for the triplet loss, coined as the tuplet loss. Second, for the proposed tuplet loss, they propose and efficient method for forming the mini-batch of and SGD. This is quite simple and straightforward to implement, and does not come at a high computational cost which come into play when people use strategies like hard negative mining.   There are few things which I would like the authors to respond to: Q1: Strong usage of "Slow Convergence issue"  In line 047, they mention that people have evaluated combining softmax loss with contrastive or triplet loss for overcoming Slow Convergence issue. This is  mentioned in the abstract and other places as well. However,  this is not entirely true. There are other reasons for which this combination  has been tried.  For example, in [24], it was shown that while training the CNN for  face dataset, using both classification and identification (pairwise loss) gave superior performance than using either alone. They showed that using both losses  they could obtain a regularization and hence a better solution, but their goal was not to correct "slow convergence". Also, the statement in conclusion at L427 is too bold stating performance of triplet loss as "unsatisfactory convergence". This is too strong and should be revised, given in their own results the triplet loss performs quite decently in some cases.  Q2: In section 3.2.2 they perform regularization on the embedding vectors to be $\ell_2$ regularized. I was curious, if they tried using cosine similarity directly to avoid this explicit regularization?   Q3: In Table 1, I would like to see the results when negative mining is added for ovo and mc, the same way it was done in Table 2.   Q4: Just to be clear, in section 4.2 when the authors discuss results in Table 2, they mention negative data mining. I am assuming this refers to negative class mining. If that is the case, it should be mentioned as that to avoid confusion given in L222 a clear distinction was made between them. Also, in the methods where no negative class mining is mentioned, how is it replaced? by random classes. This should be made explicit if that is the case?  Q5: In table 2, why were all the methods stopped at 20k iterations? I find it strange that the optimal number of early stopping iterations is the same for all these different loss functions? Can the authors explain which method was chosen to set this 20k iteration? Also, I think this should be revised and the number of iterations should be set by cross-validation for each loss independently.