#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various highdimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domaindependent learned hash code may further improve these results. Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

This paper is already available on arxiv and cited 10 times. It is a very good paper introducing a new approach to count-based exploration in deep reinforcement learning based on using binary hashcodes. The approach is interesting, the presentation is didactical, the results are good and the related literature is well covered. I learned a lot from reading this paper, so my only criticisms are on secondary aspects.
I have few global points:
- I think it would be interesting to first train the convolutional autoencoder, and only then use your exploration method to compare its performance to state-of-the-art methods. This would allow for a more detailed analysis by disentangling the factors of potential inefficiency.
- I would put the "related work" section more forward in the paper, as the presentation of the methods would benefit from the points made in the first paragraph of related work and in the comparison with [4]. As a reader, I found that these points were important and arriving too late.
- Maybe section 5.4 of supplementary material should move into the main paper, and I don't share your main conclusion about it: If you look closely at Table 6, using n(s,a) is most often better than using n(s). To me, prefering n(s) to n(s,a) based just on the max in your table is inadequate, because maybe with an "optimized" (k,\beta) pair, you will find that using n(s,a) is better...
More local points:
line 41: "However, we have not seen a very simple and fast method that can work across different
"domains." How do you disqualify VIME here?
line 113: "An alternative method must be used to ensure that distinct states are mapped to distinct binary codes." Maybe the writing could be more direc there, presenting directly the appropriate approach.
Fig5: despite magnifying the picture several times, I cannot see the baseline. Didn't you forget it?
Footnote 4: the statement is weak. You should give reference to the state-of-the-art performance papers.
In the supplementary part:
Section 1 would benefit from a big table giving all params and their values. Beyond you, it would be nice if the community could move towards such better practices. As is, Section 1 is hardly readable and it is very difficult to check if everything is given.
Section 3: why do you show these images? Do you have a message about them? Either say something of interest about them, or remove them!
line 136: can (be) obtained (this was the only typo I could find, congratulations! ;))


Fully Decentralized Policies for Multi-Agent Systems: An Information Theoretic Approach
Learning cooperative policies for multi-agent systems is often challenged by partial observability and a lack of coordination. In some settings, the structure of a problem allows a distributed solution with limited communication. Here, we consider a scenario where no communication is available, and instead we learn local policies for all agents that collectively mimic the solution to a centralized multi-agent static optimization problem. Our main contribution is an information theoretic framework based on rate distortion theory which facilitates analysis of how well the resulting fully decentralized policies are able to reconstruct the optimal solution. Moreover, this framework provides a natural extension that addresses which nodes an agent should communicate with to improve the performance of its individual policy.

This paper considers reinforcement learning in decentralized, partially observable environments. It presents an "information theoretic framework based on rate distortion theory which facilitates analysis of how well fully decentralized policies are able to reconstruct the optimal
solution" and presents an extension "that addresses which nodes an agent should communicate with to improve the performance of its individual policy." The paper presents some proof of concept results on an optimal power flow domain.
On the positive side, the idea of applying rate distortion to this problem is interesting and as far as I can tell novel. I rather like it and think it is a promising idea.
A major problem, however, is that even though the paper is motivated from the perspective of sequential problems, the formalization is (1) is a static problem. (The authors should describe in how far this corresponds to a DCOP problem). In fact, I believe that the approach is only approximating the right objective for such 1-shot problems: the objective is to approximate the optimal *centralized* policy p(u* | x). However, such a centralized policy will not take into account the affect of its actions on the distribution of information over the agents in the future, and as such much be sub-optimal from in a *decentralized* regime. Concretely: the optimal joint policy for a Dec-POMDP might sacrifice exploiting some information if it can help maintaining predictability and therefore the future ability to successfully coordinate.
Another weakness of the proposed approach is in fact this dependence on the optimal centralized policy. Even though computing the solution for a centralized problem is easier from a complexity point of view (e.g., MDPs are in P), this in practices is meaningless since---save very special cases---the state and action spaces of these distributed problems are exponential in the number of agents. As such, it is typically not clear how to obtain this centralized policy.

The paper gives a highly inadequate characterization of the related work.
For instance just the statement "In practice however, POMDPs are rarely employed due to computational limitations [Nair et al., 2005]." is wrong in so many ways:
-the authors mentions 'POMDP' but they cite a paper that introduces the 'networked distributed POMDP (ND-POMDP)' which is not a POMDP at all (but rather a special case of decentralized POMDP (Dec-POMDPs)).
-the ND-POMDP makes very strong assumptions on the interactions of the agents ('transition and observation dependence'): essentially the agents cannot influence each other. Clearly this is not the use case that the paper aims at (e.g., in power flow the actions of the agents do affect each other), so really the paper should talk about (perhaps factored) Dec-POMDPs.
-Finally, this statement seems a way to try and dismiss "using a Dec-POMDP" on this ground. However, this is all just syntax: as long as the goal is to solve a decentralized decision making problem which is not a very special sub-class** one can refuse to call it a Dec-POMDP, but the complexity will not go away.
**e.g., see: Goldman, C.V. and Zilberstein, S., 2004. Decentralized control of cooperative systems: Categorization and complexity analysis.
Also the claim "this method is the first of its kind to provide a data-driven way to learn local cooperative policies for multi-agent systems" strikes me as rather outrageous. I am not quite certain what "its kind" means exactly, but there are methods for RL in Dec-POMDPs that are completely decentralized, e.g.:
Peshkin L, Kim KE, Meuleau N, Kaelbling LP. Learning to cooperate via policy search. In Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence 2000 Jun 30 (pp. 489-496). Morgan Kaufmann Publishers Inc.
An overview of Dec-POMDP RL methods is given in:
Oliehoek, F.A. and Amato, C., 2016. A concise introduction to decentralized POMDPs. Springer International Publishing.
All in all, I think there is an interesting idea in this submission that could be shaped into good paper. However, there are a number of serious concerns that I think preclude publication at this point.


Model-Powered Conditional Independence Test
We consider the problem of non-parametric Conditional Independence testing (CI testing) for continuous random variables. Given i.i.d samples from the joint distribution f (x, y, z) of continuous random vectors X, Y and Z, we determine whether X ? ? Y |Z. We approach this by converting the conditional independence test into a classification problem. This allows us to harness very powerful classifiers like gradient-boosted trees and deep neural networks. These models can handle complex probability distributions and allow us to perform significantly better compared to the prior state of the art, for high-dimensional CI testing. The main technical challenge in the classification problem is the need for samples from the conditional product distribution f CI (x, y, z) = f (x|z)f (y|z)f (z) -the joint distribution if and only if X ? ? Y |Z. -when given access only to i.i.d. samples from the true joint distribution f (x, y, z). To tackle this problem we propose a novel nearest neighbor bootstrap procedure and theoretically show that our generated samples are indeed close to f CI in terms of total variational distance. We then develop theoretical results regarding the generalization bounds for classification for our problem, which translate into error bounds for CI testing. We provide a novel analysis of Rademacher type classification bounds in the presence of non-i.i.d nearindependent samples. We empirically validate the performance of our algorithm on simulated and real datasets and show performance gains over previous methods.

The authors propose a general test to test conditional independence between three continuous variables. The authors don't have strong parametric assumptions -- rather, they propose a method that uses samples from the joint distribution to create samples that are close to the conditional product distribution in total variation distance. Then, they feed these samples to a binary classifier such as a deep neural net, and after training it on samples from each of the two distributions, use the predictive error of the net to either accept or reject the null hypothesis of conditional independence. They compare their proposed method to those existing in literature, both with descriptions and with experimental results on synthetic and real data.
Conditional independence testing is an important tool for modern machine learning, especially in the domain of causal inference, where we are frequently curious about claims of unconfoundedness. The proposed method appears straightforward and powerful enough to be used for many causal applications.
Additionally, the paper is well-written. While the details of the theoretical results and proofs can be hard to follow, the authors do a good job of summarizing the results and making the methodology clear to practitioners.
To me, the most impressive part of the paper is the theoretical results. The proposed method is a somewhat straightforward extension of the method for testing independence in the paper "Revisiting Classifier Two-Sample Tests"  (https://arxiv.org/pdf/1610.06545.pdf), as the authors acknowledge (they modify the test to include a third variable Z, and after matching neighbors of Z, the methods are similar). However, the guarantees of the theoretical results are impressive.
A few more comments:
-Is there a discrete analog? The bootstrap becomes even more powerful when Z is discrete, right?
-How sensitive is the test on the classification method used? Is it possible that a few changed hyperparameters can lead to different acceptance and rejection results? This would be useful to see alongside the simulations
-Section 1.1 of Main Contributions is redundant -- the authors summarize their approach multiple times, and the extent appears unnecessary.
-Writing out Algorithm 2 seems unnecessary — this process can be described more succinctly, and has been described already in the paper
-Slight typos — should be “classifier” in line 157, “graphs” in 280


Gauging Variational Inference
Computing partition function is the most important statistical inference task arising in applications of Graphical Models (GM). Since it is computationally intractable, approximate methods have been used in practice, where mean-field (MF) and belief propagation (BP) are arguably the most popular and successful approaches of a variational type. In this paper, we propose two new variational schemes, coined Gauged-MF (G-MF) and Gauged-BP (G-BP), improving MF and BP, respectively. Both provide lower bounds for the partition function by utilizing the so-called gauge transformation which modifies factors of GM while keeping the partition function invariant. Moreover, we prove that both G-MF and G-BP are exact for GMs with a single loop of a special structure, even though the bare MF and BP perform badly in this case. Our extensive experiments indeed confirm that the proposed algorithms outperform and generalize MF and BP.

The paper presents two new approaches to variational inference for graphical models with binary variables - an important fundamental problem. By adding a search over gauge transformations (which preserve the partition function) to the standard variational search over (pseudo-)marginals, the authors present methods called gauged mean field (G-MF, Alg 1) and gauged belief propagation (G-BP, Alg 2). Very helpfully, both are guaranteed to yield a lower bound on the true partition function - this may be important in practice and enables empirical testing for improved performance even for large, intractable models (since higher lower bounds are better). Initial experiments demonstrate the benefits of the new methods.
The authors demonstrate technical skill and good background knowledge. The new approaches seem powerful and potentially very helpful in practice, and appear very sensible and natural when introduced in Sec 3.1 given the earlier background. The discussion and extensions in Sec 3.2 are interesting. Theorem 1 is a strong theoretical result - does it hold in general for a graph with at most one cycle?
However, I have some important questions/concerns. If these were fixed, then this could be a strong paper:
1. Algorithm details and runtime.
Both algorithms require a sequence of decreasing barrier terms \delta_1 > \delta_2... but I don't see any comments on how these should be set? What should one do in practice and how much impact could this have on runtime and accuracy?
There appear to be no comments at all on runtime. Both algorithms will likely take much longer than the usual time for MF or BP.
Are we guaranteed an improvement at each iteration? Even if not, because we always have a lower bound, we could use the max so far to yield an improving anytime algorithm.
Without any runtime analysis, the empirical accuracy comparisons are not really meaningful. There are existing methods which will provide improved accuracy for more computation, and comparison would be very helpful (e.g. AIS or clamping approaches such as Weller and Jebara NIPS 2014; or Weller and Domke AISTATS 2016, which shows monotonic improvement for MF and TRW when each additional variable is clamped).
Note that with clamping methods: if a model has at most one cycle, then by clamping any one variable on the cycle and running BP, we obtain the exact solution (compare to Theorem 1).
More generally, if sufficient variables are clamped st the remainder is acyclic then we obtain the exact solution. Of course, for a graph with many cycles this might take a long time - but this shows why considering accuracy vs runtime is important.
What is the quality of the marginals returned by the new approaches? Does a better partition function estimate lead to better marginals?
2. Explanation of the gauge transformation.
Given the importance here of the gauge transformation, which may be unfamiliar to many readers, it would help significantly if the explanation could be expanded and at least one example provided. I recognize that this is challenging given space constraints - even if it were in the Appendix it would be a big help.
l. 177-179 please elaborate on why (8) yields a lower bound on the partition function.
Minor details:
1.238-240 Interesting that G-BP performs similarly to BP for log-supermodular models. Any further comment on this?
If/when runtime is added, we need more details on which method is used for regular BP.
A few typos -
l. 53 alternative -> alternating (as just defined in the previous sentence). I suggest that 'frustrated cycle' may be a better name than 'alternating cycle' since a balanced cycle of even length could have edge signs which alternate.
l. 92 two most -> two of the most
l. 134-5 on the expense of making underlying -> at the expense of making the underlying
In Refs, [11] ising -> Ising; [29] and [30] bethe -> Bethe


Scalable Demand-Aware Recommendation
Recommendation for e-commerce with a mix of durable and nondurable goods has characteristics that distinguish it from the well-studied media recommendation problem. The demand for items is a combined effect of form utility and time utility, i.e., a product must both be intrinsically appealing to a consumer and the time must be right for purchase. In particular for durable goods, time utility is a function of inter-purchase duration within product category because consumers are unlikely to purchase two items in the same category in close temporal succession. Moreover, purchase data, in contrast to rating data, is implicit with non-purchases not necessarily indicating dislike. Together, these issues give rise to the positive-unlabeled demand-aware recommendation problem that we pose via joint low-rank tensor completion and product category inter-purchase duration vector estimation. We further relax this problem and propose a highly scalable alternating minimization approach with which we can solve problems with millions of users and millions of items in a single thread. We also show superior prediction accuracies on multiple real-world datasets.

Paper revolves over the observation that in e-commerce world customers rarely purchase two items that belong to the same category (e.g. Smartphone) in close temporal succession. Therefore, they claim that a robust recommendation system should incorporate both utility and time utility. An additional problem that is tackled in the paper is that many e-commerce systems have no explicit negative feedback to learn from (for example one can see only what items customer purchased - positives - and no explicit negatives in form of items user did not like).
I believe that the second problem they mention is not as big of a concern as advertised by authors. In absence of any explicit negative signal good replacements are long dwell-time clicks that did not end up in purchase, as well as cart additions that did not end up in the final purchase or returns. Many companies are also implementing swipe to dismiss that is useful for collecting explicit negative signal and can be applied to any e-commerce site easily.
The authors clearly define and describe the objective of modeling user intent as combined effect of form utility and time utility and identify limitations of minimizing the objective when formulated in the simplest form (as tensor nuclear norm) in terms of high computational cost. To tackle that the authors propose to relax the problem to a matrix optimization problem with a label-dependent loss and propose an efficient alternating minimization algorithm whose complexity is proportional to the number of purchase records in the matrix.
I see the biggest strength of this paper as reformulating the objective and adding relaxations that significantly reduce computational cost and still achieves good result.
Introducing time component adds another dimension and as you mentioned it results in a significant rise in computational cost O(mnl) where l is the number of time slots. What happens when you drop the time component in terms of accuracy? I saw that in your experiment you compare to many traditional collaborative filtering algorithms but you do not compare to your algorithm that doesn't take time in account or uses different definitions of what time slot is (e.g. just a s a two level definition it could be: purchased this month or not).
Related work is generally covered well. Experiments only cover offline evaluation. Therefore, we have no sense of how this algorithm would work in practice in a production system. For example I would be curious to compare against just showing popular items as recommendations (as done in this paper: E-commerce in your inbox: Product recommendations at scale, KDD 2015). Paper would be  much stronger with such results. Also, additional evaluation metrics should be considered, like traditional ranking metrics, precision at K, etc.
It would be also interesting to put this in context of ad retargeting. It is well known that users on the Web are re-targeted for the same product they bought (or same category) even after they bought the product. While some see this as a waste of ad dollars some claim this still brings in revenue as users often want to see items from same category in order to get reassurance that you bought the best product from the category.
In practice, it also happens that they return the original product and purchase the recommended one from same category. In practice, recommender production systems take this signal in account as well.


Active Exploration for Learning Symbolic Representations
We introduce an online active exploration algorithm for data-efficiently learning an abstract symbolic model of an environment. Our algorithm is divided into two parts: the first part quickly generates an intermediate Bayesian symbolic model from the data that the agent has collected so far, which the agent can then use along with the second part to guide its future exploration towards regions of the state space that the model is uncertain about. We show that our algorithm outperforms random and greedy exploration policies on two different computer game domains. The first domain is an Asteroids-inspired game with complex dynamics but basic logical structure. The second is the Treasure Game, with simpler dynamics but more complex logical structure.

Key question asked in the paper: Can an agent actively explore to build a probabilistic symbolic model of the environment from a given set of option and state abstractions? And does this help make future exploration easy via MCTS? This paper proposes a knowledge representation framework for addressing these questions.
Proposal: A known continuous state space and discrete set of options (restricted to subgoal options) is given. An options model is treated a semi-MDP process. A plan is defined as a sequence of options. The idea of a probabilistic symbol is invoked from earlier work to refer to a distribution over infinitely many continuous low-level states. The idea of state masks is introduced to find independent factors of variations. Then each abstract subgoal option is defined as a policy that leads to a subgoal for the masked states, for e.g. opening a door. But since there could be many doors in the environment, the idea of a partitioned abstract subgoal option is proposed to bind together subgoal options for each instance of a door. The agent then uses these partitioned abstract subgoal options during exploration. Optimal exploration then is defined as finding a policy via MCTS that leads to greatest reduction of uncertainty over distributions over proposed options symbolic model.
Feedback:
- There are many ideas proposed in this paper. This makes it hard to clearly communicate the contributions. This work builds on previous work from Konidaris et al. Is the novel contribution in the MCTS algorithm? The contributions should be made more explicit for unfamiliar readers.
- It is difficult to keep track of what's going on in sec 3. I would additionally make alg1 more self-inclusive and use one of the working examples (asteroid or maze) to explain things.
- Discuss assumption of restriction to subgoal option. What kind of behaviors does this framework not permit? discuss this in the paper
- Another restrictive assumption: "We use factors to reduce the number of potential masks, i.e. we assume that if state variables i and j always change together in the observations, then this will always occur. An example of a factor could be the (x, y, z) position of your keys, because they are almost never moved along only one axis". What are the implications of this assumptions on the possible range of behaviors?
- The experimental setup needs to be developed more to answer, visualize and discuss a few interesting questions: (1) how does the symbol model change with active exploration? (2) visualize the space of policies at various intermediate epochs. Without this it is hard to get an intuition for the ideas and ways to expand them in the future.
- I believe this is a very important line of work. However, clarity and articulation in the writing/experiments remains my main concern for a clear accept. Since some of the abstractions are fairly new, the authors also need to clearly discuss the implication of all their underlying assumptions.


Active Learning from Peers
This paper addresses the challenge of learning from peers in an online multitask setting. Instead of always requesting a label from a human oracle, the proposed method first determines if the learner for each task can acquire that label with sufficient confidence from its peers either as a task-similarity weighted sum, or from the single most similar task. If so, it saves the oracle query for later use in more difficult cases, and if not it queries the human oracle. The paper develops the new algorithm to exhibit this behavior and proves a theoretical mistake bound for the method compared to the best linear predictor in hindsight. Experiments over three multitask learning benchmark datasets show clearly superior performance over baselines such as assuming task independence, learning only from the oracle and not learning from peer tasks.

The authors propose an extension of the active learning framework to address two key issues: active learning of multiple related tasks, and using the 'peers' as an additional oracle to reduce the cost of querying the user all the time. The experimental evaluation of the proposed learning algorithms is performed on several benchmark datasets, and in this context, the authors proved their claims.
The main novelty lies in the extension of the SHAMPO algorithm so that it can leverage the information from multiple tasks, and also peer-oracles (using the confidence in the predictions), to reduce the annotation efforts by the user. A sound theoretical formulation to incorporate these two is proposed. Experimental results show the validity of the proposed.
The authors did not comment on the cost of using the peers as oracles, but assumed their availability.  This should be detailed more. Also, it is difficult to assess the model's performance fully compared to the SHAMPO algorithm as its results are missing in Table 1. The accuracy difference between peer and one version of the model is marginal. What is the benefit of the former?
Overall, this is a nice paper with a good mix of theory and  empirical analysis.
Lots of typos in the text that should be corrected.


K-Medoids for K-Means Seeding
We show experimentally that the algorithm clarans of Ng and Han (1994) finds better K-medoids solutions than the Voronoi iteration algorithm of Hastie et al. (2001). This finding, along with the similarity between the Voronoi iteration algorithm and Lloyd's K-means algorithm, motivates us to use clarans as a K-means initializer. We show that clarans outperforms other algorithms on 23/23 datasets with a mean decrease over k-means-++ (Arthur and Vassilvitskii, 2007) of 30% for initialization mean squared error (MSE) and 3% for final MSE. We introduce algorithmic improvements to clarans which improve its complexity and runtime, making it a viable initialization scheme for large datasets.

The authors propose to use a particular version of the K-medoids algorithm (clarans - that uses iterative swaps to identify the medoids) for initializing k-means and claim that this improves the final clustering quality. The authors have also tested their claims with multiple datasets, and demonstrated their performance improvements. They have also published code that will be made open after the review process.
Comments:
1. The paper is easy to read and follow, and the authors have done a good job placing their work in context. I appreciate the fact that the optimizations are presented in a very accessible manner in Section 4. As the authors claim, open source code is an important contribution.
2. The authors should consider the following work for inclusion in their discussions and/or comparisons:
Bahmani, Bahman, et al. "Scalable k-means++." Proceedings of the VLDB Endowment 5.7 (2012): 622-633.
3. While the authors have compared actual run times of different algorithms in Section 5, can they also provide some idea of the theoretical complexity (whenever possible)?
4. Related to previous comment: how much impact would the actual implementation have on the runtimes, particularly given that the “bf” algorithm uses BLAS routines and hence can be implicitly parallelizing matrix operations. Is there any other place in the implementations where parallel processing is done?
5. Please number the data sets in Table 3 to make it easy to compare with table 4.
6. In column 1(km++) of Table 4, why is it that km++ does not run unto 90 iterations always given that TL was based on this time?
7.  In 5.2, the claim that “clarans often finds a globally minimal initialization” seems a bit too speculative. If the authors want to claim this they must run many many rounds at least in a few datasets, and show that the performance cannot get better by much.


Matrix Norm Estimation from a Few Entries
Singular values of a data in a matrix form provide insights on the structure of the data, the effective dimensionality, and the choice of hyper-parameters on higher-level data analysis tools. However, in many practical applications such as collaborative filtering and network analysis, we only get a partial observation. Under such scenarios, we consider the fundamental problem of recovering various spectral properties of the underlying matrix from a sampling of its entries. We propose a framework of first estimating the Schatten k-norms of a matrix for several values of k, and using these as surrogates for estimating spectral properties of interest, such as the spectrum itself or the rank. This paper focuses on the technical challenges in accurately estimating the Schatten norms from a sampling of a matrix. We introduce a novel unbiased estimator based on counting small structures in a graph and provide guarantees that match its empirical performances. Our theoretical analysis shows that Schatten norms can be recovered accurately from strictly smaller number of samples compared to what is needed to recover the underlying low-rank matrix. Numerical experiments suggest that we significantly improve upon a competing approach of using matrix completion methods.

Summary of paper:
The paper proposes an estimator for the Schatten norm of a matrix when only few entries of the matrix is observed. The estimator is based on the relation between the Schatten k-norm of a matrix and the total weight of all closed k-length walks on the corresponding weighted graph. While the paper restricts the discussion to symmetric positive semi-definite matrices, and guarantees given for uniform sampling, the generic principle is applicable for any matrix and any sampling strategy. The results hold under standard assumptions, and corollaries related to spectrum and rank estimation are also provided in the appendix.
Clarity:
The paper is very well written and organised, including some of the material in the supplementary. The motivation for the work, and the discussion that builds up to the estimator and well presented. The numerical simulations are also perfectly positioned.
Quality:
The technical quality of the paper is high. The main structure of the proofs are correct, but I didn’t verify the computations in equations (26)-(36).
The most pleasing component of the theoretical analysis is the level at which it is presented. The authors restrict the discussion to symmetric psd matrices for ease of presentation, but the sampling distribution is kept general until the final guarantees in Section 3.1. The authors also correctly note estimating the Schatten norm itself is never the end goal, and hence, they provide corollaries related to spectrum and rank estimation in the appendix. The numerical simulations do not involve a real application, but clearly show the advantage of the proposed estimator over other techniques when sampling rate is very small. That being said, I feel that further numerical studies based on this approach is important.
Significance:
The paper considers a classical problem that lies at the heart of several matrix related problems. Hence, the paper is very relevant for the machine learning community. The generality at which the problem is solved, or its possible extensions, has several potential applications.
To this end, I feel that it will be great if the authors can follow up this work with a longer paper that also includes the more complicated case of rectangular matrices, analyses other sampling strategies (like column sampling / sub matrix sampling), and more thorough experimental studies.
Originality:
Since the underlying problem is classical, one should consider that many ideas have been tried out in the literature. I feel that the individual components of the discussion and analysis are fairly standard techniques (except perhaps the computational simplification for k<=7). But I feel that the neat way in which the paper combines everything is very original. For instance, the use of polynomial concentration in the present context is intuitive, but the careful computation of the variation bound is definitely novel.
Minor comments:
1. Line 28-29: Can we hope to estimate the spectrum of a matrix from a submatrix? I am not very optimistic about this.
2. Appendix B: It would be nice if a description of this method is added. In particular, what is the time complexity of the method?
3. Line 238,262,281,292: In standard complexity terminology, I think d^2p = O(something) should actually be d^2p = \Omega(something). Big-O usually means at most upto constant, whereas here I think you would like to say that the number of observed entries is at least something (Big-Omega).
4. Line 263: How do you get the condition on r? The third term in (8) is less than 1 if r is less than d^(1-1/k).


Hypothesis Transfer Learning via Transformation Functions
We consider the Hypothesis Transfer Learning (HTL) problem where one incorporates a hypothesis trained on the source domain into the learning procedure of the target domain. Existing theoretical analysis either only studies specific algorithms or only presents upper bounds on the generalization error but not on the excess risk. In this paper, we propose a unified algorithm-dependent framework for HTL through a novel notion of transformation function, which characterizes the relation between the source and the target domains. We conduct a general risk analysis of this framework and in particular, we show for the first time, if two domains are related, HTL enjoys faster convergence rates of excess risks for Kernel Smoothing and Kernel Ridge Regression than those of the classical non-transfer learning settings. Experiments on real world data demonstrate the effectiveness of our framework.

The paper presents a supervised non-parametric hypothesis transfer learning (HTL) approach for regression and its analysis, aimed at the cases where one has plenty of training data coming from the source task and few examples from the target one. The paper makes an assumption that the source and the target regression functions are related through so called transformation function (TF). The TF is assumed to have some parametric form (e.g. linking regressions functions linearly) and the goal of an algorithm is to recover its parameters. Once these parameters are learned, the hypothesis trained on the source task can be transformed to the hypothesis designated for the target task. The paper proposes two ways for estimation of these parameters, that is through kernel smoothing and kernel ridge regression. For both cases the paper presents consistency bounds, showing that one can have an improvement in the exponent of the non-parametric rate once the TF is smoother, in Holder sense, than the target regression function. In addition, the paper argues that one can run model selection over the class of TF through cross-validation and shows some bounds to justify this, which is not particularly surprising. Finally, the paper concludes with numerical experiments on two regression tasks on the real-world small-scale datasets.
The rates stated in main theorems, 2 and 3 appear to make sense. One can see this from theorem 1, which is used to prove both, and essentially states an excess risk as a decomposition into the sum of excess risk bounds for source/TF regression functions. The later are bounded through well established excess risk bounds from the literature.
I do not think this is a breakthrough paper, but it has a novel point as it addresses the theory of HTL in the non-parametric setting because it studies the Bayes risk rates rather than generalization bounds as was done previously.
The paper has a novel bit as it studies non-parametric consistency rates of HTL, while previous theoretical works focused on the risk bounds. As one should expect, the improvement then occurs in the right place, that is in the exponent of the rate, rather than in some additive or multiplicative term. At the same time the algorithm appears to be simple and easy to implement. This makes this paper a good contribution to the literature on HTL. Also, the paper is quite well positioned in the related work. Unfortunately, it doesn't compare to previous HTL approaches experimentally and experimental evaluation is, frankly, quite modest, despite the simplicity of the algorithm.


Multi-View Decision Processes: The Helper-AI Problem ������������������������������� �����������������������
We consider a two-player sequential game in which agents have the same reward function but may disagree on the transition probabilities of an underlying Markovian model of the world. By committing to play a specific policy, the agent with the correct model can steer the behavior of the other agent, and seek to improve utility. We model this setting as a multi-view decision process, which we use to formally analyze the positive effect of steering policies. Furthermore, we develop an algorithm for computing the agents' achievable joint policy, and we experimentally show that it can lead to a large utility increase when the agents' models diverge.
1 Introduction.
In the past decade, we have been witnessing the fulfillment of Licklider's profound vision on AI [Licklider, 1960]:
Man-computer symbiosis is an expected development in cooperative interaction between men and electronic computers.
Needless to say, such a collaboration, between humans and AIs, is natural in many real-world AI problems. As a motivating example, consider the case of autonomous vehicles, where a human driver can override the AI driver if needed. With advances in AI, the human will benefit most if she allows the AI agent to assume control and drive optimally. However, this might not be achievable-due to human behavioral biases, such as over-weighting the importance of rare events, the human might incorrectly override the AI. In the way, the misaligned models of the two drivers can lead to a decrease in utility. In general, this problem may occur whenever two agents disagree on their view of reality, even if they cooperate to achieve a common goal.
Formalizing this setting leads to a class of sequential multi-agent decision problems that extend stochastic games. While in a stochastic game there is an underlying transition kernel to which all agents (players) agree, the same is not necessarily true in the described scenario. Each agent may have a different transition model. We focus on a leader-follower setting in which the leader commits to a policy that the follower then best responds to, according to the follower's model. Mapped to our motivating example, this would mean that the AI driver is aware of human behavioral biases and takes them into account when deciding how to drive.
To incorporate both sequential and stochastic aspects, we model this as a multi-view decision process. Our multi-view decision process is based on an MDP model, with two, possibly different, transition kernels. One of the agents, hereafter denoted as P 1 , is assumed to have the correct transition kernel and is chosen to be the leader of the Stackelberg game-it commits to a policy that the second agent (P 2 ) best-responds to according to its own model. The agents have the same reward function, and are in this sense cooperative. In an application setting, while the human (P 2 ) may not be a planner, we motivate our set-up as modeling the endpoint of an adaptive process that leads P 2 to adopt a best-response to the policy of P 1 .
Using the multi-view decision process, we analyze the effect of P 2 's imperfect model on the achieved utility. We place an upper bound on the utility loss due to this, and also provide a lower bound on how much P 1 gains by knowing P 2 's model. One of our main analysis tools is the amount of influence an agent has, i.e. how much its actions affect the transition probabilities, both according to its own model, and according to the model of the other agent. We also develop an algorithm, extending backwards induction for simultaneous-move sequential games [c.f. Bošanskỳ et al., 2016], to compute a pair of policies that constitute a subgame perfect equilibrium.
In our experiments, we introduce intervention games as a way to construct example scenarios. In an intervention game, an AI and a human share control of a process, and the human can intervene to override the AI's actions but suffers some cost in doing so. This allows us to derive a multi-view process from any single-agent MDP. We consider two domains: first, the intervention game variant of the shelter-food game introduced by Guo et al. [2013], as well as an autonomous driving problem that we introduce here. Our results show that the proposed approach provides a large increase in utility in each domain, thus overcoming the deficiencies of P 2 's model, when the latter model is known to the AI.

The paper introduces multi-view decision processes (MVDP) as a cooperative game between two agents, where one agent may have an imperfect model of the transition probabilities of the other agent.
Generally, the work is motivated well, the problem is interesting and clearly relevant to AI and the NIPS community. The theoretical results are nicely complemented by experiments. Presentation is mostly smooth: the main goals are consistent with the narrative throughout the paper, but the some details from the experiments seem less clear (seem points below.)
Some points are confusing and need further clarification:
1. In the second footnote on page 3, what is a "worst-case best response"? This is not defined, seems like they should all share the property that they attain the maximum value against some policy.
2. The authors talk about solving using backward induction and make reference to Bozansky et al 2016 and Littman 1994, but those works treat exclsively the zero-sum case. MVDP is a cooperative game that can be at least as hard as general-sum games (by Lemma 1), and backward induction generally cannot solve these due to equilibrium selection problems leading to non-unique values for subgames, also shown in Zinkevich et al 2005. How is it possible to apply backward induction by propagating single Q and V values here then? The authors claim to find an approximate joint optimal policy, which is stronger than a Nash equilibrium, and it seems by Zinkevich et al 2005 this should at least require cyclic (nonstationary) strategies.
3. From the start the authors talked about stochastic policies, but this is a cooperative identical payoff game. It's not clear later (seeing the proof of Lemma 1) that mixed strategies are necessary due to the different views leading to an effectively general-sum game. How then could a the determinstic version of the algorithm that computes a pure stationary policy be approximately optimal? Even in the Stackelberg case, mixing could be necessary.
Minor:
The role of c in the c-inervention games is not clear. On page 6, the authors write a cost c(s) is subtracted from the reward r_t when P2 takes an action other. But then in Remark 1 gives the equation rho(s, a) = [rho(s) + cI{a_{t,2} != a^0}] / (1+c) does not subtract c. This can be easily fixed by rewording the text at bottom of page 6.


Towards Generalization and Simplicity in Continuous Control
This work shows that policies with simple linear and RBF parameterizations can be trained to solve a variety of widely studied continuous control tasks, including the gym-v1 benchmarks. The performance of these trained policies are competitive with state of the art results, obtained with more elaborate parameterizations such as fully connected neural networks. Furthermore, the standard training and testing scenarios for these tasks are shown to be very limited and prone to over-fitting, thus giving rise to only trajectory-centric policies. Training with a diverse initial state distribution induces more global policies with better generalization. This allows for interactive control scenarios where the system recovers from large on-line perturbations; as shown in the supplementary video.

The paper evaluates natural policy gradient algorithm with simple linear policies on a wide range of “challenging” problems from OpenAI MuJoco environment, and shows that these shallow policy networks can learn effective policies in most domains, sometimes faster than NN policies. It further explores learning robust and more global policies by modifying existing domains, e.g. adding many perturbations and diverse initial states.
The first part of the paper, while not proposing new approaches, offers interesting insights into the performance of linear policies, given plethora of prior work on applying NN policies as default on these problems. This part can be further strengthened by doing ablation study on the RL optimizer. Specifically, GAE, sigma vs alpha in Eq. 5, and small trajectory batch vs large trajectory batch (SGD vs batch opt). In particular, it is nice to confirm if there are subtle differences from prior NPG implementation with linear policies that made the results in this paper significantly better (e.g. in TRPO [Schulman et. al., 2015], one of the major differences was sigma vs alpha).
It could be unsurprising that a global linear policy may perform very well in narrow state space. The second part of the paper tries to address such concern by modifying the environments such that the policies must learn globally robust policies to succeed. The paper then again showed that the RBF/linear policies could learn effective recover policies. It would be great to include corresponding learning curves for these in Section 4 beyond Figure 4, comparing NN, linear and RBF. Are there figures available?
Additional comment:
- A nice follow-up experiment to show is if linear policies can solve control problems from vision, with some generic pre-trained convolutional filters (e.g. ImageNet). This would further show that existing domains are very simple and do not require deep end-to-end optimization.


Beyond normality: Learning sparse probabilistic graphical models in the non-Gaussian setting
We present an algorithm to identify sparse dependence structure in continuous and non-Gaussian probability distributions, given a corresponding set of data. The conditional independence structure of an arbitrary distribution can be represented as an undirected graph (or Markov random field), but most algorithms for learning this structure are restricted to the discrete or Gaussian cases. Our new approach allows for more realistic and accurate descriptions of the distribution in question, and in turn better estimates of its sparse Markov structure. Sparsity in the graph is of interest as it can accelerate inference, improve sampling methods, and reveal important dependencies between variables. The algorithm relies on exploiting the connection between the sparsity of the graph and the sparsity of transport maps, which deterministically couple one probability measure to another.

This paper presents a method for identifying the independence structure of undirected probabilistic graphical models with continuous but non-Gaussian distributions, using SING, a novel iterative algorithm based on transport maps. The authors derive an estimate for the number of samples needed to recover the exact underlying graph structure with some probability and demonstrate empirically that SING can indeed recover this structure on two simple domains, where comparable methods that make invalid assumptions about the underlying data-generating process fail.
Quality.
The paper seems technically sound, with its claims and conclusions supported by existing and provided theoretical and empirical results. The authors mostly do a good job of justifying their approach, but do not discuss potential issues with the algorithm. For example, what is the complexity of SING? Since it has a variable-elimination-like aspect, is complexity exponential in the treewidth of the generated graph? If so, how is this approach feasible for non-sparse graphs? In that vein, I would like to see how SING fares empirically when applied to more realistic and higher-dimensional problems and in comparison to other standard MRF-learning approaches. The latter typically make overly strong assumptions, but can still perform well given that there are rarely enough data points for true structures to be recovered.
Clarity.
The writing is good, and each subsection is clear, but the authors should better define their notation and spend more time explaining the algorithm as a whole and gluing the pieces together. In particular, what exactly happens in subsequent iterations? A quick example or a brief high-level walk-through of the algorithm would be helpful here to assemble all of the pieces that have been described into a coherent whole.
Also, is the amount of sparsity that is estimated monotonic over iterations? If SING over-estimates the amount of sparsity present in the data, can it then reduce the amount of sparsity it is estimating? The sparsity pattern identified on line 8 of Alg. 1 is used in line 2, correct? (Note that you use I_{S_{i+1}} and S^B_{I_{i}}, respectively, on these lines – is I_i the same as I_{S_{i+1}} from the previous iteration?) It’s not clear to me how this sparsity pattern gets used in the estimation of the transport map. You should more clearly define where the original and induced graphs come from. From what I can tell, the original graph is the result of the thresholded \Sigma (but no graph is discussed in Alg. 1), and the induced graph comes from some variable-elimination-like algorithm, but it’s not clear how this actually happens. Does Alg. 1 perform variable elimination? What are the factors? Why do you need to do this?
Originality.
The approach seems novel and original, although I am not that familiar with transport maps and their corresponding literature.
Significance.
The presented approach seems interesting and potentially useful, but lacks sufficient evaluation as the authors do not evaluate or discuss the complexity of their algorithm and only evaluate on two simple problems. It would also be useful to properly conclude section 5 by better explaining and contextualizing the result. In particular, is the number of samples needed high or low relative to other works that estimate similar types of graph structures, or is this completely novel analysis?
Overall.
With additional results, editing, and a more cohesive explanation of their approach, this could be a successful publication, but I do not think it has reached that threshold as it stands.
Detailed comments:
- Please define your notation. For example, in the introduction you should define the independence symbol and \bf{x}_{jk}. In section 2, you use both d_j d_k for (I think) partial derivatives, but then later use d_{jk} for (I assume) the same quantity, but define neither. You should define det() and the circle, which is typically used for function composition.  The V on line 154 should be defined. The projection (?) operator on the equation below line 162 should be defined. Etc.
- Line 75: inline citations should be Author (Year), instead of the number (use \citet instead of \citep if using natbib)
- There are a few typos and duplicated words (e.g., lines 86, 191).
- The font sizes on your figures are barely legible when printed – these should be increased.
- Alg 1., line 1, “DECREASING = TRUE” does not show when printed, perhaps due to font choice or embedding.
- Lines 156 and 173: These should be referred to as Equation (6) and Equation (7), not Line (6) and (7).
- Figure 3d shows SING(B=1), but the B=1 case has not been explained before this, so it looks like a failure case of the algorithm. Better would be to put the success case in Figure (3d) and then show the failure case in Figure (4) (separate from the others) and discuss the effect of B=1 there.
- Figure 5: it’s confusing to show the true graph (5a) as the ‘recovered’ graph. You should show the recovered adjacency matrix for SING as an added sub-figure of figure 5. Further, the x- and y- axis labels are just numbers, meaning that it’s not clear which numbers correspond to the hyperparameters, so lines 232-233 are confusing.
- Figures 3-6: would be useful to have a legend or explanation somewhere that blue means an edge and white means no edge.
----
After reading the author response, I have updated my score to 'weak accept' as I agree that the NIPS community will benefit from this work and my main concerns of complexity with respect to the use of variable elimination have been satisfied. However, the exact use of the variable elimination heuristics still remains somewhat unclear to me and the authors should clarify and be more explicit about how it is used in their algorithm.


Learning a Multi-View Stereo Machine
We present a learnt system for multi-view stereopsis. In contrast to recent learning based methods for 3D reconstruction, we leverage the underlying 3D geometry of the problem through feature projection and unprojection along viewing rays. By formulating these operations in a differentiable manner, we are able to learn the system end-to-end for the task of metric 3D reconstruction. End-to-end learning allows us to jointly reason about shape priors while conforming to geometric constraints, enabling reconstruction from much fewer images (even a single image) than required by classical approaches as well as completion of unseen surfaces. We thoroughly evaluate our approach on the ShapeNet dataset and demonstrate the benefits over classical approaches and recent learning based methods.

This submission proposes to learn 3D object reconstruction from a multi-view stereo setting end-to-end. Its main contribution is consistent feature projection and unprojection along viewing rays given camera poses and multiple images. Instead of mainly learning object shapes based on semantic image interpretation, this approach builds projective geometry into the approach to benefit from geometric cues, too. The paper proposes a way for differentiable unprojection that projects 2D image features to 3D space using camera projection equations and bilinear sampling. This is a simple yet effective way to implicitly build epipolar constraints into the system, because features from multiple views of the same world point unproject to the same voxel. While such combination of multiple views is of course not new, it is new as part of an end-to-end learnable approach.
I did not completely understand how recurrent grid fusion works, please explain in more detail. A dedicated figure might ease understanding, too. I would also appreciate a short note on GPU RAM requirements and run-time.
The paper is very well written and illustrations make it enjoyable to read. The approach is experimentally evaluated by comparison to traditional multi-view stereo matching algorithms (e.g., plane sweeping) and it is compared to the recent 3D-R2N2 approach, that also uses deep learning for 3D object reconstruction.
While all experiments are convincing and support claims of the paper, it would be good to compare to more deep learning-based 3D reconstruction approaches (several are cited in the paper, e.g. [27] + additional work like "DeMoN: Depth and Motion Network for Learning Monocular Stereo" from Thomas Brox's lab). Clear explanation of theoretical differences to [27] would further improve the paper. At present, it remains unclear whether this submission is just a multi-view extension of [27] without reading [27] itself. Better motivating the own contribution here seems necessary.
I believe this is a great paper and deserves to be published at NIPS, but also must admit that I still have some questions.


Approximate Supermodularity Bounds for Experimental Design
This work provides performance guarantees for the greedy solution of experimental design problems. In particular, it focuses on A-and E-optimal designs, for which typical guarantees do not apply since the mean-square error and the maximum eigenvalue of the estimation error covariance matrix are not supermodular. To do so, it leverages the concept of approximate supermodularity to derive nonasymptotic worst-case suboptimality bounds for these greedy solutions. These bounds reveal that as the SNR of the experiments decreases, these cost functions behave increasingly as supermodular functions. As such, greedy A-and E-optimal designs approach (1 − e −1 )-optimality. These results reconcile the empirical success of greedy experimental design with the non-supermodularity of the A-and E-optimality criteria.

This paper defines two types of approximate supermodularity, namely alpha-supermodularity and epsilon-supermodularity, both of which are extensions of the existing supermodularity definition. Then the authors show that the objectives of finding the A- and E-optimal Bayesian experimental design respectively belong to those two approximate supermodularity. Therefore, the greedy selection algorithm can obtain a good solution to the design problem that is within (1-e^{-1}) of the optimal, which is guaranteed by the standard result of supermodularity minimization.
This paper is nicely written and everything is easy to understand. I really enjoy reading it. The result gives a theoretical justification of the effectiveness of the greedy algorithm in finding A- and E- optimal design, which is further confirmed by experiments.
However, I also see some limitations of the proposed framework:
1. The analysis seems to apply only to Bayesian experimental design that the theta must have a prior. If not for the regularization term in the equation 4, it might be impossible to use any greedy algorithm. On the contrary, existing approach such as [10] does not have such limitation.
2. The relative error is different from the traditional definition, as the objective function f is normalized that f(D) <= 0 for all D. I suspect this is the key that makes the proof valid. Will this framework be applied to the unnormalized objective?
Although there are limitations, I still think the paper is above the acceptance threshold of NIPS.


On clustering network-valued data
Community detection, which focuses on clustering nodes or detecting communities in (mostly) a single network, is a problem of considerable practical interest and has received a great deal of attention in the research community. While being able to cluster within a network is important, there are emerging needs to be able to cluster multiple networks. This is largely motivated by the routine collection of network data that are generated from potentially different populations. These networks may or may not have node correspondence. When node correspondence is present, we cluster networks by summarizing a network by its graphon estimate, whereas when node correspondence is not present, we propose a novel solution for clustering such networks by associating a computationally feasible feature vector to each network based on trace of powers of the adjacency matrix. We illustrate our methods using both simulated and real data sets, and theoretical justifications are provided in terms of consistency.

Review Summary: This paper accomplishes a lot in just a few pages. This paper is generally well-written, but the first three sections in particular are very strong. The paper does a great job of balancing discussing previous work, offering theoretical results for their approach, and presenting experimental results on both generated and real data. The paper would benefit from just a touch of restructuring to add a conclusion and future work section as well as more details to some of their experimental methods and results.
Summary of paper: This paper presents two versions of their framework for clustering networks: NCGE and NCLM. Each version tackles a slightly different variation of the problem of clustering networks. NCGE is applied when there is node correspondence between the networks being clustered, while NCLM is applied when there isn’t. It is impressive that this paper tackles not one but both of these situations, presenting strong experimental results and connecting their work to the theory of networks.
This paper is superbly written in some sections, which makes the weaker sections almost jarring. For example, sections 2 and 3 do an excellent job of demonstrating how the current work builds on previous work. But then in sharp contrast lines 197-200 rely heavily the reader knowing the cited outside sources to grasp their point. Understandably (due to page lengths) one can not include detailed explanations for everything, but the authors may want to consider pruning in some places to allow themselves more consistency in their explanations.
The last subsection (page 8, lines 299-315) is the most interesting and strongest part of that final section. It would be excellent if this part could be expanded and include more details about these real networks. Also the paper ends abruptly without a conclusion or discussion of potential future work. Perhaps with a bit of restructuring, both could be included.
Section 3.2 was another strong moment of the paper, peppering both interesting results (lines 140-142) and forecasting future parts of the paper (lines 149-150). However, this section might be one to either condense or prune just a bit to gain back a few lines as to add details in other sections.

A few formatting issues, questions, and/or recommendations are below:
Page 1, lines 23,24,26, 35: Citations are out of order.
Page 2, line 74: The presentation of the USVT acronym is inconsistent with the presentation of the NCGE and NCLM acronyms.
Page 3, line 97: Citations are out of order.
Page 4, line 128: Based on the earlier lines, it seems that g(A) = \hat{P} should be g(A_i) = \hat{P_i}
Lines 159-160: In what sense does the parameter J count the effective number of eigenvalues?
Page 5, lines 180-183: It appears that $dn$ is a number, but then in line 182, it seems that $dn()$ is a matrix function. Which is it?
Lines 180-183: The use of three different D’s in this area make it a challenge to sort through what all the D’s mean. Perhaps consider using different letters here.
Line 196: Citations are out of order.
Page 6, line 247: How is clustering accuracy measured?
Page 7, line 259: There appears to be an extra space (or two) before the citation [26]
Lines 266-277: There appears to be a bit of inconsistency in the use of K. For example, in line 266, K is a similarity matrix and in line 268, K is the number of clusters.
Lines 269-270, table 2, and line 284: What are GK3, GK4, and GK5? How are they different?
Line 285: There is a space after eigenvectors before the comma.


Learning Deep Structured Multi-Scale Features using Attention-Gated CRFs for Contour Prediction
Recent works have shown that exploiting multi-scale representations deeply learned via convolutional neural networks (CNN) is of tremendous importance for accurate contour detection. This paper presents a novel approach for predicting contours which advances the state of the art in two fundamental aspects, i.e. multi-scale feature generation and fusion. Different from previous works directly considering multi-scale feature maps obtained from the inner layers of a primary CNN architecture, we introduce a hierarchical deep model which produces more rich and complementary representations. Furthermore, to refine and robustly fuse the representations learned at different scales, the novel Attention-Gated Conditional Random Fields (AG-CRFs) are proposed. The experiments ran on two publicly available datasets (BSDS500 and NYUDv2) demonstrate the effectiveness of the latent AG-CRF model and of the overall hierarchical framework.

This paper proposes a gating mechanism to combine features from different levels in a CNN for the task of contour detection. The paper builds upon recent advances in using graphical models with CNN architectures [5,39] and augments these with attention. This results in large improvements in performance at the task of contour detection, achieving new state-of-the-art results. The paper also presents an ablation study where they analyze the impact of different parts of their architecture.
Cons:
1. Unclear relationship to past works which use CRFs with CNNs [5,39] and other works such as [A,B] which express CRF inference as CNNs. The paper says it is inspired from [5,39] but does not describe the points of difference from [5, 39]. Is only attention the difference and thus the novelty? Does the derivation in Section 2 follow directly from [5,39] (except taking attention into account) or it requires additional tricks and techniques beyond [5,39]? How does the derivation relate to the concepts presented in [A]? Not only will these relationships make it easier to judge novelty, it will also allow readers to contextualize the work better.
2. Intuitively, I dont understand why we need attention to combine the information from different scales. The paper is fairly vague about the motivation for this. It would help if the rebuttal could provide some examples or more intuition about what a gating mechanism can provide that a non-gating mechanism can't provide.
3. The paper presents very thorough experimental results for contour detection, but I will encourage the authors to also study at the very least one other task (say semantic segmentation). This will help evaluate how generalizable the technique is. This will also increase the impact of the paper in the community.
4. The paper is somewhat hard to read. I believe there are some undefined variables, I haven't been able to figure out what f_l^{D,C,M} in figure 2 are.
5. There are minor typos in Table 2 (performance for [14] is misreported, ODS is either 70.25 or 71.03 in [14] but the current paper reports it as 64.1), and these numbers are not consistent with the ones in Figure 4. I would recommend that the authors cross-check all numbers in the draft.
[A] Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Philipp Krähenbühl and Vladlen Koltun
NIPS 2011
[B] Conditional Random Fields as Recurrent Neural Networks. ICCV 2015


Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Background: Deep learning models are typically trained using stochastic gradient descent or one of its variants. These methods update the weights using their gradient, estimated from a small fraction of the training data. It has been observed that when using large batch sizes there is a persistent degradation in generalization performance -known as the "generalization gap" phenomenon. Identifying the origin of this gap and closing it had remained an open problem. Contributions: We examine the initial high learning rate training phase. We find that the weight distance from its initialization grows logarithmically with the number of weight updates. We therefore propose a "random walk on a random landscape" statistical model which is known to exhibit similar "ultra-slow" diffusion behavior. Following this hypothesis we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used. We further investigate different techniques to train models in the large-batch regime and present a novel algorithm named "Ghost Batch Normalization" which enables significant decrease in the generalization gap without increasing the number of updates. To validate our findings we conduct several additional experiments on MNIST, CIFAR-10, CIFAR-100 and ImageNet. Finally, we reassess common practices and beliefs concerning training of deep models and suggest they may not be optimal to achieve good generalization.

Summary of the paper and some comments:
This paper investigates the reasons for generalization gap, namely the detrimental effect of large mini-batches on the generalization performance of the neural networks. The paper argues that the reason why the generalization performance is worse when the minibatch size is smaller is due to the less number of steps that the model is trained for. They also discuss the relationship between their method and recent works on flat-minima for generalization. The main message of this paper is that there is no fundamental issue with SGD using large minibatches, and it is possible to make SGD + large minibatches to generalize well as well. The authors propose three different methods to overcome these issues,
1) Ghost BN: They have proposed to use smaller minibatch sizes during the training to compute the population statistics such as mean and the standard deviation. During the test time, they use the full-batch statistics. I think computing the population statistics over the small minibatches, probably have some sort of regularization effect coming from the noise in the estimates. I feel like this is not very well justified in the paper.
2) Learning rate tuning, they propose to rescale the learning rate of the large mini-batches proportionally to the size of the minibatch. This intuitively makes sense, but authors should discuss more and justify it in a better way.
3) Regime adaptation, namely trying to use the same number of iterations as the model being trained. This is justified in the first few sections of the paper.
Firstly, the paper at some points is very verbose and feel like uses the valuable space to explain very simple concepts, e.g. the beginning of section 3 and lacks to provide enough justifications for their approach.
The paper discusses mainly the generalization of SGD with large minibatches, but there is also an aspect of convergence of SGD as well. Because the convergence results of SGD also depends on the minibatch size as well, please see [1]. I would like to see a discussion of this as well.
In my opinion, anyone that spend on monitoring the learning process of the neural networks would notice easily notice that the learning curves usually have a logarithmic behavior.  The relationship to the random-walk theory is interesting, but the relationship that is established in this theory is very weak, it is only obtained by approximating an approximation.
Equation 4 is interesting, for instance, if we initialize w_0 to be all zero, it implies that the norm of the w_t will grow logarithmically with respect to the number of iterations. This is also an interesting property, in my opinion.
Experiments are very difficult to interpret, in particular when those plots are printed on the paper.
The notation is a bit inconsistent in throughout the paper, some parts of the paper use bracket for the expectation and some other parts don't. Authors should explicitly state what the expectation is taken over.
In Figure 3, can you also consider adding the large minibatch cases without regime adaptation for comparison?
It would be interesting to see the learning curves of the models with respect to different wallclock time as well.
I can not see the AB in Table 1 and 2. The results and the improvements are marginal, sometimes it hurts.  It seems like largest improvement comes from RA.
The paper could be better written. I think the paragraph sections in the abstract are unnecessary. The section 7 could be renamed as "Discussion and Conclusion" and could be simplified.
[1] Li, Mu, et al. "Efficient mini-batch training for stochastic optimization." Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2014.


Trimmed Density Ratio Estimation
Density ratio estimation is a vital tool in both machine learning and statistical community. However, due to the unbounded nature of density ratio, the estimation procedure can be vulnerable to corrupted data points, which often pushes the estimated ratio toward infinity. In this paper, we present a robust estimator which automatically identifies and trims outliers. The proposed estimator has a convex formulation, and the global optimum can be obtained via subgradient descent. We analyze the parameter estimation error of this estimator under high-dimensional settings. Experiments are conducted to verify the effectiveness of the estimator.
high density region of p(x), we end up with a very different density ratio function and the original parametric pattern in the ratio is ruined. We give such an example in Section 6.
Volatile Samples The change of external environment may be responded in unpredictable ways. It is possible that a small portion of samples react more "aggressively" to the change than the others. These samples may be skewed and show very high density ratios, even if the change of distribution is relatively mild when these volatile samples are excluded. For example, when testing a new fertilizer, a small number of plants may fail to adapt, even if the vast majority of crops are healthy.
Overly large density ratio values can cause further troubles when the ratio is used to weight samples. For example, in the domain adaptation setting, we may reweight samples from one task and reuse them in another task. Density ratio is a natural choice of such "importance weighting" scheme [28,25]. However, if one or a few samples have extremely high ratio, after renormalizing, other samples will have almost zero weights and have little impact to the learning task.
Several methods have been proposed to solve this problem. The relative density ratio estimation [33] estimates a "biased" version of density ratio controlled by a mixture parameter α. The relative density ratio is always upper-bounded by 1 α , which can give a more robust estimator. However, it is not clear how to de-bias such an estimator to recover the true density ratio function. [26] took a more direct approach. It estimates a thresholded density ratio by setting up a tolerance t to the density ratio value. All likelihood ratio values bigger than t will be clipped to t. The estimator was derived from Fenchel duality for f -divergence [18]. However, the optimization for the estimator is not convex if one uses log-linear models. The formulation also relies on the non-parametric approximation of the density ratio function (or the log ratio function) making the learned model hard to interpret. Moreover, there is no intuitive way to directly control the proportion of ratios that are thresholded. Nonetheless, the concept studied in our paper is inspired by this pioneering work.
In this paper, we propose a novel method based on a "trimmed Maximum Likelihood Estimator" [17,10]. This idea relies on a specific type of density ratio estimator (called log-linear KLIEP) [30] which can be written as a maximum likelihood formulation. We simply "ignore" samples that make the empirical likelihood take exceedingly large values. The trimmed density ratio estimator can be formulated as a convex optimization and translated into a weighted M-estimator. This helps us develop a simple subgradient-based algorithm that is guaranteed to reach the global optimum.
Moreover, we shall prove that in addition to recovering the correct density ratio under the outlier setting, the estimator can also obtain a "corrected" density ratio function under a truncation setting. It ignores "pathological" samples and recovers density ratio only using "healthy" samples.
Although trimming will usually result a more robust estimate of the density ratio function, we also point out that it should not be abused. For example, in the tasks of two-sample test, a diverging density ratio might indicate interesting structural differences between two distributions.
In Section 2, we explain some preliminaries on trimmed maximum likelihood estimator. In Section 3, we introduce a trimmed DRE. We solve it using a convex formulation whose optimization procedure is explained in Section 4. In Section 5, we prove the estimation error upper-bound with respect to a sparsity inducing regularizer. Finally, experimental results are shown in Section 6 and we conclude our work in Section 7.

Trimmed Density Ratio Estimation
This paper proposes a convex method for robust density ratio estimation, and provides several theoretical and empirical results. I found the method to be well motivated, and of course Figure 3 is quite fun. Based on the evidence in the paper, I would feel comfortable recommending the method to others.
I would be happy to see these results published; however, I found the presentation to be fairly terse, and thought the paper was very difficult to follow. My main recommendation for the writing would be to focus more on core contributions, and to remove some side remarks to allow for longer discussions.
Moreover, the paper was missing some references.
Noting these issues, my current recommendation for the paper is a middle-of-the-road “accept”; however, if the authors are willing to thoroughly revise their paper, I would be open to considering a higher rating. Below are some specific comments.
Comments on motivation + literature:
- The paper opens the abstract by describing density ratio estimation (DRE) as a “versatile tool” in machine learning. I think this substantially undersells the problem: DRE is a a core statistical and machine learning task, and any advance to it should be of broad interest. I would recommend stressing the many applications of DRE more.
- On a related note, DRE has also received quite a bit of attention in the statistics literature. For example, Efron and Tibshirani (1996) use it to calibrate kernel estimators, and Fithian and Wager (2016) use DRE to stabilize heavy-tailed AB-tests. Fokianos (2004) models the density ratio to fit several densities at once. This literature should also be discussed.
- Are there some applications where trimming is better justified than others? Engineering applications relating to GANs or anomaly detection seem like very natural places to apply trimming; other problems, like two-sample testing, may be more of a stretch. (In applications like anomaly detection, it’s expected to have regions where the density ratio diverges, so regularizing against them is natural. On the other hand, if one finds a diverging density ratio in two-sample testing, that’s very interesting and shouldn’t be muted out.)
Comments on relationship to the one-class SVM:
- I found Section 2 on the one-class SVM to be very distracting. It’s too short to help anyone who isn’t already familiar with the method, and badly cuts the flow of the paper. I think a version of the paper that simply removed Section 2 would be much nicer to read. (The discussion of the one-class SVM is interesting, but belongs in the discussion section of a long journal paper, not on page 2 of an 8-page NIPS paper.)
- The use of the one-SVM in the experiments was interesting. However, I think the median reader would benefit from a lot more discussion. The point is that the proposed method gets to leverage the “background” images, whereas one-SVM doesn’t. So even though one-SVM is a good method, you bring more data to the table, so there’s no way one-SVM could win. (I know that the current draft says this, but it says it so briefly that I doubt anyone would notice it if they were reading at a regular speed.)
- As a concrete recommendation: If the authors were to omit Section 2 and devote the resulting space to adding a paragraph or two to the intro that reviews and motivates the DRE literature to the intro (including the stats literature) and to thoroughly contrasting their results with the one-SVM in the results section, I think the paper would be much easier to read.
Other comments:
- In the setting of 3.1, Fithian and Wager (2016) show that if we assume that one class has much more data than the other, then density ratio models can be fit via logistic regression. Can the authors comment on how the proposed type of trimming can be interpreted in the logistic regression case?
- Section 3.2 is awkwardly placed, and cuts the flow. I recommend moving this material to the intro.
- I wasn’t sold by the connection to the SVM on lines 120-129. The connection seems rather heuristic to me (maybe even cosmetic?); I personally would omit it. The core contributions of the paper are strong enough that there’s no need to fish for tentative connections.
- The most classical approach to convex robust estimation is via Huberization (e.g., Huber, 1972). What’s recommended here seems very different to me —- I think readers may benefit from a discussion of how they’re different.
- The approach of assuming a high-dimensional linear model for the theory and and RBF kernel for the experiments is fine. Highlighting this choice more prominently might help some readers, though.
References:
Efron, Bradley, and Robert Tibshirani. "Using specially designed exponential families for density estimation." The Annals of Statistics 24.6 (1996): 2431-2461.
Fithian, William, and Stefan Wager. "Semiparametric exponential families for heavy-tailed data." Biometrika 102.2 (2014): 486-493.
Fokianos, Konstantinos. "Merging information for semiparametric density estimation." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 66.4 (2004): 941-958.
Huber, Peter J. “Robust statistics: A review." The Annals of Mathematical Statistics (1972): 1041-1067.
##########
UPDATE
In response to the authors’ comprehensive reply, I have increased my rating to an “8”.


Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet [1], a deep residual network [2, 3] and a generative adversarial network [4], using a simulator implemented with the neon deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.

I find the proposed technique interesting, but I believe that these problem might be better situated for a hardware venue. This seems to be more a contribution on hardware representations, as compared to previous techniques like xor-net which don't ask for a new hardware representation and had in mind an application to compression of networks for phones, and evaluated degradation of final performance.  However, I do think that the paper could be of interest to the NIPS community as similar papers have been published in AI conferences.
The paper is well written and easy to understand.  The upsides of the method and the potential problems are well described and easy to follow. Improvements over previous techniques seem somewhat incremental, and for many applications it's not clear that training with a low precision format is better than training with a high precision format and then cropping down, and no experiments are performed to explore that.
My opinion on the evaluation is mixed.  For one, no experiments are done to compare to previous (crop to low precision after training) techniques.  Second, plotting validation error seems natural, but I would also like to see test numbers.  I notice that the validation result for Alexnet on ILSVRC is about 20% error, while alexnet originally reported 17% error on ILSVRC-11 and 16.4% on ILSVR-2012 test.  Allowing for direct comparison would improve the paper.  The results for CIFAR (~5% error) seem to be near state of the art.  Qualitative GAN results appear to be good.
Further Comments:
After discussion, I believe that the paper could be accepted (increased to a 7 from a 6).  Other reviewers have argued that the authors need to provide evidence that such a system would be implementable in hardware, be faster, and have a lower memory footprint.  I do not believe that those experiments are necessary.  Given the large number of network architectures that people are using currently, which ones should the authors demonstrate on?  If one architecture causes problems, is the paper unacceptable?  I am also somewhat confused that other reviews seem to give no value to drastically reducing the memory footprint (by easy calculation each layer is represented in half as much memory).  Many of the current state of the art network architectures use the maximum GPU memory during training.  While it needs more work, this technique seems to provide a no-loss answer to how to increase network capacity which is very cool indeed
This is also related to my original point about venue, as it is this paper sits somewhere between AI and systems.  If the authors performs further "hardware-centric" experiments it seems the paper would clearly be better suited for a hardware or systems venue.  If we evaluate this work as a network compression paper then the classification accuracy demonstrates success.  A network trained in half as much memory is as accurate as the full memory network.  It deserves consideration by the ACs for acceptance.


Minimizing a Submodular Function from Samples
In this paper we consider the problem of minimizing a submodular function from training data. Submodular functions can be efficiently minimized and are consequently heavily applied in machine learning. There are many cases, however, in which we do not know the function we aim to optimize, but rather have access to training data that is used to learn it. In this paper we consider the question of whether submodular functions can be minimized when given access to its training data. We show that even learnable submodular functions cannot be minimized within any non-trivial approximation when given access to polynomially-many samples. Specifically, we show that there is a class of submodular functions with range in [0, 1] such that, despite being PAC-learnable and minimizable in polynomial-time, no algorithm can obtain an approximation strictly better than 1/2 o(1) using polynomially-many samples drawn from any distribution. Furthermore, we show that this bound is tight via a trivial algorithm that obtains an approximation of 1/2.

Informally stated, the paper states the following.
There exists a family F of submodular functions valued in [0, 1], so that if after polynomially many samples any algorithm estimates some set S as the minimizer, there exists some f in F that is consistent with the samples and on which S is suboptimal optimum by at least 1/2.
The argument is based on the probabilistic method, i.e., by showing that such functions exist with non-vanishing probability.
Moreover, the class that is considered is shown to be PAC-learnable, effectively showing that for submodular functions PAC-learnability does not imply "minimizability" from samples.
However, one has to be careful about this argument.
Namely, PAC learnability guarantees low error only on samples coming from the training distribution, while to minimize the function it is necessary to approximate it well near the optimum (where we might lack samples) up to an additive factor.
Thus, it is not very surprising to me that the former does not imply the latter and that one can construct function classes to fool any distribution.
This is a purely theoretical result that both answers an interesting question, and also provides with a construction of intuitive class of submodular functions which are (as proven) hard to minimize from samples.
I only read the (sketched) proofs in the main text, which seem to be very rigorously executed.
The paper is very dense, which I do not think can be easily improved on, but it is nicely written.
Based on this grounds, I would recommend this paper for acceptance.
I think it would be more interesting if the authors present some natural settings where the learning setting they consider is used in practice.
In other words, when does learn a submodular functions from training data of the form (S_i, F(S_i))?
This setting is different from how submodular functions are learned in the vision domain (where one might argue is the primary application field of submodular minimization).
There, one is given a set of instances, which we want to have a low energy, either using a min-margin approach, or via maximum likelihood.
Questions / Remarks:
1. The argument l129-132 is unclear to me. Do you want to say that as n -> oo both P_{f\in F}(f(S) >= 1/2) and |F'|/|F| go to one, there has to exist an f in |F'| satisfying f(S) >= 1/2 - o(1)?
2. In prop. 2 you do not define F_A(B), which I guess it means F(A \cup B) - F(B), which is confusing as it is typically defined the other way round. You could also use F(A | B), which is I guess the more commonly used notation.
3. I think the discussion after prop 2 and before section 3 should be moved somewhere else, as it does not following from the proposition itself.
4. Maybe add a note that U(A) means drawn uniformly at random from the set A.
5. l212: It is not motivated why you need one predictor for each possible masking element.
6. When you minimize a symmetric submodular function it is to minimize it over all sets different from the empty set and the ground set. This is exactly what is done using Queyranne's algorithm in cubic time.
7. Shouldn't the 1 be 1/2 in the last equation of the statement of lemma 4?
8. I find the notation in the two inequalities in Lemma 5 (after "we get") confusing. Does it mean that it holds for any h(n) = o(1) function with the given probability? Also, maybe add an "and" after the comma there.
9. l98: a word is missing after "desirable optimization"


Reinforcement Learning under Model Mismatch
We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs of [1,15,11] to the model-free Reinforcement Learning setting, where we do not have access to the model parameters, but can only sample states from it. We define robust versions of Q-learning, SARSA, and TD-learning and prove convergence to an approximately optimal robust policy and approximate value function respectively. We scale up the robust algorithms to large MDPs via function approximation and prove convergence under two different settings. We prove convergence of robust approximate policy iteration and robust approximate value iteration for linear architectures (under mild assumptions). We also define a robust loss function, the mean squared robust projected Bellman error and give stochastic gradient descent algorithms that are guaranteed to converge to a local minimum.

Overview:
The authors propose an extension of Robust RL methods to model-free applications. Common algorithms (SARSA, Q-learning, TD(lambda)) are extended to their robust, model-free version.
The central point of the paper is replacing the true and unknown transition probabilities  P_i^a with a known confidence region U_i^a centered on an unknown probability p_i^a. Transitions according to p_i^a are sampled via a simulator. The robustness of the algorithms derive then from an additional optimization step in U_i^a. The authors clarify that, since the addition of term U to p must not violate the probability simplex. However, since p_i^a is unknown, the additional optimization is actually performed on a term U_hat that ignores the above restriction. This mismatch between U and U_hat, represented by a term beta, determines whether the algorithms converge towards to an epsilon-optimal policy. The value of epsilon is also function of beta.
The authors propose also an extension of RL with function approximation architectures, both linear and nonlinear. In the linear case, the authors show that a specific operator T is a contraction mapping under an opportune weighted Euclidean norm. By adopting the steady state distribution of the exploration policy as weights, and by adopting an assumption from previous literature concerning a constant alfa < 1, the authors prove that, under conditions on alfa and beta, the robust operator T_hat is a contraction, and that the solution converges close to the true solution v_pi, where the closeness depends again on alfa and beta.
Furthermore, the authors extend the results of Ref.6 to the robust case for smooth nonlinear approximation architectures. An additional assumption here is to assume a constant confidence bound U_i^a = U for all state-action pairs for the sake of runtime. The authors extend on the work of Ref.6 by introducing the robust extension of the mean squared projected bellman equation, the MSRPBE, and of the GTD2 and TDC update algorithms, which converge to a local optima given assumptions on step lengths and on confidence region U_hat.
In conclusion, the authors apply both nominal and robust Q-learning, SARSA and TD-learning to several OpenAI gym experiments. In these experiments, a small probability of transitioning to a random state is added to the original models during training. The results show that the best robust Q-learning outperforms nominal Q-learning. However, it is shown that the performance of the robust algorithms is not monotone with p: after increasing for small values of p, the advantage of the robust implementation decreases as U and U_hat increase in discrepancy.
Evaluation:
- Quality: the paper appears to be mathematically sound and assumptions are clearly stated. However, the experimental results are briefly commented. Figures are partial: of the three methods compared, Q-learning, SARSA and TD-learning, only results from Q-learning are briefly presented. The algorithms of section 4 (with function approximation) are not accompanied by experimental results, but due to the theoretical analysis preceding them this is of relative importance.
- Clarity: the paper is easy to read, with the exception of a few symbolic mismatches that are easy to identify and rectify. The structure of the paper is acceptable. However, the overall clarity of the paper would benefit from extending section 4.2 and 5, possibly by reducing some portions of section 3.
-  Originality: the model-free extensions proposed are an original contribution with respect to existing literature and to NIPS in particular.
- Significance: the motivation of the paper is clear, as is the contribution to the existing theoretical research. Less clear is the kind of application this work is intended for. In particular, the paper assumes the existence of a simulator with unknown transition probabilities, but known confidence bounds to the actual model being simulated, to obtain samples from. While this assumption is not by itself wrong, it is not clearly mentioned in the abstraction or in the introduction and takes the risk of appearing as artificial.
Other comments:
- Line 194: the definition of beta given in this line is the one Lemma 4.2 rather than the one of Lemma 3.2;
- Line 207: here and in the following, nu is used instead of theta to indicate the discount factor. Please make sure the symbology is consistent;
- Line 268: the sentence "the state are sampled with from an" might contain a typo;
- Eq. 7-8: there appears to be an additional minus signs in both equations.