Neural Architecture Optimization
Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple and efficient method to automatic neural architecture design based on continuous optimization. We call this new approach neural architecture optimization (NAO). There are three key components in our proposed approach: (1) An encoder embeds/maps neural network architectures into a continuous space. (2) A predictor takes the continuous representation of a network as input and predicts its accuracy. (3) A decoder maps a continuous representation of a network back to its architecture. The performance predictor and the encoder enable us to perform gradient based optimization in the continuous space to find the embedding of a new architecture with potentially better accuracy. Such a better embedding is then decoded to a network by the decoder. Experiments show that the architecture discovered by our method is very competitive for image classification task on CIFAR-10 and language modeling task on PTB, outperforming or on par with the best results of previous architecture search methods with a significantly reduction of computational resources. Specifically we obtain 2.11% test set error rate for CIFAR-10 image classification task and 56.0 test set perplexity of PTB language modeling task. The best discovered architectures on both tasks are successfully transferred to other tasks such as CIFAR-100 and WikiText-2. Furthermore, combined with the recent proposed weight sharing mechanism, we discover powerful architecture on CIFAR-10 (with error rate 3.53%) and on PTB (with test set perplexity 56.6), with very limited computational resources (less than 10 GPU hours) for both tasks.

>>>>>>>>>>>>>>>>> After the discussion phase  Based on the discussion with the area chair assigned to this paper, I would like to comment on the reproducibility of the paper. In fact, I doubt that this paper is easily reproducible (which also had an influence on the score I gave to this paper). I believe (given my own experience with the reproduction of the original NAS-RL by Zoph et al and supported by Footnote 3 of the authors in the paper at hand) that too many factors on different levels are required to reproduce the results: architecture options in the target network to be optimized, hyperparameters to train the target network, architectures of the encoder, performance predictor and decoder, hyperparameters to train the encoder, performance predictor and decoder, meta-hyperparameters of Algorithm 1 and the loss function (\lambda_i). Only missing one little detail can substantially change the results (because DL is very sensitive too many hyperparameter decisions).  Because we are all humans, we tend to underestimate certain aspects and forget little, but important details to report in papers. That was again shown in this paper, since I had to ask how the authors optimized the hyperparameters of the target network. (I assume that the authors only forgot this important step and they didn’t leave it out on purpose.) Therefore, to the best of my knowledge, open source code is the best way right now to ensure reproducibility.  Another aspect is that I (as a reviewer) are not able to check whether all these important, little details are reported in the paper. If source code would be available (already during the review process), I could be sure that at least the source code covers (nearly) everything. (A docker image would be even better because the dependencies would also be listed, which can also influence the results.)  I would like to apologize in advance if I’m too sceptical here; but my experience taught me in the last years that the promise of “We will release all our codes/models after the paper is published.” is not sufficient. A discussion at the AutoML workshop@ICML’18 even supported me in my believe. Therefore, I cannot give credits to the authors because of the promise in the response letter.   >>>>>>>>>>>>>>>>> Response to response letter  Thank you for your thorough response to the reviews.  I really appreciate that you ran more experiments. Unfortunately, because of the space restrictions in the response letter, many details about the experimental setup are missing and I cannot really assess the value of these new results. Nevertheless, the results sound very promising.  Finally, I would like to disagree with some of the raised points in the response letter:  > We’d like to respectfully point out that it is neither appropriate nor necessary to directly compare NAO with ENAS.  Given that we are talking about a NIPS paper, I don't think that I'm asking for too much: a fair comparison between NAO and ENAS. I agree that both approaches are working on different aspects of the underlying problem. Nevertheless, I had the feeling that the paper has not honestly discussed how these approaches could relate and what would happen if NAO and ENAS are combined.  > Third, the EE trade-off is naturally achieved via the progress of P: initially P is inaccurate and will lead to worse architectures which plays the effect of exploration.  If that would be true, you could also argue that all the effort for exploration-exploitation trade-off in Bayesian Optimization is not necessary, because in BO, we also start with an inaccurate model. However, as shown in the BO literature, the EE-trade-off (achieved by an acquisition function) is important.   > we believe there is no need to explicitly report the result of random search  Depending on how you design your search space, random search can perform quite well. Therefore, it is an important baseline to really show that you approach can do something better.  > he performance growing w.r.t. NAO iteration (in Fig. 2(b)) cannot be achieved via random search  How do you know?  > Our method is simple grid search   I strongly wonder why people are still using grid search these days after there is so much literature on proper HPO.  > we only fine-tune  My point was that you have not reported HPO in a sufficient manner. For such an expensive approach as NAO, it is crucial how expensive HPO is, how to do HPO in this setting and last but not least, how sensitive your method is wrt to its own hyperparameters.  PS: Little details were changed in the original review.  >>>>>>>>>>>>>>>>> Original Review:  The authors address the challenge of neural architecture search. Since manual design of neural network architectures is a tedious task and requires expert knowledge, automatic neural architecture search was proposed to help new users to easily apply deep learning to their problems. The authors of the paper at hand propose an approach dubbed neural architecture *optimization* (NAO).  They call it "optimization" instead of "search" because  they optimize the architecture in a continuous space and they do not search in a discrete space. To this end, they propose to (i) learn an encoder to map architectures into a continuous space, (ii) predict the performance of architectures in the continuous space by also using a neural network and (iii) use a decoder from the continuous space into an architecture  such that they can evaluate an architecture which was predicted to have a good performance. In the end, they show that this proposed approach finds well performing architectures for CIFAR-10  and Penn Tree bank (PTB).  Most previous approaches (which are based on reinforcement learning (RL) and  evolutionary strategies (ES)) use indeed a discretized space, but the proposed new approach uses a learned continuous space to search for a good architecture. On the one hand, it is indeed an interesting paper because it proposes a completely new approach. On the other hand, the question arises whether this approach is better performing than already existing approaches or whether we can learn something interesting about the problem. Unfortunately in my opinion, the answer to both questions is no. The authors report that they found well-performing network architectures which perform similarly compared to previous approaches. However, the cost to find these networks is substantially larger than before. For example, ENAS uses 0.45 GPU days to search for a network. The proposed NAO used 800 GPU days on CIFAR-10 and 500 GPU days on PTB. So, it is not feasible in practice. Already the initialization phase of NAO is extremely expensive; the authors evaluated 600 architectures out of the 1000 networks evaluated overall only in the initial phase. Thus, the initialization phase roughly costs 480 GPU days on CIFRA-10; so roughly 1000 times the time for the entire ENAS search.  Besides the problem of the experimental results, I would like to raise several other issues:  1. The overall approaches strongly reminds me of Bayesian Optimization (BO).  The authors claim that BO is something different but they are doing something very similar:   a. based on some performance observations, they build a model   b. based on the model, they choose n architectures   c. they evaluate these architectures The main difference between BO and NAO is in fact that the authors do not use an acquisition function to trade off exploration and exploitation. However, I wonder why the authors do not have to consider this exploration-exploitation dilemma; maybe because they doing the crazy expensive initialization phase with 600 randomly sampled architectures?  2. The argument of the authors is that optimization in the discrete space is too expensive (although other researchers showed some rather convincing results) and does not scale with the number of architecture hyperparameters. However, I wonder why a mapping into a space with much more dimensions (T x d) should be more efficient. The authors do not make this point very prominent but I guess the reason could be that they can use gradient methods in the continuous space. Nevertheless, a thorough experimental proof of this point is missing. They do not even show that NAO performs better than random search with this tremendous amount of GPU days.  3. Another big issue is the handling of hyperparameter tuning, i.e., the tuning of the hyperparameter of their own NAO and all hyperparameters of the neural networks (e.g., learning rate). The authors wrote: "we have not conducted extensive hyper-parameter exploration as is typically done in previous works". So, the authors have conducted some hyperparameter optimization but they do not tell us how much exactly, how expensive it was and how the hyperparameter tuning influenced the results. Given that they changed the hyperparameters settings between CIFAR-10 and PTB (for example the trade-off parameters of their loss function), I guess that these hyperparameters are indeed crucial and NAO is not robust wrt these.  4. From the perspective of optimization (/search), the number of exemplary problems (here CIFAR-10 and PTB) is far too small. There is only little substantial evidence that the proposed approach performs well and robustly also on future problem instances. Given the advances in other communities (i.e., evaluation on tens of problems), I recommend to evaluate NAO on more datasets to obtain better empirical evidence.  I'm well aware that issues 3 and 4 also apply to many other architecture search papers. Nevertheless, these are also major flaws in these papers and it is not an sufficient argument to accept the paper at hand.  Further remarks and questions: 1. Introduction  * You can also optimize in a discrete space, e.g., the MaxSAT problem is a typical discrete optimization problem. Thus, NAS should be called NAO in the first place. And it is wrong to say that optimization in a discrete space is called "search".  * "the continuous representation of an architecture is more compact" I was missing some empirical evidence for this claim. In particular, I wonder why it should be more compact if the number of dimensions in the continuous space is larger than in the discrete space. 2. Related Work  * "Using BO, an architecture’s performance is modeled as sample from a Gaussian process (GP)" Although GPs are quite common in the BO community, approaches, such as Hyperopt and SMAC, use different models. 3. Approach  * "the outputs of all the nodes that are not used by any other nodes are concatenated to form the final output of the cell". At least in the ENAS paper, the output is not concatenated but averaged  * Why do you need all hidden states for the encoding of the continuous? Why is the last state not sufficient?  * "where X is a set containing several manually constructed architecture pairs (x 1 , x 2 )" The argument against BO was the manual design of the kernel. However, also NAO requires some manual engineering.  * Algorithm 1: How expensive is each step? Or asked differently: How large is the overhead for the embedding, the model training, the surrogate optimization and the decoding? 4. Experiments  * "For the purpose of regularization, We apply" --> we  * "The detailed results are shown in Table 4.1" -> "The detailed results are shown in Table 1" 5. Conclusion  * "makes it more effective and efficient to discover better architectures" Why is it more efficient if NAO requires much more compute resources compared to ENAS?  Last but not least, since no link to an open-source implementation is provided (or even promised) and (as the authors experienced themselves) the reproducibility in the field of NAS is rather hard, I would guess that the reproducibility of the paper at hand is also rather poor.


Breaking the Span Assumption Yields Fast Finite-Sum Minimization *
In this paper, we show that SVRG and SARAH can be modified to be fundamentally faster than all of the other standard algorithms that minimize the sum of n smooth functions, such as SAGA, SAG, SDCA, and SDCA without duality. Most finite sum algorithms follow what we call the "span assumption": Their updates are in the span of a sequence of component gradients chosen in a random IID fashion. In the big data regime, where the condition number κ = O(n), the span assumption prevents algorithms from converging to an approximate solution of accuracy in less than n ln(1/ ) iterations. SVRG and SARAH do not follow the span assumption since they are updated with a hybrid of full-gradient and component-gradient information. We show that because of this, they can be up to Ω(1 + (ln(n/κ)) + ) times faster. In particular, to obtain an accuracy = 1/n α for κ = n β and α, β ∈ (0, 1), modified SVRG requires O(n) iterations, whereas algorithms that follow the span assumption require O(n ln(n)) iterations. Moreover, we present lower bound results that show this speedup is optimal, and provide analysis to help explain why this speedup exists. With the understanding that the span assumption is a point of weakness of finite sum algorithms, future work may purposefully exploit this to yield faster algorithms in the big data regime.

Summary:  The authors consider the finite-sum minimization problem.   The work aims to investigate the “span condition” of modern variance reduced methods, and the role it plays in the complexity analysis of modern variance reduced methods. The span condition is satisfied for some method if the k-th iterate is formed as the starting iterate plus a linear combination of the stochastic gradients computed by the method. This condition is satisfied for many methods, including SAG, SDCA, SAGA, Finito and MISO. However, it is not satisfied for methods such as SVRG and SARAH.   The authors ask the question whether the lack of the span condition in methods such as SVRG and SARAH might lead to better complexity estimates, both in terms of lower and upper bounds. They give positive answer to this question.  In all results the authors assume all functions f_i to be smooth (with parameter L_i), and the average to be strongly convex (with parameter \mu). Specialized results are then proved in the case when the functions f_i are convex (CASE 1), and when they are not convex (CASE 2).  Clarity:   The paper is well written; the ideas are clearly introduced, motivated and explained.  Quality/Originality/Significance:  One of the key results of this paper (Theorem 2) is to show that in the regime when the condition number is not too large when compared to the number of functions (kappa = O(n)), the lower bound proved in [17] can be improved. This is done for CASE 1. The improvement is not very large, which is not surprising. In particular, in one expression, the term “n” is replaced by “n/(1+log(n/kappa))”. However, the relative improvement grows as n/kappa grows, and is clearly unbounded. The new lower bound is obtain by a modification/tightening of the bound obtained in [17]. Because of this, the tools used to obtain this result are not highly original. However, the result itself is quite interesting.  The second key result (Theorem 1/Corollary 2) is a modified analysis of SVRG (used with a different stepsize) which matches the newly established lower bound (again, for CASE 1). The result itself can be derived from the original SVRG analysis by a different choice of the parameters of the method (stepsize, number of steps in the inner loop, number of outer loops). Hence, the result can be seen as a rather elementary observation. Still, it is interesting that this observation was not explicitly made before.   Can you give examples of important problems or reasonable situations when one might expect n/kappa to be very large? This would strengthen the importance of the regime analyzed in this paper.  The paper does not contain any experiments. I view this as a major issue and this omission weakens the paper considerably. I’d like to see the difference, through experiments, offered by the logarithmic speedup. It should be possible to test this and clearly illustrate the benefits as n/kappa grows, even if this is done on a carefully constructed synthetic problems. Various types of experiments can be done and should have been done.     The authors provide a compelling explanation of why the “hybrid methods“ (i.e., methods such as SVRG that do not follow the span assumption) allow for this faster rate. This is done in Section 3.   Additional contributions:  a) Proposition 1: Provides a lower bound for SDCA (Omega(n log(1/eps))); which means that SDCA can not attain logarithmic speedup. Can SDCA be modified to allow for logarithmic speedup?  b) Section 4: Provides logarithmic speedup for SVRG in CASE 2. However, I do not see a lower bound provided for this case. Can you establish a lower bound?  Small issues:  1) Line 139: Missing right parenthesis 2) Line 139: “can can” -> “can” 3) Line 168: “X^*” -> “x^*” 4) Footnote 7: “sequence” -> “sequences” 5) Line 303: “shwartz” -> “Shwartz”  Comment: My answer to Q4 should be ignored (it's a dummy response; one should really have a NA option) since there are no computational results to be reproduced.   -----  POST AUTHOR FEEDBACK:  I've read the rebuttal, and the other reviews. I am keeping the same score.


A Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents
In multiagent domains, coping with non-stationary agents that change behaviors from time to time is a challenging problem, where an agent is usually required to be able to quickly detect the other agent's policy during online interaction, and then adapt its own policy accordingly. This paper studies efficient policy detecting and reusing techniques when playing against non-stationary agents in Markov games. We propose a new deep BPR+ algorithm by extending the recent BPR+ algorithm with a neural network as the value-function approximator. To detect policy accurately, we propose the rectified belief model taking advantage of the opponent model to infer the other agent's policy from reward signals and its behaviors. Instead of directly storing individual policies as BPR+, we introduce distilled policy network that serves as the policy library in BPR+, using policy distillation to achieve efficient online policy learning and reuse. Deep BPR+ inherits all the advantages of BPR+ and empirically shows better performance in terms of detection accuracy, cumulative rewards and speed of convergence compared to existing algorithms in complex Markov games with raw visual inputs.

The paper describes a new version of BPR+, a Bayesian Policy Reuse algorithm for adapting to opponent/teammate behaviors in multiagent settings. It adds two elements: an opponent model, and a distilled policy network, that help improve some shortcomings of the previous versions.  The paper is well-written, the motivation and problems with the previous algorithms are well-articulated. On a positive note, the experiments do a good job of showing the strengths of each of the new elements of Deep BPR+, in three domains, mixing both cooperative and competitve, testing each component separately. It is clear how much each element contributes in these problems. The experiments against online unknown policies is especially valuable and complementary to the motivation.   As a down side, the novelty is somewhat low because the main algorithms already exist, and the new elements here are straight-forward adaptations of ideas from deep learning architectures that have been used in several other places already. After reading the start of Section 3, I was expecting that adding deep learning would mean the ability to scale to larger or more complex domains. That may be true, but it's not made clear by this paper (nor is it motivated as such).  Minor comments: - "LOLA[4] studies the emergence of Nash equilibrium in iterated prisoners' dilemma". This is not quite true; it found *tit-for-tat* in IPD, which is much more surprising. It found the only mixed Nash equilibria in iterated matching pennies, but in any case this was not necessarily intended and there is no proof that LOLA is trying to find an equilibirum.  Questions: - The construction of the distilled policy network sounds very similar to the architecture behind DRON. Can you clarify the main difference(s)? DRON seems to have a mixture-of-experts component used to decide how to switch.. is this also true for DPN?  - Is the Omniscient policy in the experiments exact? (i.e. computed in the tabular form? If so, how was it computed (value iteration?)  Suggested future work: 1. I would encourage the authors to look into more challenging domains either with partial observability or more than two players where the gains from opponent modeling are more clear. 2. On a similar note, I encourage the authors to relate or combine this approach with recent game-theoretic approaches: the overall process resembles fictitious play and/or the oracles algorithm in [1,2], where the DPN can be interpreted as the average or meta-policy (here it acts as a natural baseline to start from, but in self-play, could also provably converge to an equilibrium?). However, there was no explicit opponent modeling in those works, so Deep BPR+ can enhance these methods while potentially keeping the guarantees.   [1] Heinrich & Silver 2016. Deep Reinforcement Learning from Self-Play in Imperfect-Information Games. https://arxiv.org/abs/1603.01121  [2] Lanctot et al. 2017. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning.  **** Post-rebuttal:   Thank you for the clarifications. I raised my score after the discussion, upon reflecting on the novelty (rectified belief update) and the wording attached to the numbers: I think this could work could be of interest to the growing multiagent learning community at NIPS.


Empirical Risk Minimization in Non-interactive Local Differential Privacy Revisited *
In this paper, we revisit the Empirical Risk Minimization problem in the noninteractive local model of differential privacy. In the case of constant or low dimensions (p n), we first show that if the loss function is (∞, T )-smooth, we can avoid a dependence of the sample complexity, to achieve error α, on the exponential of the dimensionality p with base 1/α (i.e., α −p ), which answers a question in [19]. Our approach is based on polynomial approximation. Then, we propose player-efficient algorithms with 1-bit communication complexity and O(1) computation cost for each player. The error bound is asymptotically the same as the original one. With some additional assumptions, we also give an efficient algorithm for the server. In the case of high dimensions (n p), we show that if the loss function is a convex generalized linear function, the error can be bounded by using the Gaussian width of the constrained set, instead of p, which improves the one in [19].

- Summary:   This paper considers the problem of empirical risk minimization in the non-interactive setting of the local model of differential privacy. In this setting, each user (holding one data point) is required to send a differentially private signal to the server without any prior interaction with the server or other users. Then, the server collects the users' signals and uses them to solve the ERM problem. The most relevant previous work is [19] that shows that any protocol that is based on first (or second) order methods (e.g., gradient descent and other variants) must require sample size \Omega(\alpha^{-p}) if it were to achieve error \alpha (where p is the dimensionality of the parameter space). This reference also gives upper bounds of the same order for non-interactive ERM under Local Differential Privacy (LDP) for the class of Lipschitz loss functions and the class of Lipschitz, convex loss functions.   This paper revisits this problem under some smoothness assumptions on the loss function, and devises new algorithms for this problem based on polynomial approximation techniques. The goal is to improve the previous bounds for non-interactive ERM under LDP for some classes of smooth loss functions.   When the sample size (n) < the dimensionality (p): despite success in showing improved sample complexity bounds, the resulting bounds still exhibit exponential dependence on p. For a sub-class of smooth functions (with bounded derivatives up to some constant order), the resulting bound is still ~ \alpha(-\Omega(p)), albeit with a better exponent than the one in [19] devised for Lipschitz loss functions. For a more restrictive class of smooth functions (with bounded derivative of any order), the resulting bound shows a better dependence on \alpha (namely, \alpha^{-4}), yet it still has from an extra factor that is exponential in p (about 4^{p^2}).   When p > n: the paper gives an algorithm that results in sample complexity bounds that only depend (albeit exponentially) on the Gaussian width of the parameter set and n (when the loss function is a generalized linear function).   The paper also shows improved constructions that has better running time and communication complexity than the ones in [19].   - Strengths:   1- The paper studies an important problem. The work is well-motivated. The introduction, as well as the outline of the paper and the high-level description of the algorithms and technical contributions, are nicely written. However, there are some parts in the technical sections that are not as clear (e.g., undefined notation, missing definitions), but these were not very critical (also these issues are understandable considering the space limitations).  2- The paper gives new algorithms for non-interactive ERM under LDP based on techniques from polynomial approximation. The paper also shows improved constructions in terms of running time and communication complexity based on some ideas from [2]. (however, there are some issues need to be addressed and clarified further (See "Weaknesses" below)).   3- Rigorous and original analyses.   - Weaknesses:   1- Despite the strengths of the paper, the results are not quite impressive. The improvements in the bounds are hardly any useful even for p = 5, e.g., a sample complexity bound that scales with 4^{p^2} \alpha^{-4} is still not meaningful in this case (regardless of the fact that it's better than \alpha^{-2p} for some range of \alpha and p).   2- The paper raises an interesting question in the introduction and uses it to build up part of the motivation for this work, but it does not seem to address at all. The paper notes that in some special cases of loss functions (e.g., squared loss [the linear regression case in [19]] and logistic loss [24]) one can have bounds that are free from this exponential dependence on p in the non-interactive setting, and continues to say "this propels us the following natural questions .... Is it possible to introduce some natural conditions on the loss function that guarantee non-interactive _x000f_-LDP with sample complexity that is not exponential in the dimensionality p?". Although the class of smooth functions considered in this paper contains the two aforementioned examples of loss functions, there is no indication that the derived bounds can ever be free from this exponential dependence on p for those special cases of loss functions.   3- The bounds derived for generalized linear functions in terms of Gaussian width (for p > n case) are also hardly practical. Even if the Gaussian width is as small as log(p), the error bounds still scale roughly as ~ (1/n)^{1/\Omega(n)} (Corollary 3). Comparing this to known non-private nearly dimension-independent risk bounds for generalized linear models, it's not clear if the derived bounds are any useful.   4- There is an unquantified dependence on p in Corollary 2, Theorems 4 and 5 via the parameter D_p. The paper introduces D_p (once in the prelim section) as a "universal constant" that depends only on p. But, I don't see how it's a constant if it depends on p? Can this "constant" be at least upper and lower bounded by some truly universal constants independent of p? I think this dependence needs to be explained further. As stated, it's not clear if this D_p does not add an extra factor to the bound making it worse.   5- Some of the discussion in the paragraph after Corollary 2 is actually not clear. Why is it when h --> \infty, the dependence on the error becomes \alpha^{-2}? Isn't it alpha^{-4}?  6- In the analysis of Algorithm 3, the issue of shared/public randomness required in the setup needs to be further discussed.   ------------------------------- -------------------------------    ****Post Rebuttal Comments:  -----------------------------------  After reading the authors' rebuttal, I am inclined to keep my score as is. The authors agreed with several points made in my review, and they clarified most of the issues I had questions about. I think the work is theoretically interesting and removing the 1/\alpha from the base of the exponential dependence is nice. However, I still question the extent of the practicality of the main results especially with the prohibitive exponential dependence on the square of the dimensions in the final bounds. Also, the paper could not provide any insight for the reason why there are non-interactive schemes for some special cases of smooth losses (e.g, linear regression) that do not suffer from this exponential dependence at all.


Learning from Group Comparisons: Exploiting Higher Order Interactions
We study the problem of learning from group comparisons, with applications in predicting outcomes of sports and online games. Most of the previous works in this area focus on learning individual effects-they assume each player has an underlying score, and the "ability" of the team is modeled by the sum of team members' scores. Therefore, current approaches cannot model deeper interaction between team members: some players perform much better if they play together, while some players perform poorly together. In this paper, we propose a new model that takes the player-interaction effects into consideration. However, under certain circumstances, the total number of individuals can be very large, and number of player interactions grows quadratically, which makes learning intractable. In this case, we propose a latent factor model, and show that the sample complexity of our model is bounded under mild assumptions. Finally, we show that our proposed models have much better prediction power on several E-sports datasets, and furthermore can be used to reveal interesting patterns that cannot be discovered by previous methods.
scores. Through the process, one can also rank a player by his or her score in the model. For example, [13] extends the Bradley-Terry model to the group comparison setting; [12] proposed a TrueSkill algorithm to learn individual scores using a Bayesian model, which has now been used by game companies and sport analysts.
However, a common weakness of previous methods is that they ignore the player-interaction effects. In a team challenge, the players work together and will always influence each other, and this player-interaction effect can significantly alter game results. To make the prediction more accurate, incorporating the player-interaction effects are demanding. On the other hand, people are also interested in the cooperation effects between players. Team coach can pair a better team based on both individual abilities and cooperation abilities; game designers such as Blizzard can use the results to design their heroes. This brings us to the questions we wish to answer in this paper:
• Can incorporating player-interaction effects improve the prediction accuracy of the model?
How to interpret those effects?
• If the total number of players is too large, how can our algorithm scale up and meanwhile maintain good generalization error and efficient computational time?
To answer the first question, we propose a new model that can incorporate pairwise effects, and show that the pairwise effects can be learned when there are not too many players. The player-interaction can not be fully modeled by pairwise effects. This is the first step, and investigating effects with order higher than two is our future work. As for the second question, we propose a latent factor model to describe pairwise interactions between players, and propose an efficient stochastic gradient descent algorithm to solve it. A theoretical bound of the sample complexity is provided under mild conditions.
In the experimental part, we test our model on online game datasets and show that our proposed models have much higher prediction accuracy than previous individual-score based models. For example, in Heroes of the Storm data our new models can get around 80% accuracy, while state-ofthe-art models such as Trueskill can only achieve 60% accuracy.

Summary: This paper develops a model that can capture player-interactions from group comparisons (team-play win/loss info). In an effort to address higher-order interactions with a reasonable size of data set, it then proposes a latent factor model and the sample complexity analysis for the model is done under certain scenarios. Experiments are conducted on real-world on-line game datasets, comparing the win/loss prediction accuracy of the proposed approach to the prior methods such as BTL [12] and Trueskill [11].  Detailed comments:  The paper studies an interesting problem, and investigates the role of player-interactions which has been out of reach in the literature. One noticeable observation found in the paper is that the proposed approach may be able to identify the best team members with good chemistry, as suggested in Table 3. Also it is clearly written. The main concerns of this reviewer lie in (a) the performance of the proposed methods with comparison to the prior works; (b) experiment settings; (c) the novelty of the proposed approach.  A1. (Accuracy performance, Table 1): It seems the accuracy performance of the proposed methods is not much better than prior methods, except for HotS Tournament (H) dataset, in which players are advanced. Wondering if the other datasets reflect the cases in which players are not professional and the team chemistry is not apparent. What if the team chemistry is apparent while players are not professional? It is not clear about the scope of scenarios in which the proposed approach is outstanding – wide or narrow?  A2. (Player-interaction ranking performance, Table 3): Wondering if the BTL and/or Trueskill yield similar top-5 and/or bottom-5 pairs (one can compute the team score as a simple aggregation of the estimated individuals' scores, and use them to rank player-interaction). If not, it would be good to emphasize this, as it reveals that the proposed methods can capture interaction effect which are not explored well in the prior approaches.   A3. (Individual ranking performance): Many literature have focused on the ranking performance for individuals (entire or a few significant, say top-K). Wondering if the proposed approach yields the good performance for individuals as well.   B1. (1st experiment setting – heroes (game characters) are individuals): This setting looks arguable because it does not capture the interaction between heroes and players – some players might be very good when playing with particular heroes. Perhaps this setting makes sense only when all players are professional and so are good enough in dealing with any character – but wondering if the setting is typical.   B2. (2nd experiment setting – players are individuals): Due to the similar reason above, this setting is not quite convincing – it still does not capture the chemistry between players and heroes. I believe a reasonable justification behind the two settings should be provided – otherwise, the performance results  which are based on the settings are shaking.  C. (Novelty of the proposed approach): A series of the proposed approaches seem to rely on Eq. (1) in which input vector x_t is ternary: taking +1/-1 for participating individuals; 0 otherwise. But a similar approach (except the regularization term) was already proposed in the following literature: Menke-Martinez, “A Bradley-Terry artificial neural network model for individual ratings in group competitions”. I understand the proposed latent model is distinct from this, but it may be viewed as a reasonable addition to the basic framework in Menke-Martinez.


Learning SMaLL Predictors
We introduce a new framework for learning in severely resource-constrained settings. Our technique delicately amalgamates the representational richness of multiple linear predictors with the sparsity of Boolean relaxations, and thereby yields classifiers that are compact, interpretable, and accurate. We provide a rigorous formalism of the learning problem, and establish fast convergence of the ensuing algorithm via relaxation to a minimax saddle point objective. We supplement the theoretical foundations of our work with an extensive empirical evaluation.

Summary of paper: The paper addresses the problem of learning in resource constrained setting using multiple linear predictors (max of linear predictors like DNF). The problem of learning k-DNFs (a subset of multiple linear predictord) is known to be hard however to solve this non-convex problem, they use various relaxations to formulate this as a minmax saddle point objective which they solve using standard mirror-prox algorithm. The paper then goes on to show that the approach is fruitful in practice through experiments.  Comments for authors: In the introduction (line 28-31), you seem to sugest that you can solve the problem of k-DNFs however you do not actually give any guarantees for the final learnt hypothesis that your algorithm outputs, please make that clear.  Requiring a different weight vector for each positive example, this seems very costly and potentially avoidable. It would be good to discuss how the clustering affects this, probably show a plot for different cluster sizes vs performance.  Overall comments: Using Multiple linear predictors as the base hypothesis class is a good direction as it is a richer class of functions compared to standard sparse linear regressors (LASSO). Viewing the problem as a minmax saddle point problem is an interesting take and I find the use of the boolean constraint matrix novel. As is, the algorithm is not efficient (dimension of weight vector can be of the order of the number of samples) and they do not give any theoretical guarantees for the hypothesis returned by their algorithm. However, the experimental results do seem to suggest that in practice this approach gives a compact sparse representation with low computational overhead. Overall, the paper is well written, provides a useful framework for solving a common problem and from a practical standpoint gives gains over most standard methods (not always large) while giving a handle on the exact sparsity required in the final hypothesis.


Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
Large amounts of labeled data are typically required to train deep learning models. For many real-world problems, however, acquiring additional data can be expensive or even impossible. We present semi-supervised deep kernel learning (SSDKL), a semi-supervised regression model based on minimizing predictive variance in the posterior regularization framework. SSDKL combines the hierarchical representation learning of neural networks with the probabilistic modeling capabilities of Gaussian processes. By leveraging unlabeled data, we show improvements on a diverse set of real-world regression tasks over supervised deep kernel learning and semi-supervised methods such as VAT and mean teacher adapted for regression.

The paper paper applies Deep Kernel Learning [DKL, 1] to Semi-Supervised Regression. DKL is a combination of a Gaussian Process and a Deep Neural Network (DNN). The idea is to use DNN as a feature transformer inside the kernel of a Gaussian Process (GP). In other words, the the GP operates on the outputs of the DNN. Both the GP and the DNN can be trained using SGD in end-to-end fashion. This paper proposes to apply DKL to semi-supervised regression by minimizing the posterior predictive variance on the unlabeled data. Authors demonstrate improved performance on semi-supervised problems compared to conventional DKL (trained on labeled data only) as well as adapted versions of semi-supervised classification techniques such as VAT [2].  The paper is technically correct and well-written. I have the following questions and concerns: 1. Do I understand correctly that the idea of variance minimization is specific to DKL and would not work for GPs with conventional kernels? The variance of the predictive distribution serves as a proxy for how the unlabeled data points are close to labeled data in the output space of the DNN? While the paper spends several paragraphs on the motivation for minimizing the predictive variance I still find it somewhat unclear. In the rebuttal the authors confirmed my claim above. I encourage them to try to make the motivation more clear for the camera-ready version.  2. To the best of my knowledge this paper is the first to consider deep semi-supervised regression. Thus, there are no known reference results that could be used to compare against the proposed method. The authors adapt several SSL techniques such as VAT [2] to regression for the sake of comparison, but it is not clear if the baselines are tuned sufficiently. For this reason, it would be beneficial to evaluate the proposed technique on semi-supervised classification, where multiple baseline results are available. The authors leave semi-supervised classification for future work.  To address this issue the authors will publicly release their code according to the rebuttal.  3. The authors use cross-validation for tuning hyper-parameters. Is it possible to tune the hyper-parameters by maximizing the likelihood / evidence lower bound (ELBO)? As discussed in [3], only very small validation sets can be used in realistic applications of semi-supervised learning; using the bayesian model selection can be especially beneficial in this application, if it would allow to avoid using a validation set.  According to the rebuttal the authors tune some of the hyper-parameters with marginal likelihood optimization, and will make this point more clear in the camera-ready version.  The paper explores semi-supervised regression, an unexplored but potentially important area of machine learning. It proposes new interesting methodology for this problem and shows promising results. I recommend an accept for this paper.  I was satisfied with the author's feedback and the discussion phase didn't raise any important issues with this paper. I thus increase my score to 7.  [1] Deep Kernel Learning; Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing [2] Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning; Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Shin Ishii [3] Realistic Evaluation of Deep Semi-Supervised Learning Algorithms; Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, Ian J. Goodfellow


Solving Large Sequential Games with the Excessive Gap Technique
There has been tremendous recent progress on equilibrium-finding algorithms for zero-sum imperfect-information extensive-form games, but there has been a puzzling gap between theory and practice. First-order methods have significantly better theoretical convergence rates than any counterfactual-regret minimization (CFR) variant. Despite this, CFR variants have been favored in practice. Experiments with first-order methods have only been conducted on small-and medium-sized games because those methods are complicated to implement in this setting, and because CFR variants have been enhanced extensively for over a decade they perform well in practice. In this paper we show that a particular first-order method, a state-ofthe-art variant of the excessive gap technique-instantiated with the dilated entropy distance function-can efficiently solve large real-world problems competitively with CFR and its variants. We show this on large endgames encountered by the Libratus poker AI, which recently beat top human poker specialist professionals at no-limit Texas hold'em. We show experimental results on our variant of the excessive gap technique as well as a prior version. We introduce a numerically friendly implementation of the smoothed best response computation associated with first-order methods for extensive-form game solving. We present, to our knowledge, the first GPU implementation of a first-order method for extensiveform games. We present comparisons of several excessive gap technique and CFR variants.

Previous results on zero sum imperfect information games such as poker have mostly used counterfactual regret minimization (CFR) and variants to find approximate equilibria, even though CFR has worse theoretical guarantees than first order methods (FOM).  This paper shows that a specific first order method called the excessive gap technique with aggressive step size (EGT/AS) is competitive with CFR and it's variants.  This means that it performs worse than the best CFR variants, but better than others.  The paper is a somewhat incremental extension of previous work laying out the theory of EGT and the specific distance functions needed.  Previous work also introduced EGT/AS, and showed that it worked on small to medium sized games.  The main contributions of this paper are 1. Showing that EGT/AS works reasonably well on large games, specifically poker endgames arising from Libratus. 2. Some technical and numerical details concerning solving EGT subprograms in numerically stable ways, and a bit of (relatively straightforward) discussion about mapping these computations to GPUs. I would advocate accepting this paper, as demonstrating performance of EGT/AS on interesting games is a useful contribution, and is useful for guiding future research.  The sense I get from this paper is that the space of algorithms spanned by CFR, EGT, and their variants is far from settled, and it seems likely that future work will find better variants that match the best theoretical guarantees with better empirical performance.  That blending isn't this paper, but it's useful to know that EGT/AS is an empirical contender (even if it isn't as good as the best CFR variants for reasonable amounts of convergence).  As a general aside: I am curious if or when when poker algorithms will adopt actual ML techniques.  It's reasonable to include this series of "clever brute force, non-ML" algorithms in NIPS since they are solving a goal clearly within the  scope of NIPS applications, but it's interesting that poker seems to be going the way of chess rather than go and avoi ding primarily ML solutions.  On the subject of reproducibility: the methods presented are clear, so the main reproducibility barrier is having the surrounding machinery for a Libratus-equivalent engine.  This is hard to avoid for this type of scaling paper, so it should not count much against acceptance.  Various notes: 1. The abstract should say "imperfect information" clearly.  It's mathematically implied by not saying "perfect information", but worth being explicit. 2. The first sentence frightens me a little bit (line 21-22).  Hopefully most business and military games are not zero sum. :) 3. Line 99: "and is strongly convex modulus" should be "and has...". 4. Line 108: What norm is ||A||?  From Nesterov it looks like a mixed p-norm operator norm, which no one is going to guess without looking at Nesterov, and I'm confused by the norm would be asymmetrically defined given the symmetric structure of the bilinear program.  Also, what are the a_1, a_2 constants?  Ah, are those the unnecessary linear terms in the bilinear function phi?  Finally, it's bad to introduce phi in line 85, discard it, and then introduce another function named phi in equation (3). 6. Why is the (E)xcessive (G)ap (C)ondition called EGV? 7. It's worth some discussion whether EGT/AS or variants are applicable to the nonzero sum setting, and more generally to games with more than 2 players.  By default I believe the answer is no, since two player zero sum has additionally convexity structure which these algorithms are exploiting heavily.


Unsupervised Depth Estimation, 3D Face Rotation and Replacement
We present an unsupervised approach for learning to estimate three dimensional (3D) facial structure from a single image while also predicting 3D viewpoint transformations that match a desired pose and facial geometry. We achieve this by inferring the depth of facial keypoints of an input image in an unsupervised manner, without using any form of ground-truth depth information. We show how it is possible to use these depths as intermediate computations within a new backpropable loss to predict the parameters of a 3D affine transformation matrix that maps inferred 3D keypoints of an input face to the corresponding 2D keypoints on a desired target facial geometry or pose. Our resulting approach, called DepthNets, can therefore be used to infer plausible 3D transformations from one face pose to another, allowing faces to be frontalized, transformed into 3D models or even warped to another pose and facial geometry. Lastly, we identify certain shortcomings with our formulation, and explore adversarial image translation techniques as a post-processing step to re-synthesize complete head shots for faces re-targeted to different poses or identities.

[Summary]  This paper proposes a method to project a source facial image to a target facial image with the help of 2D keypoints (or, facial landmarks). In particular, the method estimates the depth of the 2D keypoints of the source images using information from both images, and the method estimates the 3D-to-2D affine transform from the source to the target. With this transformation, a traditional keypoint-based face warping (implemented in OpenGL) algorithm and CycleGAN are used to map the source image to the target image. Note that the estimation of the depth and affine transform can either depends on only the 2D keypoints or both the keypoints and images.    [Pros]  The paper proposes three slightly different ways to estimate the depth of 2D keypoints and the 3D-to-2D transformation from the source image to the target image. The formulation of all the variants are reasonable, and explicitly predicting depth as an intermediate step is interesting.  Detailed comparisons are performed among these variants to justify the effectiveness of the final model, which estimates the depth first and incorporate the keypoint-based estimation of the affine transform into the neural network.   The results in Figure 4 for replacing faces are interesting.   [Cons]  1. It is unclear why the paper addresses an useful task. The problem setting assumes the keypoints or also the images for both the source and target faces are known (or, detected for the keypoints).  a) Suppose both images and keypoints are given (case A in the paper), why projecting the first image to the second image is a meaningful task?  b) When only the keypoints are given for the target face (case B), there are also many existing works for face morphing. Why estimating an affine transform first and then transform the source face is a better way of face morphing?  c) For specific morphing tasks like frontalization, the minimal information needed is the rotation angle. Requiring keypoints for the target face are unrealistic or costly. If the keypoints are detected, we have already known the target images (refer to 1.a) … If the keypoints are given manually, it is too costly. (It is possible that I misunderstood the experimental setting for the frontalization, which was not well-clarified in the experimental section.)   2. The paper emphasized “unsupervised learning,” but the method does not seems to be “unsupervised.”  a) The acquisition of keypoints (either a pretrained detector or ground truth) are obtained from supervision. So, at least for the face rotation and replacement, the setting is not fully unsupervised. b) The models are trained using paired facial images from the same identities. It is already a type of supervision to know if images are from the same identity or not.  Most experimental results in this paper are about keypoints. The results for the tasks of face rotation are limited. And the qualitative results in Figure 3 are not visually strong. The identity does not seem to be preserved well during the rotation.  Also, no experiments are present to show the usefulness of the frontalized images. Experiments on facial recognition may be helpful.  There are existing works for both face frontalization and replacement, but no qualitative or quantitative comparison is performed.   [Overall]  The paper proposed interesting techniques, but it is not clear what are the central problems addressed by these techniques and what insights are provided. The experimental results focused mainly on the intermediate results (i.e., the keypoint mapping and keypoint-depth estimation). The results for face frontalization and replacement are limited.  --------------------------  Thank you for the authors’ response.   The response tried to address the major concerns I have.  I agree with the author that the key-point transform can be useful for movie making, but I still feel the significance of the method is limited and not sufficiently argued.   Essentially, the paper address the problem of a generalized stereo-based 3D reconstruction for key-points using a learning-based method. The estimated transform between the two sets of key-points is applied to face frontalization and face replacement.  However, a large number of works exist on stereo-to-3D reconstruction for both sparse key-points and dense images. Stereo-based face reconstruction is particularly well-studied. What might be interesting is that the proposed method does not necessarily require the two images to be captured at the same time and potentially not for the same person. But given the consistent structure of human faces, the results are not that surprising.  I acknowledge the authors' explanation about the “unsupervised,” which was also my understanding of this paper. However, the title and abstract could mislead people to believe this paper is addressing unsupervised 3-D key-point learning problem. Moreover, in this misunderstood case, the paper could be more relevant to Thewlis et al.’s ICCV paper (which was suggested by R1).   The face frontalization and replacement are pretty separate from the depth and transform estimation. It can be also done with the depth and transform estimated by other stereo-to-3D methods. However, the paper lacks comparison with other stereo-to-3D  methods using the same image generator. Moreover, it also lacks comparison with face frontalization methods in other frameworks.  Overall, the paper does not define its problems clearly and shows a tendency to mix up things. So while appreciating the authors’ efforts in the rebuttal, I lean to reject this paper and hope the paper can get improved in the future.


Sequential Test for the Lowest Mean: From Thompson to Murphy Sampling
Learning the minimum/maximum mean among a finite set of distributions is a fundamental sub-task in planning, game tree search and reinforcement learning. We formalize this learning task as the problem of sequentially testing how the minimum mean among a finite set of distributions compares to a given threshold. We develop refined non-asymptotic lower bounds, which show that optimality mandates very different sampling behavior for a low vs high true minimum. We show that Thompson Sampling and the intuitive Lower Confidence Bounds policy each nail only one of these cases. We develop a novel approach that we call Murphy Sampling. Even though it entertains exclusively low true minima, we prove that MS is optimal for both possibilities. We then design advanced self-normalized deviation inequalities, fueling more aggressive stopping rules. We complement our theoretical guarantees by experiments showing that MS works best in practice.

This paper considers a variant of the pure exploration multi-armed bandit problem, where we are given a threshold gamma and we wish to identify whether the minimum mean among the arms is larger or smaller than gamma.  An algorithm is designed that identifies the correct hypothesis with probability at least 1-delta.  Lower bounds on expected sample size are also provided, and the algorithm's expected sample size is shown to match (exactly, not up to constants) asymptotically in the limit as delta goes to 0.  The paper assumes that the sampling distributions are independent and from a known exponential family.  I'm positive on this paper.  It contains a thorough analysis and a novel algorithm with strong results for this problem. One could object to the strength of the parametric sampling assumption, but this is offset by the precision of the analysis, and I find the assumptions to be fine.  There is an older literature on something called "multiple comparisons with a standard" and "multiple comparisons with the best".  In this problem, we have a collection of arms, typically with independent sampling distributions in exponential families, and we wish to make statistically valid statements about the difference between each arm's mean and either some known value (gamma) or the best arm.  In particular, here is a quote from the abstract of Kim and Nelson (2005): "The goal is to find systems whose expected performance measures are larger or smaller than a single system referred to as a standard and, if there is any, to find the one with the largest or smallest performance.  Our procedures allow.. known or unknown expected performance of the standard."  So it seems that this literature is very close to the problem considered here, but not identical.  While Kim and Nelson seem to do hypothesis testing of mu^* > gamma, the paper seems different because their procedure does additional statistical inference.  The paper would better serve to teach the ML community about older relevant literature if it reviewed the most relevant of this literature.  A survey into the literature is Goldsman and Nelson (1998).  Goldsman, D., and B. L. Nelson. 1998. Comparing Systems via Simulation. In Handbook of simulation: Principles, Methodology, Advances, Applications, and Practice, ed. J. Banks, 273–306. New York: John Wiley & Sons.  Kim, Seong-Hee. "Comparison with a standard via fully sequential procedures." ACM Transactions on Modeling and Computer Simulation (TOMACS) 15.2 (2005): 155-174.  Another criticism is that the problem considered seems unusual.  Are there applicatios where I would want to know whether mu* > gamma, but I wouldn't also want to know the identity of the best arm?  Overall, I found the paper very clear, and the analysis novel and of high quality.  Significance is affected by the applicability of the problem considered.  Here are a few typos and small comments:  - Are conditions needed on the prior?  I don't see any stated, but at the bare minimum it seems that we need the prior to have support on both H_> and H_ gamma, but I wouldn't also want to know the identity of the best arm?"  In the Monte Carlo Tree Search application, the paper unfortunately doesn't provide enough details given my background to evaluate whether the problem formulation really does have direct applicability to MCTS.  I agree that in spirit it is close, but after skimming the given reference [33] I wonder:  - in 33, pruning (Algorithm 3) is done by comparing the estimated min/max value of a particular child node against the root.  There is error in estimating the min/max value of the root while applying the method in this paper would treat it as a constant (gamma). - in 33, samples of a child nodes value don't seem to come iid, but instead are biased through hierarchical estimation: the value of node u is estimated as the min/max value of its child nodes.  Even if the estimates of the values of the leaf nodes are unbiased, it seems that estimates will be biased higher in the tree. - in MCTS, if I sample 1 child node u, then that can improve the estimates for another node v at the same level because it can help estimate the value of nodes at lower levels shared by these nodes u and v. - is the measure of correctness used really the most appropriate thing for MCTS?  The number of nodes tested is extremely large in MCTS, and it seems that the sample complexity required to make every decision correctly with reasonable probability would just be too large.  Instead, it seems like one would expect some errors, and one would wish that the size of the errors wouldn't be too large most of the time.  This would seem to suggest an error measure that put more weight on incorrect conclusions for arms with large gaps between the value and gamma.  In the quality control applications, there is also room for improving the clarity of the application context.  For example: - regarding the e-learning application, we typically grade students by estimating their average performance across a range of subjects rather than their minimum.  Are there existing e-learning settings where performance is certified in this way?  - when certifying a batch of machines has a minimum probability of successfully producing a widget, I'd feel better if I had more detail about a situation where one would want to do this.  Does this arise for regulatory reasons?  If a plant owner were doing this, and found that the minimum probability fell below a threshold, then she would probably want to know which machine was causing the problem so that she could fix / replace it.  If it is for regulatory reasons, why do they care about the worst machine instead of caring about the fraction of the widgets (produced by all the machines) that have quality below a threshold?  Is the proposed loss function the best one? - unfortunately I do not understand the anomaly detection setting on line 38. - In crowdsourcing, when establishing the quality of workers, what would I do if I found out that a cohort contained a careless one?  Would I simply go get a new cohort?  Would I have some other non-crowd workers do the task I originally planned for the crowd?  It seems like an approach that is more efficient than each of these would be to try to figure out which one(s) were careless and ignore their work.  At the same time, I feel that the motivating quality control applications are nearly on par in terms of actual applicability with many other theoretical bandit papers published in the past at NIPS.  (For example, the assumption of each arm having iid rewards common in the stochastic bandits literature is routinely violated in applications.)  Finally, the Bernoulli argument is a good one in favor of the parametric assumption.


CatBoost: unbiased boosting with categorical features
This paper presents the key algorithmic techniques behind CatBoost, a new gradient boosting toolkit. Their combination leads to CatBoost outperforming other publicly available boosting implementations in terms of quality on a variety of datasets. Two critical algorithmic advances introduced in CatBoost are the implementation of ordered boosting, a permutation-driven alternative to the classic algorithm, and an innovative algorithm for processing categorical features. Both techniques were created to fight a prediction shift caused by a special kind of target leakage present in all currently existing implementations of gradient boosting algorithms. In this paper, we provide a detailed analysis of this problem and demonstrate that proposed algorithms solve it effectively, leading to excellent empirical results.

UPDATE AFTER AUTHORS' RESPONSE  Regarding "using one tree structure", I think I understand now, and I think the current wording is confusing.  Both the manuscript and the response made me think that the *same* tree splits (internal nodes) are used for all of the boosting iterations.  But looking at the argmin line in Algorithm 2, I think the intent is to say "the same feature is used to split all internal nodes at a given level of a tree" (aka, oblivious tree).  If that is not right, then I am still confused.  Regarding one random permutation, please update text to be more clear.   SUMMARY  The paper identifies two potential sources of overfitting, one related to how high cardinality categorical features are encoded, and the other due to reuse of labeled examples across gradient boosting iterations.  A new boosting variation, CatBoost, addresses both problems by treating the examples as an ordered sequence that is accessed in an online or prequential fashion (ala A.P. Dawid's work). The algorithm builds a nested sequence of models that are indexed against the sequence of labeled examples.  Model i is allowed to use all the labeled examples seen before the i'th point in the example sequence, but none afterwards.  This helps avoid estimation bias from reusing target labels (supported by theoretical analysis). Empirically, CatBoost is more accurate than popular boosting implementations (LightGBM and XGBoost) with comparable or faster training time.  The impact from different components of the algorithm are measured empirically.  REVIEW  This is a very good paper.  It makes several valuable contributions:  * Help clarify a source of overfitting in boosting.  This should help   others continue research into the problem. * Propose a new, interesting way to avoid target leakage when   representing categorical features with target statistics.  The risk   of target leakage is well-known, and the paper shows that the new   approach leads to better accuracy than the simple approaches people   are currently using. * Propose ordered boosting to solve the overfitting problem, and   describe an efficient implementation of it. * Solid empirical study of the new algorithm.  The main weakness is not including baselines that address the overfitting in boosting with heuristics.  Ordered boosting is non-trivial, and it would be good to know how far simpler (heuristic) fixes go towards mitigating the problem.  Overall, I think this paper will spur new research.  As I read it, I easily came up with variations and alternatives that I wanted to see tried and compared.   DETAILED COMMENTS  The paper is already full of content, so the ideas for additional comparisons are really suggestions to consider.  * For both model estimations, why start at example 1?  Why not start   at an example that is 1% of the way into the training data, to help   reduce the risk of high variance estimates for early examples?  * The best alternative I've seen for fixing TS leakage, while reusing   the data sample, uses tools from differential privacy [1, 2].  How   does this compare to Ordered TS?  * Does importance-sampled voting [3] have the same target leakage   problem as gradient boosting?  This algorithm has a similar property   of only using part of the sequence of examples for a given model.   (I was very impressed by this algorithm when I used it; beat random   forests hands down for our situation.)  * How does ordered boosting compare to the subsampling trick mentioned   in l. 150?  * Yes, fixes that involve bagging (e.g., BagBoo [4]) add computational   time, but so does having multiple permuted sequences.  Seems worth a   (future?) comparison.  * Why not consider multiple permutations, and for each, split into   required data subsets to avoid or mitigate leakage?  Seems like it   would have the same computational cost as ordered boosting.  * Recommend checking out the Wilcoxon signed rank test for testing if   two algorithms are significantly different over a range of data   sets.  See [6].  * l. 61: "A categorical feature..."  * l. 73: "for each categorical *value*" ?  * l. 97: For clarity, consider explaining a bit more how novel values   in the test set are handled.  * The approach here reminds me a bit of Dawid's prequential analysis,   e.g., [5].  Could be worth checking those old papers to see if there   is a useful connection.  * l. 129: "we reveal" => "we describe" ?  * l. 131: "called ordered boosting"  * l. 135-137: The "shift" terminology seems less understandable than   talking about biased estimates.  * l. 174: "remind" => "recall" ?  * l. 203-204: "using one tree structure"; do you mean shared \sigma?  * Algorithm 1: only one random permutation?  * l. 237: Don't really understand what is meant by right hand side of   equality.  What is 2^j subscript denoting?  * l. 257: "tunning" => "tuning"  * l. 268: ", what is expected."  This reads awkwardly.  * l. 311: This reference is incomplete.   REFERENCES  [1] https://www.slideshare.net/SessionsEvents/misha-bilenko-principal-researcher-microsoft  [2] https://www.youtube.com/watch?v=7sZeTxIrnxs  [3] Breiman (1999).  Pasting small votes for classification in large databases and on-line.  Machine Learning 36(1):85--103.  [4] Pavlov et al. (2010).  BagBoo: A scalable hybrid bagging-the-boosting model.  In CIKM.  [5] Dawid (1984).  Present position and potential developments: Some personal views: Statistical Theory: The Prequential Approach.  Journal of the Royal Stastical Society, Series A, 147(2).  [6] Demsar (2006).  Statistical comparisons of classifiers over multiple data sets.  Journal of Machine Learning Research, 7:1--30.


Sequential Context Encoding for Duplicate Removal
Duplicate removal is a critical step to accomplish a reasonable amount of predictions in prevalent proposal-based object detection frameworks. Albeit simple and effective, most previous algorithms utilize a greedy process without making sufficient use of properties of input data. In this work, we design a new two-stage framework to effectively select the appropriate proposal candidate for each object. The first stage suppresses most of easy negative object proposals, while the second stage selects true positives in the reduced proposal set. These two stages share the same network structure, i.e., an encoder and a decoder formed as recurrent neural networks (RNN) with global attention and context gate. The encoder scans proposal candidates in a sequential manner to capture the global context information, which is then fed to the decoder to extract optimal proposals. In our extensive experiments, the proposed method outperforms other alternatives by a large margin.

- Abstract Although Duplicate Removal is an important step in object detection, since only the score of each candidate region is considered, the information of the input data was not sufficiently used by the conventional method. This paper proposes a new Duplicate Removal method based on RNN. Based on each candidate area, informative features are extracted by using appearance feature, position and ranking information in addition to the score. Then, they are treated as series data and are input into the RNN-based model to improve the final accuracy by capturing global information. The number of candidate regions is enormous to the number of objects that are to be left. Therefore, this paper proposes to reduce the box gradually by dividing it into two stages. In the two stages, the RNN model of the same structure was used. In stage I, to remove simple boxes the model is trained by using NMS results as a teaching signal. In stage II, to remove difficult boxes, the model is trained by using the grand-truth boxes. Experiments showed that mAP is increased in the SOTA object detection methods (FPN, Mask R - CNN, PANet with DCN) with the proposed method.  - Pros: -- This paper is easy to understand.  -- This paper pointed out the existing Duplicate Removal and proposed a new method to solve it. -- The proposed method is well designed.  - Cons: -- The proposed method is complicated. -- Improvement of detection accuracy seems to be marginal against the complexity of the method.  -- This paper does not include theoretical insights. -- Calculation of NMS is necessary for learning in stage I.  -- There is no description about the calculation cost. Since object detection is a trade-off between detection accuracy and calculation speed mainly in inference, a description on calculation cost is indispensable.  -- It would be better to try on other datasets. In addition, not only the overall mAP, but also more detailed analysis of the results is required. -- - In Table 7, the paper compares only with NMS. This paper should also compare with Box voting and Soft NMS. -- The publication year of [21] might be CVPR 2018.  Although methods are well designed, it is unclear about the calculation cost, and theoretical insights and new findings are not enough at this time.


Learning Optimal Reserve Price against Non-myopic Bidders
We consider the problem of learning optimal reserve price in repeated auctions against non-myopic bidders, who may bid strategically in order to gain in future rounds even if the single-round auctions are truthful. Previous algorithms, e.g., empirical pricing, do not provide non-trivial regret rounds in this setting in general. We introduce algorithms that obtain a small regret against non-myopic bidders either when the market is large, i.e., no single bidder appears in more than a small constant fraction of the rounds, or when the bidders are impatient, i.e., they discount future utility by some factor mildly bounded away from one. Our approach carefully controls what information is revealed to each bidder, and builds on techniques from differentially private online learning as well as the recent line of works on jointly differentially private algorithms.

Summary:  This paper studies the interesting and important problem of online learning for pricing problems when the buyers are strategic. There is a seller who interacts with a set of buyers over a series of time steps. The seller has a fresh copy of the good for sale every day. She sets a price at the beginning of each day. Then, a bidder arrives and submits a bid: If the bid is higher than the price of the day, he gets the item and pays the price. The question is: how should the seller set the price? He could use an off-the-shelf no-regret learning algorithm. However, if the same buyer returns several times, she might be incentivized to misreport her value in order to fool the learning algorithm into giving her a lower price. Much of the existing literature on this topic assumes the buyers are myopic, which means they do not take future rounds into account when bidding in the current round. This paper does not make this assumption.  This paper presents a no-regret pricing algorithm that satisfies a relaxed version of incentive compatibility. It works under one of two assumptions: Either the bidders discount utility in future rounds by some discount factor bounded away from 1, or any one bidder comes in only a small fraction of the rounds. Their algorithm discretizes the set of prices and treats each discretized point as an expert. At each round, each expert’s reward equals their corresponding price if the bidder bid above their price, and zero otherwise. The algorithm uses an existing differentially privacy tool called “tree-aggregation” to keep track of noisy estimates of each expert’s gain. The algorithm picks a reserve on each day using follow-the-perturbed-leader. The authors generalize to multiple bidders and multiple units of items for sale, too.  Comments:  According to the authors, one of the key insights is to use joint differential privacy rather than differential privacy itself. However, joint differential privacy is not mentioned in Section 3 as far as I can tell, which is a little confusing. Perhaps this is because Section 3 is a simplified version of their main algorithm. In my experience, take-it-or-leave-it auctions, which are commonly referred to as posted price mechanisms, do not take bids as input. The buyer either buys the good and pays the price, or does not buy the good. In what scenario would he be required to submit a bid?   The authors write that online learning for auction design has only been studied “very recently” (line 16), which is not true. For example, see the following paper and subsequent work. Kleinberg and Leighton. The Value of Knowing a Demand Curve: Bounds on Regret for Online Posted-Price Auctions. FOCS 2003.  The authors write that “typical mechanism design approach may seek to design online learning mechanisms such that truthful bidding forms an equilibrium (or even better, are dominating strategies) even if bidders are non-myopic, and to get small regret given truthful bids. However, designing truthful mechanisms that learn in repeated auctions seems beyond the scope of existing techniques.” Can the author comment on what they mean by “beyond the scope of existing techniques?” Isn’t the point of research to come up with new techniques? :-) I think it’s fine to pursue the path the authors chose, but I’d recommend finding a better argument.  In the related work, the authors write that “most previous work achieved approximate incentive compatibility so that misreporting cannot increases a bidder’s utility by more than an epsilon amount.” In contrast, the authors’ relaxed version of incentive compatibility ensures that “bidders would only submit ‘reasonable’ bids within a small neighborhood of the true values.” What are the pros and cons of both relaxations to incentive compatibility? This might be helpful for understanding why there ought to be another paper on this topic.  =====================After rebuttal===================== Thanks for the clarifications!


Wasserstein Variational Inference
This paper introduces Wasserstein variational inference, a new form of approximate Bayesian inference based on optimal transport theory. Wasserstein variational inference uses a new family of divergences that includes both f-divergences and the Wasserstein distance as special cases. The gradients of the Wasserstein variational loss are obtained by backpropagating through the Sinkhorn iterations. This technique results in a very stable likelihood-free training method that can be used with implicit distributions and probabilistic programs. Using the Wasserstein variational inference framework, we introduce several new forms of autoencoders and test their robustness and performance against existing variational autoencoding techniques.

MAIN IDEAS  The paper introduces a new family of divergence measures, which it calls the c-Wasserstein family of divergences. In application (in this paper) the distance between a generative joint density p(x, z) and its approximation q(x, z) is minimized, where (x, z) ~ training data.  The "transport cost" function of the c-Wasserstein divergence depends on distributions p and q, as well as its usual arguments. Theoretical properties are provided (all of this is novel work), and a Monte Carlo estimate shown. The MC estimate is biased for a finite sample size, and the paper shows how to remove the bias in one specific case when p = q.  The gradients of this loss are obtainable by back-propagating through the so-called "Sinkhorn iterations" from optimal transport theory.  RELATIONSHIP TO PAST WORK  The paper builds nicely on Operator Variational Inference (from NIPS last year or the year before), and Renyi divergence variational inference (also at NIPS).  Note to authors: the paper is very heavily biased toward citing work from one inner circle: 7/10 of the first cited papers are from the same group (!). Surely the world is larger?   STRENGTHS  - A beautiful new theoretical umbrella that ties f-divergences and Wasserstein divergences together.  - Very well written paper.  WEAKNESSES  - In time gone by, or "the days still influenced by legacy effects", as the paper puts it in its opening paragraph, divergence measures in approximate inference would always be used with respect to a true partition function, exact moments, and so on. One would compare against a ground truth MCMC estimate, do some thermodynamic integration, etc. Now, the evaluation is very fuzzy: the authors look at the mean squared deviation in the latent space over 30 re-runs of each method. (We assume by mean squared deviation that 30 latent samples were drawn for each datapoint, for each method, and the deviation from the mean of those samples reported?) The paper could state why these metrics are insightful? A method that encodes x into a delta function at zero, would have a zero latent error, and so why is "outperforms" (Line 210) such a big deal?  - The flexibility through C^{pq} introduces another layer of complexity that a modeller has to mitigate.  - Is a bias introduced by running the Sinkhorn iterations for a finite number of steps?  - The paper is very terse (especially section 5.1) and   Typos:  Line 7 in Algorithm 1, sum(u_\tau ... Line 139, should it not map to the expected value of p(x | z)? Line 181, equation, end,  D(p(z) || q(z)) Line 224, J_e_nsen-Shannon


Causal Inference via Kernel Deviance Measures
Discovering the causal structure among a set of variables is a fundamental problem in many areas of science. In this paper, we propose Kernel Conditional Deviance for Causal Inference (KCDC) a fully nonparametric causal discovery method based on purely observational data. From a novel interpretation of the notion of asymmetry between cause and effect, we derive a corresponding asymmetry measure using the framework of reproducing kernel Hilbert spaces. Based on this, we propose three decision rules for causal discovery. We demonstrate the wide applicability and robustness of our method across a range of diverse synthetic datasets. Furthermore, we test our method on real-world time series data and the real-world benchmark dataset Tübingen Cause-Effect Pairs where we outperform state-of-the-art approaches.

This paper proposes a novel approach to causal discovery based on kernel conditional deviance methods. I found the paper very well structured and well written. After setting out the problem and giving a nice visual motivation for their approach, the main contributions are clearly described, and the related work is very nicely summarized in the next section. In the description of the proposed approach, the notation was well introduced, the approach is quite intuitive and accessible through the provided description, and I didn’t find any problems with the mathematical description and derivations. Finally, the set-up of the simulation study was well described so that I would expect to be able to reproduce the results, and the results are quite impressive in that the proposed approach performs well in comparison to existing approaches.  A few minor comments:  - For me, the motivating example in the introduction would have been easier accessible with more details which values of x and y, respectively were used in generating the plots.  - In the simulation experiments in Table 3 in the supplements, it is nice that their proposed method achieves perfect performance in almost all scenarios, but that makes me wonder when it does not achieve perfect performance and how it compares in these scenarios to the alternative methods. I.e. I would be interested in more simulations with larger noise. Furthermore, I would be interested in seeing how KCDC performs for linear or quadratic association between X and Y.  - There were a couple of minor typos which could be removed from one more proofread.


Neural Networks Trained to Solve Differential Equations Learn General Representations
We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layers are general, and that they learn generalized coordinates over the input domain. Deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. We find excellent agreement between the two methods, and note that our method is much faster, particularly for continuously-parametrized problems. Finally, we also apply our method to networks trained on MNIST, and show it is consistent with, and complimentary to, another study of intrinsic dimensionality.

SUMMARY The paper presents a series of experiments in which singular vector canonical correlation analysis (SVCCA) is used to measure the similarity between neural network representations learned from different random initializations and from slightly different problems. In most prior work, the generalizability of neural network representations is measured through transfer learning experiments, which can be computationally expensive; the key insight of this paper is that SVCCA provides an alternative way to measure feature generalizability that tends to give similar conclusions as transfer learning, but which is orders of magnitude faster to compute.  The authors use SVCCA to perform extensive experiments measuring the degree to which differential equation neural networks (DENN) vary with different random seeds, and vary across related problem instances. Experiments with SVCCA suggest that lower network layers are more general, while higher layers are more specific to the particular problem instance on which the network was trained. Lower layers are more reproducible, giving rise to representations that are more stable under retraining, and across all layers reproducibility grows as network width increases. A subset of these experiments are reproduced with transfer learning, and give similar conclusions but take orders of magnitude longer to run.  PROS - The paper demonstrates that SVCCA can be a computationally inexpensive way to study neural network representations - Experiments with SVCCA suggest a number of interesting features of neural networks - The paper is extremely well-written and clearly presented  CONS - The motivation for focusing on DENNs is unclear - I don’t know how much the conclusions drawn from experiments on DENNs would generalize to other more commonly-used applications of neural networks.  WHY FOCUS ON DENNS? The paper motivates the use of DENNs with two major reasons: (Lines 61-68): First, that DENNs are a powerful method for solving differential equations, and second that they “offer the opportunity to study the behavior of neural networks in a well-understood context”. I agree that these are both interesting reasons for studying DENNs; however as presented the bulk of the paper has very little to do with DENNs at all, and instead focuses on the degree to which neural network representations vary when trained in different conditions. The questions that the paper raises around the intrinsic dimensionality, reproducibility, and specificity of learned representations are very interesting and of extremely broad applicability; however given the far-reaching implications of these experimental questions, I’m surprised that the authors chose to focus on the relatively niche area of DENNs rather than more well-studied applications like image recognition, speech recognition, language modeling, or machine translation.  HOW MUCH DO CONCLUSIONS GENERALIZE? The paper’s experiments suggest a number of intriguing conclusions about learned representations; however due to the focus on DENNs I’m unsure to what extent these conclusions would generalize to more standard applications of neural networks, especially since there are some quirks of DENNs that are quite different from many other types of neural networks.  First, the problem setup and loss for DENNs are not very representative of typical applications of neural networks; the network inputs the (x, y) coordinates of a point the plane, and outputs a scalar value; the loss function (Supplementary Equation 1) incorporates a squared Euclidean distance between the second derivative of the network’s output and a target function s. This problem setup and loss function are extremely different from most applications of neural networks, which typically input some natural data (images, text, speech, etc) and are trained with cross-entropy or regression loss functions.  Second, all networks studied are extremely small: they are four-layer fully-connected layers, with between 8 and 192 hidden units per layer. This means that the largest networks studied have on the order of 150K learnable parameters; in comparison a ResNet-50 on ImageNet has about 25M learnable parameters. Will the conclusions of the paper generalize to larger networks?  Third, all networks use tanh nonlinearities; ReLU nonlinearites are much more common for most problems. The authors do provide a convincing explanation for why tanh is preferred over ReLU for DENNs (Supplementary Lines 10-16), though I still wonder whether the results would apply to ReLU networks.  OVERALL On the whole I enjoyed reading the paper. It was extremely well-written, and used well-designed experiments and metrics to draw several interesting conclusions about neural network representations, demonstrating that SVCCA can be a valuable tool in understanding neural networks. However, due to the exclusive focus on DENNs I am unsure how much the conclusions will apply to other more widely-used types of neural networks; experiments on standard tasks (image classification, language modeling, etc) would have significantly strengthened the paper. Despite this shortcoming, overall I lean toward acceptance.  FINAL SCORE I appreciate the additional experiments presented on ReLU networks on MNIST, and I hope these can be included in the camera-ready version of the paper.


HitNet: Hybrid Ternary Recurrent Neural Network
Quantization is a promising technique to reduce the model size, memory footprint, and computational cost of neural networks for the employment on embedded devices with limited resources. Although quantization has achieved impressive success in convolutional neural networks (CNNs), it still suffers from large accuracy degradation on recurrent neural networks (RNNs), especially in the extremely lowbit cases. In this paper, we first investigate the accuracy degradation of RNNs under different quantization schemes and visualize the distribution of tensor values in the full precision models. Our observation reveals that due to the different distributions of weights and activations, different quantization methods should be used for each part. Accordingly, we propose HitNet, a hybrid ternary RNN, which bridges the accuracy gap between the full precision model and the quantized model with ternary weights and activations. In HitNet, we develop a hybrid quantization method to quantize weights and activations. Moreover, we introduce a sloping factor into the activation functions to address the error-sensitive problem, further closing the mentioned accuracy gap. We test our method on typical RNN models, such as Long-Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). Overall, HitNet can quantize RNN models into ternary values of {-1, 0, 1} and significantly outperform the state-of-the-art methods towards extremely quantized RNNs. Specifically, we improve the perplexity per word (PPW) of a ternary LSTM on Penn Tree Bank (PTB) corpus from 126 to 110.3 and a ternary GRU from 142 to 113.5.

This paper introduces a variation on standard RNN formulations named HitNet. HitNet improves the SOTA of quantization performance for standard RNNs such as LSTM/GRU. Quantization has the benefit of allowing RNNs to operate with vastly reduced memory and computation requirements, but generally introduces a heavy penalty in terms of the relevant loss function (in this case, perplexity).  The authors note that weights and activations in for LSTMs have hugely different distributions (approximately zero mean Gaussian for weights, whereas output activations are within [0,1] with most of the mass being at the extremes of the range). Quantisation for only one of weights or activations, whilst computing the other at full precision, produces a modest decrease in performance, but quantising both at once introduces a heavy penalty. Taking three standard quantization methods (Uniform Quanttization, Threshold Ternary Quantization, and Bernoulli Ternary Quantization) the authors make a compelling case that TTQ is preferred for quantizing weights, whereas BTQ is better for activations, due to the different distributions above. UQ is shown to be a poor proxy for both types of value, even though it has 4 possible values rather than 3.  After the analysis, the authors introduce slope adjustable versions of standard activation function sigmoid and tanh. The sigmoid version was present in previous RBM work cited, whereas the tanh version is (to my knowledge) novel. The temperature parameter lambda controls the slope of the activation, and the authors demonstrate that sweeping this for sigmoid produces more polarised output distributions for an LSTM. By using lambda < 1, the output activations are closer to bipolar, and as such quantization produces less of a performance penalty. The authors choose a range for lambda by sweeping on both a full precision and a quantised model, finding a range of values which performs well for both. HitNet is comprised of the modified activations functions at the relevant places inside LSTM/GRU, as well as BTQ for activations and TTQ for weights.  The authors evaluate on PTB, Wiki-2 and Text8. Significant improvements in perplexity are recorded compared to many other methods of quantizing models for PTB. For the larger datasets, competing benchmarks are not available, but HitNet seems to perform relatively close to the full precision models.  Overall I find this paper very strong. The motivation is clear, the methodology is simply explained and seems easily reproducable. All experiments are described with sufficient hyperparameters, and the improvements over SOTA are significant. I recommend accepting to NIPS.  Minor points:  L120: "The recurrent timestep is set to 35" - I believe this refers to "sequence length" which I would find clearer.  L139: I find the set notation difficult to understand here - should this be "the set of all i, such that |x_i| > \theta"?  A diagram of the modified sigmoid and tanh activation functions, for a range of lambdas, would be an easily added element to aid comprehension.