A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
Detecting test samples drawn sufficiently far away from the training distribution statistically or adversarially is a fundamental requirement for deploying a good classifier in many real-world machine learning applications. However, deep neural networks with the softmax classifier are known to produce highly overconfident posterior distributions even for such abnormal samples. In this paper, we propose a simple yet effective method for detecting any abnormal samples, which is applicable to any pre-trained softmax neural classifier. We obtain the class conditional Gaussian distributions with respect to (low-and upper-level) features of the deep models under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance. While most prior methods have been evaluated for detecting either out-of-distribution or adversarial samples, but not both, the proposed method achieves the state-of-the-art performances for both cases in our experiments. Moreover, we found that our proposed method is more robust in harsh cases, e.g., when the training dataset has noisy labels or small number of samples. Finally, we show that the proposed method enjoys broader usage by applying it to class-incremental learning: whenever out-of-distribution samples are detected, our classification rule can incorporate new classes well without further training deep models.

This paper proposes a new method for detecting out-of-distribution samples and adversarial examples in classification tasks when using deep neural networks. The core idea is simple. Replace the softmax classification layer by  Gaussian Discriminative Analysis. The assumption is that  given a layer of the network (e.g., the penultimate layer of the net) the conditional distribution of the features given a class, follow a Gaussian distribution. The method then consists in estimating the mean of each distribution and  the covariance assuming that the covariance matrix is the same for every class (similar to LDA as noted in Appendix).  Then, by using the fact that they have now a Generative model (p(x/y)), the can produce a "confidence score" for a new sample. This is score is simply the Mahalanobis distance to the closest distribution (the log of the probability for the closest class).  They show different applications of this framework to detect: out-of-distribution samples (in an experimental setup similar to previous work [22]), state-of-the-art adversarial attacks, and finally they introduce a way to deal with incremental learning.  The major strength of the work is its simplicity. The method is quite simple. The method is backed by empirical results. All experiments show superior performance in comparison to previous work.  The major issues are: -- No analysis. It is not clear whether the conditionally Gaussian distributed assumption is reasonable or not. Moreover, an augmented confidential score computed from the output at different levels of a Deepnet is proposed. But, this will be reasonable only if the output at each layer is Gaussian.  Intuition says that this is not the case unless we are very deep in the net. No analysis is given but only an empirical comparison of the results by taking features at different levels (Figure 2). More analysis is needed.  -- There is no justification why some choices are made. For example, why is the covariance matrix  the same for every class?   -- Novelty is limited. The paper builds a lot on top of [22]. There is no additional analysis. For instance, why does the input preprocessing improve the detection of out-of-distribution samples?  -- Several explanations are unclear. In particular Section 2.3  (incremental learning) is not clearly explained/analyzed. It seems that the idea is to use the features of a deep neural net computed at the last layer as a way to reduce the dimension, and then learn a Gaussian mixture where one can control the number of classes in an online fashion. Is this the case? Also, more analysis to support some conclusions (l173-l174 "if we have enough base classes and corresponding samples ...") is needed.   Is is possible to decide that we are in facing a new class? How could this be done? Algorithm 2, assumes that a set of samples from a new class (labeled) are given, so the method just need to updated the Gaussian mixture.  -- Details for reproducibility are lacking. For instance, l192- "The size of features maps (...) reduced by average pooling for computational efficiency" Which is the final dimension? Also, is this only a matter of computational efficiency? or it is also due the fact that one needs to learn a covariance matrix? More analysis is needed.  -- Experimental setup. The whole evaluation should be done assuming that you don't know how it looks the OOD set (or the adversarial attack). I mean, if you know how it looks, then it's just a typical toy problem of two class classification. I understand this is the way previous work address this problem, but It's time that this is changed. Regarding that point, the paper *does* this analysis ("comparison of robustness")  and the results are pretty interesting (performance goes a little down but results are still very good). But, the experimental section needs to better present and discuss these results. This has to be (better) highlighted in the experimental section).   -- Experimental setup regarding Incremental Learning (Section 3.3) is unclear (many unclear sentences). For instance, "we keep 5 exemplars for each class …. roughly matches with the size of the covariance matrix"). In fact,  the whole experiment is unclear to me. Figure 4 is not commented (there are some crossing in the base class accuracy experiment).   Minor comments  -- How are datasets handled. Are all images the same size (cropped/resized)? --l2 "the fundamental" → "a fundamental" -- l134-l142. Input preprocessing. "small controlled noise". What is added is not noise. The sample is perturbed in a way that the confidence score is increased. This is clearly not random, so please do not refer to this to noise. --"out proposed method" → our method or the proposed method -- metrics are not defined (TNR,TPR,AUPR,AUROC,detection accuracy) (in particular how is computed the detection accuracy) -- l213 - are further → is further -- l230-l234. This is unclear. -- l248 - regression detectors → regression detector -- l251 (and others) - "the LID" → LID or "the LID method" -- l252 -  the proposed method is tuned on → the proposed method tuned on  -- l254 - When we train → when training -- l260 - all tested case → all tested cases -- l261 - Algorithm 2 → Algorithm~2. -- l264 - and Euclidean classifier → and an Euclidean classifier -- l273 - ImageNet → imageNet classes -- l282-l285 - Conclusions should summarize the method and the findings.   **AFTER REBUTTAL** I thank the authors for their response. The authors did a good job clarifying many of the raised points in the reviews. The strength of the paper is that the proposed method is simple and does a good job in detecting out-of-distribution samples (SOTA). Nevertheless, I still think the paper lacks more analysis, *better and clearer justifications*, ablation study, better experimental setup (more controlled experiments).


Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
We introduce a new RL problem where the agent is required to generalize to a previously-unseen environment characterized by a subtask graph which describes a set of subtasks and their dependencies. Unlike existing hierarchical multitask RL approaches that explicitly describe what the agent should do at a high level, our problem only describes properties of subtasks and relationships among them, which requires the agent to perform complex reasoning to find the optimal subtask to execute. To solve this problem, we propose a neural subtask graph solver (NSGS) which encodes the subtask graph using a recursive neural network embedding. To overcome the difficulty of training, we propose a novel non-parametric gradientbased policy, graph reward propagation, to pre-train our NSGS agent and further finetune it through actor-critic method. The experimental results on two 2D visual domains show that our agent can perform complex reasoning to find a near-optimal way of executing the subtask graph and generalize well to the unseen subtask graphs. In addition, we compare our agent with a Monte-Carlo tree search (MCTS) method showing that our method is much more efficient than MCTS, and the performance of NSGS can be further improved by combining it with MCTS.

The paper introduces an RL problem where the agent is required to execute a given subtask graph which describes a set of subtasks and their dependency, and proposes a neural subtask graph solver (NSS) to solve this problem. In NSS, there are an observation module to capture the environment information using CNN, and a task module to encode the subtask graph using recursive-reverse-recursive neural network (R3NN). A non-parametric reward-propagation policy (RProp) is proposed to pre-train the NSS agent and further finetune it through actor-critic method.  Pros: 1. In general, the problem introduced in this paper is interesting and the method which uses CNN to capture the observation information and R3NN to encode the subtask graph is a good idea. 2. The method is evaluated in two experiments which outperforms the baselines in some different settings.  Cons: 1. Writing: many details of the proposed method are included in the supplementary material which makes it difficult to understand by reading the main paper only. For example, even the objective is not defined in the main paper and all the training processes are presented in the supplementary material. 2. The baseline used in this paper is simple. Why not compare to the schemes listed in the related work section?


Flexible Neural Representation for Physics Prediction
Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail. Inspired by this ability, we propose a hierarchical particlebased object representation that covers a wide variety of types of three-dimensional objects, including both arbitrary rigid geometrical shapes and deformable materials. We then describe the Hierarchical Relation Network (HRN), an end-to-end differentiable neural network based on hierarchical graph convolution, that learns to predict physical dynamics in this representation. Compared to other neural network baselines, the HRN accurately handles complex collisions and nonrigid deformations, generating plausible dynamics predictions at long time scales in novel settings, and scaling to large scene configurations. These results demonstrate an architecture with the potential to form the basis of next-generation physics predictors for use in computer vision, robotics, and quantitative cognitive science.
object interactions. However, a number of challenges have remained in generalizing this approach to real-world physical dynamics, including representing arbitrary geometric shapes with sufficient resolution to capture complex collisions, working with objects at different scales simultaneously, and handling non-rigid objects of nontrivial complexity .   t  t+1  t+2  t+3  t+4  t+5  t+6  t+7  t+8  t+9 Ground Truth Prediction Figure 1: Predicting physical dynamics. Given past observations the task is to predict the future physical state of a system. In this example, a cube deforms as it collides with the ground. The top row shows the ground truth and the bottom row the prediction of our physics prediction network.
Several of these challenges are illustrated in the fast-moving deformable cube sequence depicted in Figure 1. Humans can flexibly vary the level of detail at which they perceive such objects in motion: The cube may naturally be conceived as an undifferentiated point mass as it moves along its initial kinematic trajectory. But as it collides with and bounces up from the floor, the cube's complex rectilinear substructure and nonrigid material properties become important for understanding what happens and predicting future interactions. The ease with which the human mind handles such complex scenarios is an important explicandum of cognitive science, and also a key challenge for artificial intelligence. Motivated by both of these goals, our aim here is to develop a new class of neural network architectures with this human-like ability to reason flexibly about the physical world.
To this end, it would be natural to extend the interaction network framework by representing each object as a (potentially large) set of connected particles. In such a representation, individual constituent particles could move independently, allowing the object to deform while being constrained by pairwise relations preventing the object from falling apart. However, this type of particle-based representation introduces a number of challenges of its own. Conceptually, it is not immediately clear how to efficiently propagate effects across such an object. Moreover, representing every object with hundreds or thousands of particles would result in an exploding number of pairwise relations, which is both computationally infeasible and cognitively unnatural.
As a solution to these issues, we propose a novel cognitively-inspired hierarchical graph-based object representation that captures a wide variety of complex rigid and deformable bodies (Section 3), and an efficient hierarchical graph-convolutional neural network that learns physics prediction within this representation (Section 4). Evaluating on complex 3D scenarios, we show substantial improvements relative to strong baselines both in quantitative prediction accuracy and qualitative measures of prediction plausibility, and evidence for generalization to complex unseen scenarios (Section 5).

The authors propose a novel hierarchical object representation based on particles to cover both rigid geometrical shapes and deformable materials. Each scene is represented as a graph, with disconnected components corresponding to the objects and the support of the scene. Each graph has a tree-like structure, where higher levels correspond to coarser scales, and the leaves correspond to the original particles placed in the object. They also propose an adapted neural network architecture, called Hierarchical Relation Network, that learns to predict physical dynamics for this representation. This architecture propagates exterior effects (forces, collisions) as well as the effect of past states through the graph, first from the leaves to their ancestors, then among sibling nodes, and finally back to the leaves. This multiscale approach is end to end differentiable, allowing this propagation mechanism to be learned.  The approach is very nicely motivated, and addresses an important problem: learning to model and predict physical states and dynamics of complex objects, and their interactions, in an end to end learnable fashion, so that it could be integrated in larger learning systems. This could be very useful in scenarios of computer vision, (eg. for action recognition tracking and future prediction), robotics (action conditioned future prediction is crucial for navigation) and reinforcement learning. Overall the paper is well written. The authors cite and compare to the most relevant prior work in a convincing manner - although it would be worthwhile citing "Learning Physical Intuition of Block Towers by Example" by Lerer et al. as well as "IntPhys: A Benchmark for Visual Intuitive Physics Reasoning" by Riochet et al. The approach is novel, and the experimental set up they propose is challenging, and would certainly serve future research purposes.   In terms of weaknesses of the approach, the proposed message passing scheme, relying on the hierarchical representation, does not seem very principled. It would be nice to know more precisely what has inspired the authors to make these design choices. Perhaps as a consequence, in qualitative results show sometimes objects seem to rip appart or smoothly deform into independent pieces. Qualitatively, some results are impressive, while some do not conform to our expectations of what would happen; cf my previous comment on unexpected deformations of objects, but also some motions: cubes moving slightly on the ground although they should be still, etc. More importantly, the applicability of the method is questionable, as it requires ground truth for the local and global deltas of each particle for training. Furthermore, even for inference, it would require the initial states of the particles. This would be a huge challenge in computer vision applications, and in any other applications relying on real videos, rather than synthetic videos. This greatly limits the applicability of the proposed representation, and as a consequence of the proposed architecture. Therefore I have doubts at this point that the proposed architecture has "the potential to form the basis of next generation physics predictors for use in computer vision, robotics, and quantitative cognitive science". To claim this, the authors should be able to show preliminary results of how their method could be used, even in very simple, but realistic settings.  Nevertheless I believe that it is worthwhile sharing these results with the community, as well as the code for the method and the environment, as this is an important problem, and the authors attack it with an ambitious and novel (although slightly ad hoc) approach.  To be fully confident that this paper should be accepted, I would like the authors to clarify the following points: - it is not clear how the quantitative results are obtained: what data exactly is used for training, validating and testing ?  - ternary input to the phi network, encoding the type of pairwise relationship -> is this effective ? What happens otherwise, if this is not given, or if weights are not shared ? Also wouldn't it make sense to have categorical input in the form of a one hot vector instead ? - the algorithm described for the creation of the graph G_H is inconsistent with the description of the remaining edges (l.122 to 1.131). According to the algorithm, there should only be edges between siblings, from the root to the leaves and back, and from the root to all intermediate nodes. Maybe this is due to l.127: "each new node taking the place for its child leaves" should be "each new node taking the place of the root" ? This would then also be more consistent with Algorithm 1. Please clarify. In Figure 3b) it is not clear what the red links represent, if the edges are supposed to be directed. - I don't understand the process for learning the r_ij. If it can be different for each pair of particle (or even for each source particle i), how can it be learned using the losses described in 4.3 ? We can obtain their values on the objects of the training set, but then for each new object, we need to gather synthetic videos of the object moving, in order to learn its properties? Wouldn't it be simpler to have fixed parameters ? Furthermore, if the stiffness of the material is to be learned, why not learn the mass as well ? - in the algorithm described for the creation of the graph G_H, it is said that one should "add edges between cluster nodes if the clusters are connected by edges between leaves" - how is there a presence or an absence of an edge in the first place ?  Nitpicking: - are humans really able to predict subtle deformations of masses at different scales ? Or can they simply perceive them and say whether they are plausible or not ?(l.39-44)  Typos: - par(i) should be par(p) l.128 - Figure 3: "is constraint" should be "is constrained" - l.152; l should be p_l ? - l 155 repetition with the following: "Effects are summed together"  -------------------------------- Final decision after rebuttal: The authors have addressed all of my concerns, except for one which I share with R3 that the experimental protocol for the quantitative results is not described.  Overall however I think that the paper presents an ambitious and novel method to a very important problem, which they convincingly test in a challenging experimental set up. I agree with R3 that the qualitative evaluation should include the weaknesses in the main paper and I think it is beneficial that the authors have agreed to do so.  I remain convinced that it would be worth sharing these results with the community. I can only urge the authors to respect their commitment to share the code and environment, and to add the missing details about the experimental protocol, to ease further research in this area and comparison to their work. I stand by my initial decision that the paper should be accepted as the approach is a very well executed stepping stone for others to improve on in the extremely challenging setting proposed by the authors.


DeepExposure: Learning to Expose Photos with Asynchronously Reinforced Adversarial Learning
The accurate exposure is the key of capturing high-quality photos in computational photography, especially for mobile phones that are limited by sizes of camera modules. Inspired by luminosity masks usually applied by professional photographers, in this paper, we develop a novel algorithm for learning local exposures with deep reinforcement adversarial learning. To be specific, we segment an image into sub-images that can reflect variations of dynamic range exposures according to raw low-level features. Based on these sub-images, a local exposure for each sub-image is automatically learned by virtue of policy network sequentially while the reward of learning is globally designed for striking a balance of overall exposures. The aesthetic evaluation function is approximated by discriminator in generative adversarial networks. The reinforcement learning and the adversarial learning are trained collaboratively by asynchronous deterministic policy gradient and generative loss approximation. To further simply the algorithmic architecture, we also prove the feasibility of leveraging the discriminator as the value function. Further more, we employ each local exposure to retouch the raw input image respectively, thus delivering multiple retouched images under different exposures which are fused with exposure blending. The extensive experiments verify that our algorithms are superior to state-of-the-art methods in terms of quantitative accuracy and visual illustration.

The paper proposes an image retouching method in a local processing and fusion manner. The method first segments the image into sub-images, then each sub-image is passed into a reinforment-learning-based method to output an optimal exposure for this region. The exposure-corrected images under all the exposures (estimated from all the sub-images) are fused through a blending approach.   Pros:  1. The authors propose an image retouching method by exploiting local adjustment instead of global adjustment. This could better handle the cases that bright and dark regions appear in the same picture. The three components are reasonably assembled. 2. The proposed method achieves good quantitative result in terms of PSNR.  Cons: 1. The proposed method consists of three components, segmentation, local retouching, and fusion. The authors use the method [8] for segmentation, and the method [24] for blending. And the retouching strategy is very similar to [13], which also uses reinforcement-learning to update the exposure and a discriminator to represent the loss/AE function (and approximating the loss using CNN). The major difference is that [13] applies global adjustment instead of local one. Considering those, the proposed method is more a systematic work by combining several existing components, thus marginalize its novelty. If there are other differences between [13] and the local retouching component of this method (which I did not see much), the authors should provide a comparison by replacing the component with the method in [13], to demonstrate the advantages of the proposed one. 2. The authors provide the quantitative results on MIT-Adobe 5K in terms of PSNR. As there are no ground-truth images for this task, are the expert retouched results treated as the reference? If so, the results using this metric is less informative, as the quantitative evaluation with respect to the expect retouched results could be biased.  Since the task is subjective, a user study to evaluate the results is suggested.  3. From the quantitative results, the improvement over [13] is little and less than expected (less than 0.06db considering [13] also use the approximation function). I would like to see more analysis on this.  4. In Figure 3, the results using [5] look better than the proposed method. Some more analysis on this and quantitative comparison with [5] are recommended.  5. In Figure 3, the results using the proposed method look over-exposed. What is the reason for that? Is it because the exposed images are favored by the discriminator? I would like to see some analysis on that.   In general, the novelty of the paper is limited and the experimental validation is a bit lacking.   ---------- The rebuttal addresses some of my mis-understanding before, especially on the non-differentiable form and the generative loss approximation. And therefore the usage of asynchronous reinforcement learning makes sense. In this sense, the proposed method is technically sound and novel in terms of the modification on the local adjustment.   For the comparison to [13] with local adjustment in the rebuttal, I don't quite understand the setup (the setup still feels like the proposed framework). And the provided quantitative is hard to explain, which is much worse (0.36db lower) than global retouching itself. A more reasonable way is directly applying the global retouching [13] on different segments to estimate their exposure settings (differentiable loss directly on the segment) and then merge all the retouched results of different exposure settings via the exposure fusion. Since there are a few modifications and there lacks of ablation study (and difficult to evaluate) for each small modification. I think this experiment at least helps understand whether the improvement is from simply local adjustment or others.


Norm matters: efficient and accurate normalization schemes in deep networks
Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weightdecay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used L 2 batch-norm, using normalization in L 1 and L ∞ spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks.

This paper introduces and motivates L1 batch normalization (and variations: L_infinity batch-norm, etc.). With a theoretical study of the impact of standard (L2) batch normalization on the optimization process, the authors show that applying batch-norm makes the optimization weight-norm-invariant and is equivalent to just rescaling the learning rate (under the assumption that batch-norm is applied after a linear layer). Experiments confirm this. It is then proposed to use other kinds of normalization, namely L1, L_infinity, and a parameterized family between both, for computational efficiency reasons (faster to compute and requiring less precision, making it available for half-precision hardware), and it is shown that similar performance (in terms of accuracy) is reached as with the usual batch-norm (with experiments on ImageNet with ResNets). Finally, a similar study is conducted for weight normalization [Salimans & Kingma]. The difference with batch-norm is that the normalization is not the one of each neuronal activity over the mini-batch, but over of all input weights of each neuron together (independently of the mini-batch). As for batch-norm, It is proposed to decouple the norm of the weights from their direction, i.e. to use a representation of the kind x = rho u where u is unit-normed, for some Lp norm.  Pros: - the paper is very well written. Not just easy to read, but very well motivated, with explanations, physical meaning and intuition, etc. The style of paper I'd like to see more often. - The topic of understanding and improving batch-norm is of prime interest. Significant insights are brought. - Experiments are meaningful.   Remarks: - The analysis is specific to the case where batch-norm is applied after a linear layer. It seems this analysis should be still valid after ReLu, maxpoolings, etc. But what about after other activation functions? The experiments are performed with VGG and ResNets, which are conv-ReLu blocks. What about standard networks with really non-linear "non-linearities", i.e. not ReLu but maybe Elu, SeLu, etc.? Is the approach performance similar in practice for such non-"linear" networks? - Similarly, what about other gradients than the standard L2 one, for instance natural gradient, Adam, etc? as Adam is the most common optimizer now. Does the theoretical study still hold for such gradients? Any experiments with Adam? Actually, the paper does not state which optimizer is used, though I guess it might be SGD. - About notations: the paper is clear, but maybe some potential ambiguity in notations could be removed. For instance, sometimes the L2 norm is performed as a sum over all examples in the mini-batch; sometimes it is a sum of all input units for the considered neuron. Would it be possible to state explicitly the space over which the sum is performed? e.g. L2(mini-batch), L2(inputs). For instance line 160, or in equation 7, it is not straightforward to understand that the L2 norm stands for an expectation E_batch (and not on dimensions of x^(1) if any, e.g.). By the way, the expectation symbol would look better using \mathop{\mathbb{E}} for instance.  Edit after authors' response: I would be interested in seeing the extension of the claim to momentum methods / Adam, as it is not straightforward and would be of high interest in the community. I keep the same score.


Sigsoftmax: Reanalysis of the Softmax Bottleneck
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an output activation function for breaking the softmax bottleneck without additional parameters. We re-analyze the softmax bottleneck from the perspective of the output set of log-softmax and identify the cause of the softmax bottleneck. On the basis of this analysis, we propose sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Sigsoftmax can break the softmax bottleneck. The experiments on language modeling demonstrate that sigsoftmax and mixture of sigsoftmax outperform softmax and mixture of softmax, respectively.

The paper analyzes ability of the soft-max, if used as the output activation function in NN, to approximate posterior distribution. The problem is translated to the study of the rank of the matrices contating the log-probabilities computed by the analyzed activation layer. It is shown that the soft-max does not increases the rank of the input response matrix (i.e. output of the penultimate layer) by more than 1. The authors propose to replace soft-max by the so called sigsoftmax (i.e. product of sigmoid and soft-max functions). It is shown that the rank of sigsoftmax matrix is not less the rank of soft-max. The other outcome of the paper is an empirical evaluation of several activation functions on two language modeling tasks.  The provided theorems studying rank of response matrices obtained by various activation functions have their values per se. However, a practical impact of the results and their relevance to the studied language modeling problem are not fully clear from the paper.  In particular, the presented results characterize the rank of the matrix containing the output responses. The rank itself is clearly a proxy for true objective which is not defined in the paper but according to the experiments it seems to be the perplexity of the learned model. The question is how is the rank (i.e. quantity analyzed in the paper) related to the true objective (i.e. the perplexity). E.g. if some activation function produces response matrices with slightly higher ranks (compared to other activation function) does it imply that the corresponding NN model will have lower perplexity ? This question is not discussed in the paper.  The provided thorems suggest that the proposed sigsoftmax is only slightly better than the softmax. In particular, according Theorem 4 the sigsoftmax is proved not to be worse, in terms of the rank, than the softmax.  The empirical study carried on two datasets shows improvements of the sigsoftmax function over softmax by approximately 1 unit. It is not clear if this improvement is statistically relevant because: i) the number of datasets is limited and ii) the confidence of the estimate is not given (the reported STD is not a good measure of the confidence because it measures variation of the results w.r.t. network initialization according to line 268-269).   A pragmatic way to increase expressiveness of NN with the soft-max (and other functions) would to simply increase dimension $d$ of the hidden (penultimate) layer. It would increase the number of parameters, however, the same happens if one uses the mixture of soft-max (MOS). It is not clear why such obvious baseline was not evaluated in the paper.


DAGs with NO TEARS: Continuous Optimization for Structure Learning
Estimating the structure of directed acyclic graphs (DAGs, also known as Bayesian networks) is a challenging problem since the search space of DAGs is combinatorial and scales superexponentially with the number of nodes. Existing approaches rely on various local heuristics for enforcing the acyclicity constraint. In this paper, we introduce a fundamentally different strategy: we formulate the structure learning problem as a purely continuous optimization problem over real matrices that avoids this combinatorial constraint entirely. This is achieved by a novel characterization of acyclicity that is not only smooth but also exact. The resulting problem can be efficiently solved by standard numerical algorithms, which also makes implementation effortless. The proposed method outperforms existing ones, without imposing any structural assumptions on the graph such as bounded treewidth or in-degree.

The authors study the problem of structure learning for Bayesian networks. The conventional methods for this task include the constraint-based methods or the score-based methods which involve optimizing a discrete score function over the set of DAGs with a combinatorial constraint. Unlike the existing approaches, the authors propose formulating the problem as a continuous optimization problem over real matrices, which performs a global search, and can be solved using standard numerical algorithms. The main idea in this work is using a smooth function for expressing an equality constraint to force acyclicity on the estimated structure. The paper is very well written and enjoyable to read. I have the following comments:  - Least-Squares loss is used in this work as the score function. It seems that the main reason for this choice is its desirability from the optimization point of view. I was wondering if there are any structure related guarantee when using this score function. For example, will the global optimum be in the Markov equivalence class of the ground truth DAG? Will we be able to detect the v-structures?, etc.  - Two restricting assumptions in this work are linearity and the assumption of full observability. Both of these assumptions are perfectly reasonable for this stage of the research, but I was wondering if the authors have examined their approach on models which are violating these assumptions? Specifically, in the case of non-linear SEM, how would be the output related to the ground truth DAG? Or more importantly, how essential is additive noise for the performance of Least-Squares loss?  - Augmented Lagrangian method is used for solving the equality-constrained program. My concern is that in this method, the penalty term may force some sparsity restrictions on the learned structure. This may prevent the proposed approach from working well for all desired levels of sparsity.  - The main technical tool in this work is the use of function h(.) for characterizing acyclicity. This characterization seems to be new in the literature, but it would be a more clarifying if the authors mention that the connection between the trace of the powers of the adjacency matrix and cycles in the corresponding graph was known in the literature (for example, Frank Harary, Bennet Manvel "On the number of cycles in a graph," 1971).  My concern about the function h(.) is regarding property (b) of function: "The values of h quantify the “DAG-ness” of the graph". I am not sure how true this claim is: First of all, the value of h depends on the edge weights in the graph. Therefore, two graphs with exactly same structure but different weights will get different values for h(W). This gets more problematic as we can design a graph with more cycles, but smaller weights and end up getting a lower value for h compared to another graph with fewer cycles but larger weights.  - Another concern of mine regarding function h(.) is that it over counts the cycles: If there is just one loop (cycle of length 1) in the graph, its effect will appear in all powers of the adjacency matrix. Also, if there is a cycle of length 3 in the graph, it will cause a non-zero value for all of its three corresponding vertices on the diagonal of W^3. In line 422 in the supplementary materials, it is mentioned that tr(B+B^2+...) counts the number of cycles in B. Based on the aforementioned examples, can we say that this sentence is not necessarily true? A clarification in this regard would be appreciated.  - I suspect that for small values of coefficients, specifically, if all were less than 1, for example, w_{ij}~unif([0.1 0.5]) the error of the proposed method will increase. This is due to the fact that h(w) will be small and the algorithm may make more errors. It would be appreciated if the authors also examine their proposed method on such values.  - The simulations and experiments are sufficient and seem fair.   -----After author feedback:   I thank the authors for their responses. Based on the responses, I increase my score to 8. However, I still have concerns regarding the used score function, the properties of function h(.), and the performance on real data.


3D-Aware Scene Manipulation via Inverse Graphics
We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.

== Summary == This paper tackles the road scene manipulation problem by disentangling texture, semantic, and geometric information from a single image. A deep encoder-decoder network is introduced to learn such a disentangled representation. For each of the object instance, the method further infers the 3D object pose as its 3D-aware representation. Several image manipulation applications are demonstrated, thanks to a fully-disentangled 3D-aware representation. Experimental evaluations are conducted on KITTI and Cityscape benchmarks.   == Quality and Originality == Overall, this paper is quite interesting and experimental results are promising. However, reviewer does have a few concerns about the proposed 3D-aware model. Hopefully, these concerns can be addressed in the rebuttal.  == Object shape representation == Reviewer is a bit concerned as only one car model is used across all car instances (L169-L170). It does not sound like a principled approach for 3D disentangling in general. For example, the proposed approach is going to fail on object categories such as trucks, cyclists, or pedestrians.   == Object occlusion == Reviewer is also concerned whether the proposed 3D-aware geometric inference method is robust to partial occlusions. For example, it might not be a good strategy to treat each car independently if one is occluded by the other. In some cases, the context information could help when there is partial occlusion. However, reviewer is not sure whether the current architecture (Figure 2: looks like the three streams are independent) addresses the issue mentioned here.   == Indoor vs. Road Scene == As it is not shown in the main paper, reviewer would like to know the performance in indoor scene. This is a very critical point, as the title of the submission is about scene manipulation instead of road scene manipulation.  == Missing references == References are incomplete regarding the interpretable 2D/3D representation in deep encoder-decoder networks. -- Attribute2Image: Conditional Image Generation from Visual Attributes, Yan et al. In ECCV 2016. -- Neural Face Editing with Intrinsic Image Disentangling, Shu et al. In CVPR 2017. -- Material Editing Using a Physically Based Rendering Network, Liu et al. In ICCV 2017. -- MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction, Tewari et al. In ICCV 2017. -- Self-Supervised Intrinsic Image Decomposition, Janner et al. In NIPS 2017.


Deep Generative Models for Distribution-Preserving Lossy Compression
We propose and study the problem of distribution-preserving lossy compression. Motivated by recent advances in extreme image compression which allow to maintain artifact-free reconstructions even at very low bitrates, we propose to optimize the rate-distortion tradeoff under the constraint that the reconstructed samples follow the distribution of the training data. The resulting compression system recovers both ends of the spectrum: On one hand, at zero bitrate it learns a generative model of the data, and at high enough bitrates it achieves perfect reconstruction. Furthermore, for intermediate bitrates it smoothly interpolates between learning a generative model of the training data and perfectly reconstructing the training samples. We study several methods to approximately solve the proposed optimization problem, including a novel combination of Wasserstein GAN and Wasserstein Autoencoder, and present an extensive theoretical and empirical characterization of the proposed compression systems.

This works uses generative models for the task of compression. The premise behind this paper is that distribution preserving lossy compression should be possible at all bitrates. At low bitrates, the scheme should be able to recover data from the underlying distribution, while at higher bitrates it produces data (images) that  reconstructs the input samples.  That the scheme works as advertised can be seen rather convincingly in section 5 containing the evaluations.   The paper is well written and constitutes a very interesting exercise.  The compression setup consists of an encoder to compress the input image E, and a stochastic component B, and a generator G recovers the input. Together, they are trained to minimize a distortion measure. The objective function is formulated as a Lagrangian consisting of the distortion measure between input and reconstruction and a divergence term, implemented as a GAN which takes effect at low bitrates so that it recovers images from the underlying distribution.   Several other generative compressive setups are compared against. But the closest comparison (in terms of approach) seems to be the work "Generative Adversarial Networks for learned image compression" (termed GC in the results). In this connection, they point out that if one were simply to minimize the distortion measure as done in GC, it results in the setup largely ignoring the noise injected except at very low bitrates where it is the only factor involved. I think I can intuitively understand this part, but a little clarification would be useful.  The reasoning in section 3, on why one cannot simply replace the distortion metric using a Wasserstein distance seems to be that the encoder is rate constrained and therefore optimizing this metric would not result in optimizing over all couplings. They then go on to decompose the distortion measure (equation (4)) as the sum of two terms, one of which is the wasserstein distance and the other, a term containing the distortion rate which goes to zero at large values.. By doing so (the generator is learnt separately) they can replace the distortion measure with the said Wasserstein term and optimize it.   The connection where they jointly train the setup to function essentially as a VAE-GAN by combining the two terms in the objective - the distortion as the primal, and the divergence as the dual is a most interesting trick for future use.   One aspect that I could not piece together (it is likely that I am missing it) is how the Wasserstein++ algorithm manages to enforce the 1-Lipshitz constraint for the GAN part of the objective. The 1-Lipshitz requirement (as one would expect) is explicitly mentioned in equation 11 for the GAN term. As far as I can see, weight clipping/gradient penalty does not seem to be mentioned in Wasserstein++.   As regards the evaluation studies produced in section 5,  I am curious about use of the Frechet Inception Distance as a metric for use in GANs.   On the whole, I am very impressed with this work. It presents a way to attain generative compression at both low and high bitrates, with reconstructions getting better at higher bitrates and the model recovering distribution images (in GAN like fashion) at low bitrates. The use of Wasserstein autoencoders is also quite interesting.  Answers to specific review criteria below: 1) Quality: Excellent.  2) Clarity: Very good. The mathematical details could perhaps be worked out as an extended tutorial. 3) Originality: Very good. The material presents Wasserstein autoencoders and GANs (combined as Wasserstein++) as the approach to effect compression. This is rather unique.  4) Significance: Very good. Generative compression is a significant area of research.


Manifold Structured Prediction
Structured prediction provides a general framework to deal with supervised problems where the outputs have semantically rich structure. While classical approaches consider finite, albeit potentially huge, output spaces, in this paper we discuss how structured prediction can be extended to a continuous scenario. Specifically, we study a structured prediction approach to manifold valued regression. We characterize a class of problems for which the considered approach is statistically consistent and study how geometric optimization can be used to compute the corresponding estimator. Promising experimental results on both simulated and real data complete our study.

Summary: This paper is an extension of the results presented in “A Consistent Regularization Approach for Structured Prediction” by Ciliberto et al. It focuses on the specific case where the output space is a Riemannian manifold, and describes/proves sufficient conditions for loss functions defined over manifolds to have the properties of what is called a  “Structure Encoding Loss Function” (SELF). Ciliberto et al presents an estimator that, when used with a SELF, has provable universal consistency and learning rates; this paper extends this estimator and these prior theoretical results to be used also with the aforementioned class of loss functions defined over manifolds, with a specific focus placed on the squared geodesic distance. After describing how inference can be achieved using the previously defined estimator for the specific output spaces defined here, experiments are run on a synthetic dataset with the goal of learning the inverse function over the set of positive-definite matrices and a real dataset consisting of fingerprint reconstruction.   Comments: This work is well-written and well-organized, and it is easy to follow all of the concepts being presented. Furthermore, its primary achievement - further characterizing a set of output spaces/functions for which previous methods can be applied - provides a useful extension of that work.  The primary criticism I have of this paper is that it is somewhat incremental in nature - the SELF framework, corresponding consistency and learning rate results, and estimator are all borrowed from Ciliberto et. al. In contrast, all this paper does is describe one set of output spaces/losses for which those previous results can be applied. This might be more forgivable if it was then shown that these methods, applied to many examples of problems satisfying the required assumptions/properties, prove themselves to be very useful. However, though the experiments provide interesting preliminary results, only one task consisting of real data was used to demonstrate the utility of the described approach. At the very least, more examples need to be provided of tasks whose output spaces satisfy assumptions 1 and 2 that are not discrete or euclidean - but it would be preferred if more complete experimentation was done for more tasks.  Miscellaneous: -Line 60: “we consider a structured prediction approach to ma following…” -Line 82: “section section 3”  With further experimentation on a variety of tasks on which other methods may struggle, this paper would make an excellent contribution. As it is currently, however, I think it is a borderline case. I am leaning towards the side of accepting it since everything that is presented is useful and interesting.  POST AUTHOR RESPONSE UPDATE: Thanks for the response. The comparison against the other baseline definitely improves the quality of the synthetic experiment. Additionally, I am glad that you are able to provide another example that will clarify the utility of the proposed methods. Regardless, the point still stands that your experimental comparisons are somewhat weak. If were able to provide interesting results for this new task, then I might have increased your score, but as the work stands I feel like the score I originally gave is still appropriate.


Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences
Differential privacy comes equipped with multiple analytical tools for the design of private data analyses. One important tool is the so-called "privacy amplification by subsampling" principle, which ensures that a differentially private mechanism run on a random subsample of a population provides higher privacy guarantees than when run on the entire population. Several instances of this principle have been studied for different random subsampling methods, each with an ad-hoc analysis. In this paper we present a general method that recovers and improves prior analyses, yields lower bounds and derives new instances of privacy amplification by subsampling. Our method leverages a characterization of differential privacy as a divergence which emerged in the program verification community. Furthermore, it introduces new tools, including advanced joint convexity and privacy profiles, which might be of independent interest.

This paper derives differential privacy guarantees for algorithms that combines subsampling and differentially private mechanisms. It first presents some general tools, and then applies it to different subsampling methods. Analyzing the privacy amplification of subsampling is a very important and interesting problem in differentially private algorithm design. And this paper provides results on different sampling methods and algorithms with different privacy guarantees. More comparison with previous work would make the result more solid, and the presentation might be improved a bit for better readability.  More detailed comments: - It is important to mention related work which proves privacy bound of privacy amplification with sampling in more detail. For example, [1] provides a bound for Gaussian mechanism with sampling. How does the result compare? It might be good to illustrate the comparison with figures and values in some real scenarios. - The presentation in Section 3 might be more clear with subsections for divergences and privacy profiles. - I think Section 4 is an important (if not the most important) section of the paper. So giving it a more informative name may help emphasize its importance and the significance of the paper.    Small typo: - Line 11: there are two "introduces" - Line 114: is the "increasing" supposed to be "decreasing"? Otherwise this couldn't be compatible with limit equal to 0.


Modelling sparsity, heterogeneity, reciprocity and community structure in temporal interaction data
We propose a novel class of network models for temporal dyadic interaction data. Our objective is to capture important features often observed in social interactions: sparsity, degree heterogeneity, community structure and reciprocity. We use mutually-exciting Hawkes processes to model the interactions between each (directed) pair of individuals. The intensity of each process allows interactions to arise as responses to opposite interactions (reciprocity), or due to shared interests between individuals (community structure). For sparsity and degree heterogeneity, we build the non time dependent part of the intensity function on compound random measures following (Todeschini et al., 2016). We conduct experiments on realworld temporal interaction data and show that the proposed model outperforms competing approaches for link prediction, and leads to interpretable parameters.

Summary:  This work extends the Todeschini et al. (2016) framework for sparse community structure to a temporal Hawkes process network, and thus can capture both temporal reciprocity and community structure, which are properties often studied in (temporal) social interaction data. The framework is built on using compound CRMs, and then asymptotic properties of the model are described. Lastly, empirical results for link prediction, community structure, and power law distribution are presented on a variety of temporal social network datasets.  Strengths: - extends the static Todeschini et al. (2016) framework to a temporal setting, which has not been done before - Complex and flexible model for capturing sparsity, degree heterogeneity, and reciprocity, which are desirable properties for many real-world social networks with temporal structure — many existing models are either too simplistic (often ignoring temporal structure) - Develops an efficient inference algorithm for a complex model, and demonstrates on large network datasets - Well-written and good figures  Weaknesses: - Can be viewed as gluing together several existing components of network models: e.g., Caron&Fox and Todeschini et al., and Blundell et al. - Inference details are sparse, may be difficult to reproduce - Experiments are simplistic and applications  Quality:  The paper is technically correct. The claims of sparsity, heterogeneity, and community structure follow from the inherited random measure frameworks of Todeschini and Caron&Fox, and is sparsity (asymptotic) properties are also shown in Section 4.2. The self-excitation comes from the use of the Hawkes process.  The inference in the model appears to be quite scalable, as demonstrated on several large graphs. The model shows significant improvement on the link prediction task. Furthermore, it would be interesting to compare to (or even discuss more in depth) some of the more recent sparse graph models for temporal network data (e.g. Palla et al. and Ng&Silvea), which I don't believe incorporate reciprocity into the models.   The paper would benefit from more of a discussion on the strengths and weaknesses of this approach for modeling real-world temporal data. Currently, it does not appear that weaknesses are addressed.   The submission is clear and fairly well-written, and the figures are informative and intuitive, but in general, more details can be given as well as intuition on a dense topic.   Would be useful to the reader to motivate the 2-step inference procedure a bit further. For instance, the Jacob et al. paper was motivating the modular “cutting” with robustness and model misspecification. In addition, the paper should discuss the strengths and weaknesses of this inference approach.  Additionally, the inference section is a bit glossed-over in details. The current description is not the most useful for someone intending to implement the method from scratch, e.g., stating that MCMC and HMC were used is not sufficient. There is no discussion on the sensitivity of parameters and tuning that is often required. Though the inference is using an open-source package, the paper itself should include more thorough details for reproducibility purposes.  Originality & related work:  The main contribution of this paper is providing a new temporal network model that simultaneously captures sparsity, self-excitatory patterns, and community structure, and demonstrating inference on large graph data sets. However, all of these aspects can be viewed as a combination of well-known existing models for graphs (e.g., Caron--Fox, Todeschini et al., and Blundell et al.). The extension of the Caron--Fox and Todeschini et al. models to a temporal setting is novel, and very few models have approached this.   The related work is very thorough. (The bibliography should be checked for consistency and updated, e.g., Borgs et al. is now published, inconsistencies in the NIPS conference name, etc.)  Significance:  I’m unsure how useful the methods in this paper are for practical network modeling, and the paper may not add much additional value beyond existing work that models self-excitatory patterns in networks (e.g., the Blundell et al. paper). Though performed on large data sets, the experiments are toy experiments that don’t carefully study the data and application, and therefore the conclusions about the data are not particularly unique or surprising.   On the other hand, this is the first temporal version of the Caron--Fox framework that also includes latent community structure and also reciprocity, and could be viewed as a nice "first step" toward modeling temporal network data.   ---  Updated review:  Overall, I think the execution appears to be fairly strong, which is why I’d given it a weak accept score. I think that it is important to have new models of the relatively-new sparse graph frameworks this paper builds on that can be applied to real network data.  As this is a modeling and inference paper, the inference section needs to be able to be read on its own without referring to other work, and an honest assessment of the limitations of the model should be included. The experiments should go beyond just reporting prediction results (with error bars) and visualization of structure, and draw out the important insights of the data applications. The proofs in the appendix could also use more guiding descriptions.


Cluster Variational Approximations for Structure Learning of Continuous-Time Bayesian Networks from Incomplete Data
Continuous-time Bayesian networks (CTBNs) constitute a general and powerful framework for modeling continuous-time stochastic processes on networks. This makes them particularly attractive for learning the directed structures among interacting entities. However, if the available data is incomplete, one needs to simulate the prohibitively complex CTBN dynamics. Existing approximation techniques, such as sampling and low-order variational methods, either scale unfavorably in system size, or are unsatisfactory in terms of accuracy. Inspired by recent advances in statistical physics, we present a new approximation scheme based on cluster variational methods that significantly improves upon existing variational approximations. We can analytically marginalize the parameters of the approximate CTBN, as these are of secondary importance for structure learning. This recovers a scalable scheme for direct structure learning from incomplete and noisy time-series data. Our approach outperforms existing methods in terms of scalability.

The paper introduces a generalization of previous variational methods for inference with jumps processes; here, the proposal approximating measure to the posterior relies on a star approximation. In application to continuous-time Bayesian networks, this means isolating clusters of nodes across children and parents, in order to build an efficient approximation to the traditional variational lower bound. The paper further presents examples and experiments that show how the proposed approach can be adapted to structure learning tasks in continuous-time settings.  This is an interesting and topical contribution likely to appeal to the statistical and probabilistic community within NIPS. The paper is, in overall, well-written and reasonably well-structured. It offers a good background on previous work, helps the reader to understand its relevance and put its results in context within the existing literature. For these reasons, I wish to vote for weak acceptance. However, I have two major comments regarding rigor and content within some parts of the text. Specifically,  1 - The very first formulae presented around equation (1) seems wrong, unless I am missing something trivial. I pressume the negative lnP(Y) on the above equation has to be added, not substracted. As these are fairly common concepts and inequalities, this mistake sheds a lot of doubt to the reader, when approaching the sheer amount of calculations and formulae typical in such variational works.  2 - Linked to the above and in terms of reproducibility of the results. It is good to see in Section 3 that differential equations for the entropy and energy simplify to previously presented results in the literature. However, the bit that justifies acceptance in my opinion is the structure learning component in Section 4. In this case, pretty much all derivations are left to the appendices, the thin is rushed and feels hard to parse. The average reader will likely never look at appendices and this reduces accessibility. While it is understandable that not everything can fit, it would be valuable to add further analytic intuition behind formulae in (7)-(8).  Finally, there exist minor typos that warrant a re-read.  -----------  After rebuttal:  I appreciate the clarifications from the authors and still think this is an above average contribution likely to spark interest. In my opinion 6 is still an appropriate score.


RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Traditional computer graphics rendering pipelines are designed for procedurally generating 2D images from 3D shapes with high performance. The nondifferentiability due to discrete operations (such as visibility computation) makes it hard to explicitly correlate rendering parameters and the resulting image, posing a significant challenge for inverse rendering tasks. Recent work on differentiable rendering achieves differentiability either by designing surrogate gradients for non-differentiable operations or via an approximate but differentiable renderer. These methods, however, are still limited when it comes to handling occlusion, and restricted to particular rendering effects. We present RenderNet, a differentiable rendering convolutional network with a novel projection unit that can render 2D images from 3D shapes. Spatial occlusion and shading calculation are automatically encoded in the network. Our experiments show that RenderNet can successfully learn to implement different shaders, and can be used in inverse rendering tasks to estimate shape, pose, lighting and texture from a single image.

The paper proposes an end-to-end differentiable renderer which takes a voxelized shape as input and outputs the rendered image.  Pros: The idea is clean and simple. Compared to previous differentiable renderers, it can condition on lights and textures, and is extensible to inverse rendering tasks. When applying on inverse rendering tasks on human faces, it can generate images of better quality.  Cons: 1. The projection unit and 2D convolution (the rendering part) may not be necessary to learn. For example, in Phong shading, the normals can be computed using gradients of the depth map, and the dot operation of the lighting direction and normals can also be differentiable. In Barron & Malik 2015 they already have such differentiable renderer with lighting represented by spherical harmonics. It is also unclear to me why the visibility of objects need to be learned with the projection network. Don't you just need to pick the voxel closest to the camera to know visibility? 2. In Section 4.2, it may not be fair since the other networks need to learn the 3D transformations, while in RenderNet, the 3D transformations are explicitly done. 3. In figure 2, marching cubes is an algorithm to generate a mesh from 3D voxels. I am not sure why it is compared with renderers. 4. It would be more convincing if the reconstructed 3D shape in section 4.3 can be shown in a figure. 5. Lack of quantitative evaluation in section 4.3.  I also have some questions about the technical details: 1. I am confused about the output of the RenderNet. In line 136 the paper mentions the output is the shading, but in Figure 1 the output seems to be the Normal Map which is converted to shading. Line 137 also mentions that normal is an optional output. If you directly output shading, do you mean the Phong shading can be learned by the network itself without the need to explicitly specifying S = max(0, l*n + a) as in line 175? In other styles of rendering, do you also directly output the final shading? How do you use the input light position in those cases?  Please address the above concerns.


Learning Bounds for Greedy Approximation with Explicit Feature Maps from Multiple Kernels
Nonlinear kernels can be approximated using finite-dimensional feature maps for efficient risk minimization. Due to the inherent trade-off between the dimension of the (mapped) feature space and the approximation accuracy, the key problem is to identify promising (explicit) features leading to a satisfactory out-of-sample performance. In this work, we tackle this problem by efficiently choosing such features from multiple kernels in a greedy fashion. Our method sequentially selects these explicit features from a set of candidate features using a correlation metric. We establish an out-of-sample error bound capturing the trade-off between the error in terms of explicit features (approximation error) and the error due to spectral properties of the best model in the Hilbert space associated to the combined kernel (spectral error). The result verifies that when the (best) underlying data model is sparse enough, i.e., the spectral error is negligible, one can control the test error with a small number of explicit features, that can scale poly-logarithmically with data. Our empirical results show that given a fixed number of explicit features, the method can achieve a lower test error with a smaller time cost, compared to the state-of-the-art in data-dependent random features.

### Post-Rebuttal ###  I have read the rebuttal. Thank you very much for the clarifications.  ##################  # Summary The paper builds on [1] and proposes an algorithm for feature selection from an a priori specified finite dictionary/set of features. In particular, [1, Algorithm 3] propose an approach for minimization of the expected loss of a linear predictor that aims at finding a `good' sparse solution. The main idea of the algorithm from [1] is to iteratively add features by picking a previously unselected feature that amounts to the largest reduction in the expected risk. Then, a linear model is trained using the extended feature representation and afterwards the whole process is repeated. The authors use pretty much the same idea and take a large dictionary of features to represent the data. Following this, they run Algorithm 3 from [1] to pick `informative' features and generate a sparse feature representation. In particular, the step 2 from Algorithm 1 proposed by the authors can be shown to be `almost equivalent' to the first step of Algorithm 3 in [1].  The approach is motivated by the problems with scalability of kernel methods and related to learning with multiple kernels. In particular, a positive definite Mercer kernel can be represented as an inner product of possibly countably infinitely many features/eigenfunctions. For multiple kernels, it is possible to concatenate the feature representations corresponding to different kernels. For a Mercer kernel, the authors aim to find subsets of the features that produce a good approximation of the inner product defined by it.  There are two theoretical results: i) A bound on the Rademacher complexity of the hypothesis space defined by a dictionary of pre-selected features. This result builds on [3] and [4] and it is a rather straightforward derivation, following the previous results. ii) A bound on the expected risk of the estimator generated by the proposed algorithm. There are two terms that are bounded when deriving this result. The first term bounds the difference between the risk of an estimator generated by Algorithm 1 and the best estimator over the hypothesis space defined by a dictionary of features. The most important result for this term follows from [1]. The second term bounds the difference between the expected risk of the best estimator over the hypothesis space defined by the dictionary and that over the combination of multiple kernels with possibly infinitely many features. To the best of my knowledge, this piece is novel and interesting. Here it is important to note that similar bounds (finite vs. infinite dictionary) have been considered in [5] for random Fourier features.  The bound on the expected risk consists of three terms: i) a sample complexity term accounting for the concentration with respect to the number of instances (the standard \sqrt{n} rate), ii) an approximation term that accounts for the effect of adding more features and extending the feature representation (approximation properties improve exponentially, quite interesting), and iii) a spectral property term that accounts for the coefficient mass present over the (possibly infinitely many) remaining features defined on the whole feature space.   The empirical evaluation is performed using a dictionary of random Fourier features and the approach is compared to 5 baselines. The first baseline is the random Fourier feature approximation of the Gaussian kernel. The second and the third baselines sample a large pool of features and then perform feature selection using a scoring function and kernel alignment, respectively. The fourth approach is learning with the standard Gaussian kernel and the fifth one is multiple kernel learning with a combination of Gaussian and linear kernels. Overall the results show an improvement over the competing baselines and good test prediction times.  # Comments - The paper is well-written and easy to follow. The derivations are complete and understandable. The provided proofs are also legible and cite all the auxiliary results from previous work. - I recognize that there is a subtle difference between Algorithm 1 proposed by the authors and Algorithm 3 from [1]. In particular, [1] consider feature selection with respect to the expected risk and state that the result holds for the empirical risk, as well. The authors, on the other hand, consider empirical risk when picking a new feature and this makes their result from Theorem 2 novel and interesting. - Related to the previous point, Algorithm 1 should include a regularization term with a hyperparameter controlling the model complexity? - A limitation of the paper is the fact that the total number of features in the dictionary needs to be finite. A number of approaches have been developed for boosting features that work with infinite dictionaries [6-10]. I also think that approaches considered in [6, 8, 10] are quite related to the goal of this paper. Moreover, it would be very interesting to compare the effectiveness of feature selection to an approach like [9]. For a dictionary of features selected by sampling from a Gaussian distribution with thick tails the feature selection could be competitive with [9], where a mixture of Gaussian distributions models the spectral density of a stationary kernel and outputs a feature representation. - The spectral term from the bound in Theorem 2 deserves some more space. I do see that the spectral decay of the kernel can affect the eigenvalue decay but in this particular case we have coefficients of the model. It would be instructive to assume some decay (e.g., polynomial or exponential) and produce an estimate or rate for this term (this should be related to the remark on the Parseval's theorem). - Related to the previous point, the paper makes a strong assumption that a large dictionary of features specified a priori covers the `important' coefficients/features. In general, this is quite difficult to satisfy and the bound indicates that the concentration might not happen at all if the dictionary is not well-equipped to model the conditional dependence of interest. - A minor issue with the empirical evaluation is the fact that the bandwidth parameter was fixed and not cross-validated. The gap between the proposed approach and other baselines might be `over emphasized' as a result of this. - In Table 2, why is there such a large difference in test time predictions between LKRF, EERF, and the proposed approach? Are they not predicting with the same number of features? - Line 97, citation is needed for Parseval's theorem.   # References +[1] S. Shalev-Shwartz, N. Srebro, T. Zhang -- Trading accuracy for sparsity in optimization problems with sparsity constraints (SIAM Journal of Optimization, 2010) +[2] M. Mohri, A. Rostamizadeh, A. Talwalkar -- Foundations of machine learning (MIT Press, 2012) +[3] P. Bartlett, S. Mendelson -- Rademacher and Gaussian complexities: risk bounds and structural results (JMLR, 2002) +[4] C. Cortes, M. Mohri, A. Rostamizadeh -- Generalization bounds for learning kernels (ICML, 2010) --- -[5] A. Rahimi, B. Recht -- Uniform approximation of functions with random bases (IEEE, 2008) -[6] F. Huang, J. T. Ash, J. Langford, R. E. Shapire -- Learning deep ResNet blocks sequentially using boosting theory (ICML, 2018) -[7] C. Cortes, M. Mohri, U. Syed -- Deep boosting (ICML, 2014) -[8] C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, S. Yang -- AdaNet: adaptive structural learning of artificial neural networks (ICML, 2017) -[9] Z. Yang, A. Smola, L. Song, A. G. Wilson -- A la carte-learning fast kernels (AISTATS, 2015) -[10] D. Oglic, T. Gaertner -- Greedy feature construction (NIPS, 2016)


Improving Neural Program Synthesis with Inferred Execution Traces
The task of program synthesis, or automatically generating programs that are consistent with a provided specification, remains a challenging task in artificial intelligence. As in other fields of AI, deep learning-based end-to-end approaches have made great advances in program synthesis. However, compared to other fields such as computer vision, program synthesis provides greater opportunities to explicitly exploit structured information such as execution traces. While execution traces can provide highly detailed guidance for a program synthesis method, they are more difficult to obtain than more basic forms of specification such as input/output pairs. Therefore, we use the insight that we can split the process into two parts: infer traces from input/output examples, then infer programs from traces. Our application of this idea leads to state-of-the-art results in program synthesis in the Karel domain, improving accuracy to 81.3% from the 77.12% of prior work.

Summary: This paper provides a novel program synthesis approach by inferring the intermediate states: execution trace. Unlike conventional methods that treat program synthesis as an end-to-end task based on input/output (i/o) examples, in this manuscript, the authors attempt to infer the execution trace from i/o examples, and further synthesize the program given the i/o and execution trace. The motivation is clear and straightforward since generating programs from execution traces should be easier. Experiments on Karel program synthesis task show the effectiveness of the proposed approach. The research question is significant and very hard. It is good to see more program characteristics are included in the design of the neural network.   Strengths: 1.    Though straightforward, the proposed framework for program synthesis is novel. It splits the program synthesis into two steps by introducing the execution traces.  2.    The experiments are well-designed and the results are superior to the competing methods greatly. 3.    This paper is well-organized and the writing is good. Weaknesses: 1.    A few parts of this paper are not very clear or not sufficiently provided, such as the model details. Section 4.3 should be addressed more to make it clearer since some concepts/statements are misleading or confusing. 2.    The trace to code model is complex, which requires as many LSTMs as the number of input/output pairs, and it may be hard to be applied to other program synthesis scenarios.  3.    Apart from the DSL, more experiments on dominant PL (e.g., Python) would be appreciated by people in this research field.  Details are described as below: 1.    For a given program, how to generate diverse execution traces that can be captured by the I/O -> Trace model? Since execution traces are generated by running the program on N I/O pairs, it is possible that some execution traces have a large overlap. For example, in the extreme case, two execution traces may be the same (or very similar) given different I/O pairs.  2.    The authors involve the program interpreter in their approach, which is a good trial and it should help enhance the performance. However, I am curious about is it easy to be integrated with the neural network during training and testing? 3.    The concept of state is not very clear, from my understanding, it represents the grid status (e.g., agent position) and it is obtained after applying an action of the trace. Line 186-line 187, is the “elements” equivalent to “states”? or “actions”? More should be elaborated. 4.    Line 183-line 184, is it necessary to use embedding for only four conditionals (of Boolean type)? only 16 possible combinations.  5.    As depicted in Figure 3, if more I/O pairs are provided, the Trace->Code should be very complex since each i/o example requires such an LSTM model. How to solve this issue? 6.    In essence, the Trace->Code structure is a Sequence to Sequence model with attention, the only differences are the employment of I/O pair embedding and the max pooling on multiple LSTM. How are the I/O pair embeddings integrated into the computation? Some supplementary information should be provided.  7.    It is interesting to find that model trained on gold traces perform poorly on inferred traces, the authors do not give a convincing explanation. More exploration should be conducted for this part. 8.    It would be better if some synthesized program samples are introduced in an appendix or other supplementary documents.


Dimensionality Reduction has Quantifiable Imperfections: Two Geometric Bounds
In this paper, we investigate Dimensionality reduction (DR) maps in an information retrieval setting from a quantitative topology point of view. In particular, we show that no DR maps can achieve perfect precision and perfect recall simultaneously. Thus a continuous DR map must have imperfect precision. We further prove an upper bound on the precision of Lipschitz continuous DR maps. While precision is a natural measure in an information retrieval setting, it does not measure "how" wrong the retrieved data is. We therefore propose a new measure based on Wasserstein distance that comes with similar theoretical guarantee. A key technical step in our proofs is a particular optimization problem of the L 2 -Wasserstein distance over a constrained set of distributions. We provide a complete solution to this optimization problem, which can be of independent interest on the technical side.

Paper Summary: The authors quantify the quality of a dimensionality reduction map in terms of its precision and recall. Taking a topological point of view, they derive several interesting results such as for sufficiently low dimensional mappings, perfect precision and recall is not possible simultaneously (Thm. 1, Prop. 2). They further quantify the tradeoff showing that for maps with perfect recall, precision rate decays exponentially (as a function of the ‘gap’ between the intrinsic and the embedding dimension of the input data) (Thm. 2, 3). They further refine their quality estimate from simple precision-recall tradeoff to essentially how bad are the recall elements in terms of their (Wasserstein) distance. They provide the analogous tradeoff bound for Wasserstein distance (Thm. 4).  Review:  * I want to disclose to the authors that I have reviewed a previous version of this manuscript *  I really appreciate that the authors have significantly improved their manuscript and in my view merit a publication.   Overall I really like the paper. The authors provide a nice theoretical perspective of what to expect from (sufficiently low dimensional) mappings from an information retrieval (precision/recall-type) perspective. This has significant implications in regards to what kinds of embedding results can one expect from popular visualization techniques such as t-SNE. This paper proves that any such embedding method cannot achieve both precision and recall simultaneously and therefore in a sense the practitioner should be careful in interpreting the embedding results from such visualizations.   While I have not checked the proofs is detail (due to short review time and the appendix is 17 pages long), I do believe that the proof ideas are correct.   I have a few minor suggestions which I really hope the authors would include in the final version of their manuscript.  - As mentioned in my previous review, in the context of manifolds, analyzing the case of m < n < N (where m is the embedding dimension, n is the manifold dimension and N is the ambient dimension) feels very awkward to the reader. After all you are destroying the injectivity structure. While this difference is now discussed in Sec. 4, it would be nice to include a short discussion on merits of studying such maps in Sec. 1.1. This could be nicely motivated it by giving examples of visualization maps like t-SNE, and would overall strengthen the message of the paper.   - There are some recent theoretical results on the cluster structures that t-SNE reveals. See recent work by Arora et al. “An Analysis of the t-SNE Algorithm for Data Visualization” in COLT 2018. I believe that a map that recovers correct clusters can be viewed as the one which has a specific precision/recall tradeoff. It would be nice to place Arora’s result for t-SNE in this paper’s context. I believe including a short discussion on this in Sec 4 would make the discussion more “up-to-date”.  - One main criticism I have about the current work that the authors don’t explore or emphasize the that practical significance of their results. It would be nice to include a more detailed experiments section where they study (sufficiently) low dimensional mappings and explore some of the precision/recall tradeoffs. Having a detailed experimental section will significantly strengthen the overall message and applicability and significance of this paper.  Minor comments/typos: - [line 80] “??” needs to be fixed. - [lines 136-137] “perfection” should be “perfect” - [Eq. 1] For completeness, define Gamma function. - [line 182] probabilities q1 and q2 are basically unreadable (because of the domain of integration), consider rephrasing it more aesthetically. - [line 197] “Gamma(P_U, P_f^-1(V))” Consider using a different symbol than Gamma (as it is already used in Eq. 1). - [line 234] “we then project these 10 dimensional…”, exactly what projection map is being used? Linear? Non-linear?  - [line 282-283] while the C^infy case of Nash embedding is for a quadratic polynomial, the result in [28] is a (smooth) approximation to the C^1 Nash embedding result where the polynomial is in fact linear.   ---- I have read the authors' response.


Online Robust Policy Learning in the Presence of Unknown Adversaries
The growing prospect of deep reinforcement learning (DRL) being used in cyberphysical systems has raised concerns around safety and robustness of autonomous agents. Recent work on generating adversarial attacks have shown that it is computationally feasible for a bad actor to fool a DRL policy into behaving sub optimally. Although certain adversarial attacks with specific attack models have been addressed, most studies are only interested in off-line optimization in the data space (e.g., example fitting, distillation). This paper introduces a Meta-Learned Advantage Hierarchy (MLAH) framework that is attack model-agnostic and more suited to reinforcement learning, via handling the attacks in the decision space (as opposed to data space) and directly mitigating learned bias introduced by the adversary. In MLAH, we learn separate sub-policies (nominal and adversarial) in an online manner, as guided by a supervisory master agent that detects the presence of the adversary by leveraging the advantage function for the sub-policies. We demonstrate that the proposed algorithm enables policy learning with significantly lower bias as compared to the state-of-the-art policy learning approaches even in the presence of heavy state information attacks. We present algorithm analysis and simulation results using popular OpenAI Gym environments.

This submission introduces a novel framework for handling adversarial attacks on the observation space used by RL value estimation techniques. The new method is particularly interesting because it claims to address a few issues with current methods: the proposed method is attack-model agnostic, online, and adaptive. Although I find the work very interesting and I consider the problems that the paper is studying very important, I am not very familiar with most of the work that the authors have cited, in part because this work builds on many ArXiv pre-prints that I was not aware of.  I find this paper to be very hard to read for someone who is not familiar with the latest results in addressing adversarial attacks on Deep RL. The authors chose some weird notational standards, the algorithmic  framework is carelessly presented, and the main theoretical result is barely interpreted or compared to previous work. Although I appreciate the empirical results and the claimed improvements, I don't think this paper is very accessible.   Please see a list of specific comments.   L25, 55: is there a reason why some citations are listed separately? I.e. why [9], [10] and not [9, 10]? L28-31: Can you add some references to support the first statement in the Introduction? Particularly, the phrase "even small perturbations in state inputs can result in significant loss ..." seems like something that should have been quantified or empirically demonstrated already. L34: "the the" -> "the" L43: What does it mean to "optimize the observation space" ? Please fix this ambiguity. Table 1: It seems odd to add the column on "Mitigation", as all the listed methods support it. Also, what is the difference between a "robust method" and "a method supporting mitigation"?  L49: Please define the TRPO acronym (e.g. on L27) L55, L85, L174: You should avoid using the term "state-of-the-art" in isolation. Both the context (i.e. environment type, assumptions) and the comparison metric should always be clear whenever a method is considered state-of-the-art. L65: "in [20] in" -> "in [20] for" Section 2: The MDP notation is very sloppy here. E.g.  * It is unclear whether you assume deterministic policies,  * Are we dealing with episodic tasks? Is T fixed, probabilistic?  * The convention is most of the literature is to use "E_{\pi}" to define the return. Any particular reason why the more cumbersome "E_{s_0,a_0, ...}" was chosen here? Figure 1: Is there a reason you chose not to reference the ICLR 2018 paper? Also, why didn't you use the same illustration as they used, as the alternative one you provided does not seem more informative? It seems good to maintain consistency with the work you cite. L104 (eq (3)): I have hard time reading this expectation. What is the index i used for and what does it mean to condition on "m_i, s_i"? Please specify exactly the process for which we compute the return in Eq (3).  Section 3, first paragraph. I really like the motivation provided here, especially the use of the advantage function. Consider adding a sentence on the fact that the discussion applies mostly in the case where we have good estimates for the Q-value and not in the early stages of learning.  L148: "can not" -> "cannot" L154 (eq (5)): Just like Eq (3), this equation is overloaded and hard to interpret. First, how can an action be equal to a policy? I think you should replace "\pi_{*,t}" with "\pi_{*} (s_t)". Also, I am confused by the last part of the equation, when a_{master,t} is said to be part of the set "\{\pi_{nom}, \pi_{adv}}". What does that even mean? Please clarify. Figure (2): I believe it would be useful to explain what "m_i"'s correspond to in the proposed architecture. L164: "reported [13]" -> "reported in [13]". L215: Isn't the "actual advantage" the one defined under the true advantage? Consider using "observed advantage" instead.  L223: I find it odd that you refer to [11] here instead of the paper that derived those bounds: Kakade & Langford (2002). Moreover, I believe you should add a more extensive discussion here on how one should interpret Proposition 1 and 2 + comparison to Kakade & Langford's bounds and a proper comparison is granted here. L267: I think it would be enough to say here "s_i + U(-a,a) for a>0" instead of the convoluted presentation in the submitted paper.   L275: "Openai" -> OpenAI  ----After author feedback---- I would like to thank the authors for the additional feedback, which provides clarifications for the major conceptual concerns I have raised. Moving some important clarifying remarks from the Supplemental material to the main paper is a great suggestion. Also, I agree that improving the accessibility of the paper to a wider audience is necessary and will increase the value of this work.


PacGAN: The power of two samples in generative adversarial networks
Generative adversarial networks (GANs) are a technique for learning generative models of complex data distributions from samples. Despite remarkable advances in generating realistic images, a major shortcoming of GANs is the fact that they tend to produce samples with little diversity, even when trained on diverse datasets. This phenomenon, known as mode collapse, has been the focus of much recent work. We study a principled approach to handling mode collapse, which we call packing. The main idea is to modify the discriminator to make decisions based on multiple samples from the same class, either real or artificially generated. We draw analysis tools from binary hypothesis testing-in particular the seminal result of Blackwell [4]-to prove a fundamental connection between packing and mode collapse. We show that packing naturally penalizes generators with mode collapse, thereby favoring generator distributions with less mode collapse during the training process. Numerical experiments on benchmark datasets suggest that packing provides significant improvements.

This paper considers the idea of using a mini-batch of samples to train the discriminator (packing) to mitigate the mode collapse issue in training GANs. This idea has been around in the literature and is known as mini-batch discrimination [1]. As the authors have mentioned, there are many ways for applying this technique. The authors propose a very simple implementation that just uses a fully connected layer in the input to connect the packed samples to the discriminator network. It is surprising and interesting that such a simple implementation works in practice. It seems to be a counter-intuitive structure as it does not preserve the locality of the pixel information in samples of a batch. I do not know if this would be problematic in larger experiments that make use of convolutional units at the input layers of their discriminator. Moreover, such a structure is not necessarily permutation invariant, i.e. does not treat all the samples in the batch symmetrically.  Analysis:  The paper seems to be exaggerating the importance and implications of the theoretical analysis. First of all the analysis only applies to the TV distance. Note that TV distance is not used in practice for training GANs as it is not practical. Moreover, the kind of relationship that is proved between mode collapse and packing in the analysis seems to be objective dependent and might not be true for some plausible objectives. A simple example to see that such a relationship does not hold is when some variant of KL divergence is used as an objective. Note that KL(P^N, Q^N) = N KL(P,Q). Therefore, packing in this case does not seem to have any effect in changing the geometry of the objective. Thus, it is not appropriate to make general claims about the fundamental relationship between packing and mode collapse (see lines 10-11) based on these limited theoretical results.  The statements of the theorems 3 and 4 are very difficult to digest. Unfortunately, no intuition or clarification is provided to make them more understandable. Moreover, from the results of theorem 2 and 3, it is not readily clear that for a given epsilon, delta and tau if the lower bound of Theorem 2 would be greater than the upper-bound in Theorem 3 for some packing order m. Note that this is analytically crucial in order to make sure that a packed discriminator can distinguish mode collapse. This point does not seem to be proven explicitly, but has been mentioned as an implication of theorem 2 and 3; see lines 206-210 and 218-221. It seems that the authors only justify this point using plots in Figure 3 and Figure 11. Note that the epsilon is different for red and blue curves in Figure 3 and 11.   Experiments:  In the experimental section, the authors have done a good job of isolating the effects of packing on mode collapse. The improvements in mode collapse reduction, compared to the baseline methods, seem to be substantial, especially in the stacked MNIST and celebA experiments. However, the set of baseline methods do not include the most stable/state of the art methods/objectives for GANs. For example, the more stable set of methods that use the Wasserstein family of distances with some regularization, e.g. [2], are absent from the experiments. Therefore, it is difficult to confirm that such a large improvement is not due to the instability of the baseline methods in training.  Moreover, no comparison with the prior methods that use mini-batch discrimination is presented. This comparison is crucial because those methods stem from the same idea as packing.  It is also important to include some experiments on larger/more challenging GAN datasets such as CIFAR-10 or ImageNet. Note that the current set of experiments are more focused on evaluating the mode collapse. But the quality/sharpness of the generated samples is also important in generative models. For these larger datasets, the metrics such as inception or FID score could be used to measure the quality of the produced images as well as the mode collapse problems.   The authors also claim that the introduction of packing has little computational overhead. It is not readily obvious that the introduction of packing does not affect the convergence speed of the methods. It would be very useful if some experimental results are presented to prove the above claim.   Presentation:  In terms of clarity and presentation, the paper requires a lot of improvement. The authors keep mentioning the relationship between the ROC curve and mode collapse region as a fundamental and intuitive point in understanding the analysis. If that is really the case, maybe it would be better to reshuffle the content so that such this point is presented in the main body. As it was mentioned above, the theorems are not clearly presented and the reader is left alone to make sense of very sophisticated expressions without any clarifications or intuitions.  Some minor comments:  beta is not defined in Theorem 3. -   In Figure 11, even with different epsilons, at least a packing of order 5 is required to distinguish mode collapse. It would be interesting if the authors can provide a relationship between the required order of packing and the epsilon gap in H1 and H0.  [1] Salimans, Tim, et al. "Improved techniques for training GANs." Advances in Neural Information Processing Systems. 2016.  [2] Gulrajani, Ishaan, et al. "Improved training of Wasserstein GANs." Advances in Neural Information Processing Systems. 2017.