Learning Conditioned Graph Structures for Interpretable Visual Question Answering
Visual Question answering is a challenging problem requiring a combination of concepts from Computer Vision and Natural Language Processing. Most existing approaches use a two streams strategy, computing image and question features that are consequently merged using a variety of techniques. Nonetheless, very few rely on higher level image representations, which can capture semantic and spatial relationships. In this paper, we propose a novel graph-based approach for Visual Question Answering. Our method combines a graph learner module, which learns a question specific graph representation of the input image, with the recent concept of graph convolutions, aiming to learn image representations that capture question specific interactions. We test our approach on the VQA v2 dataset using a simple baseline architecture enhanced by the proposed graph learner module. We obtain promising results with 66.18% accuracy and demonstrate the interpretability of the proposed method. Code can be found at github.com/aimbrain/vqa-project.

-------------- Summary: --------------  The submission proposed a graph-based, question-image fusion mechanism for visual question answering. This approach learns an embedding of question and object detection features and uses it to predict question-image dependent adjacency matrices -- essentially a graph structure with nodes corresponding to object detections and edges weighted according to this learned metric. This learned module makes up the core technical contribution of the work as both spatial GCNs and the use of object detection features come from existing work.    -------------- Clarity: --------------  [C1] The portions of the method discussed in the submission are relatively clearly described. However, the model description stop abruptly after computing the GCN node representations. relying instead on Figure 2 and L92/L201 to convey the rest of the model (max-pool followed by full-connected layers). While these remaining components are quite simple, the choice of performing a max-pool is interesting but remains unjustified in the current approach.  [C2] The baselines are described incredibly quickly and without sufficient detail for the 'attentional' model to understand the comparison made in Table 1.  [C3] Significant space in the submission is used to display multiple large example images, Fig 1, 3, and 4, but there is limited discussion of these figures. Further, I'm not sure what are the take-aways from the examples in Fig 3 and 4. The authors point to these images as examples of interpretability, but the graph structure (the novel addition over standard attentional frameworks) doesn't seem to clearly be very interpretable.   [C4] Many of the figure captions are uninformative or incomplete.  [C5] The paper does not achieve state of the art results as claimed. I don't particularly find small improvements in SoTA to be that good of a measure of a paper anyway, but given that the claim appears in multiple places in the submission, it is worth pointing out.   -------------- Quality: --------------  [Q1] I'm all for the diversity of ideas and feel the fact that this submission performs similarly (or somewhat worse than) existing work while introducing a significantly different attentional mechanism is valuable. That said, given the lack of analysis or discussion it is difficult to find insights about the problem from this work.  [Q2] The proposed graph learning approach is not well explored. It is unclear what sort of spatial relationships the Guassian GCN kernels are learning. It is unclear what question-conditioned object relationships the graph learner module is representing. It is not discussed what effect the top-k or max-pool design decisions have on the model. It would be good if authors could provide some discussion of these.  [Q3] I do not see sufficient discussion or experiments of interpretability to substantiate the claims that the graph based attention model is more interpretable. It would be good for the authors to define what they mean by interpretable and provide some analysis based on human studies.  -------------- Originality: --------------  [O1] Some related work on attentional GCNs (i.e GCNs which learn edge weights) is missed. For example, GRAPH ATTENTION NETWORKS from ICLR 2018. I admit that ICLR and this submission deadline are fairy close (~2 weeks) but this work has been publicly available for some time.  [O2] The core novelty of this work is in the prediction of question-conditioned edge weights between object nodes in the graph structure. To my knowledge, this is fairly novel in the context of VQA.  -------------- Significance: --------------  [S1] Without additional analysis to identify the behavior, strengths, and weaknesses of the proposed method, I don't think the current submission would significantly advance understanding on the topic of VQA or question-conditioned attention mechanisms. Which is unfortunate given the direction itself may be valuable.  Post Rebuttal: The author response addressed some of my concerns and I've updated my recommendation accordingly. I urge the authors to put concerted effort into explore and exposing all the interesting design decisions and model behaviors of their new architecture in the final draft. The proposed solution is a fairly significant shift from existing approach that performs fairly similarly and it is useful to the community to illuminate why.


Probabilistic Matrix Factorization for Automated Machine Learning
In order to achieve state-of-the-art performance, modern machine learning techniques require careful data pre-processing and hyperparameter tuning. Moreover, given the ever increasing number of machine learning models being developed, model selection is becoming increasingly important. Automating the selection and tuning of machine learning pipelines, which can include different data preprocessing methods and machine learning models, has long been one of the goals of the machine learning community. In this paper, we propose to solve this meta-learning task by combining ideas from collaborative filtering and Bayesian optimization. Specifically, we use a probabilistic matrix factorization model to transfer knowledge across experiments performed in hundreds of different datasets and use an acquisition function to guide the exploration of the space of possible pipelines. In our experiments, we show that our approach quickly identifies highperforming pipelines across a wide range of datasets, significantly outperforming the current state-of-the-art.

This work presents an approach to deal with Automatic Machine Learning, optimizing not only the hyperparameters of a Machine Learning model but also different machine learning models to choose the best one, what it calls pipelines. This idea is implemented in Autoweka (SMAC), and also TPE can handle this task. In order to do so, it uses Probabilistic Matrix Factorization.  Related to:  Gaussian Process priors over this technique are used to predict the performance of pipelines in a new dataset. Mnih, Andriy, and Ruslan R. Salakhutdinov. "Probabilistic matrix factorization." Advances in neural information processing systems. 2008. The AutoML task is already solved: Kotthoff, Lars, et al. "Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA." The Journal of Machine Learning Research 18.1 (2017): 826-830. The surrogate model of AutoWeka is SMAC, which is proven to be a robust (and simple!) solution to this problem.  Strengths: -> Clever use of Probabilistic Matrix Factorization to solve this task. -> Experiments are performed in a plethora of datasets. -> Good ideas for further work.  Weaknesses: -> Contribution is low. No new models are presented. The contribution is employing Probabilistic Matrix Factorization to solve this problem. -> Clarity and organization. I think that this paper may be hard to read if you are not an expert of the area. Sometimes, variables are not provided an intuitive interpretation. -> The title is misleading. I would suggest to change it to Automatic Machine Learning via Collaborative Filtering. High-dimensional BO can exist in a problem with 10000 variables for example. -> In the experiments, some "hyper-hyperparameters" of this approach are given static values without justification. -> I think that this is a work in progress, at least not complete for a NIPS conference level. Many ideas remain to be implemented.  Does this submission add value to the NIPS community? : It has the potential, but not yet. I think that, besides the contribution is not as high as providing a new model, it is clever. Some of the presented further work must be added to this paper. I would highlight the addition of specific information of the datasets to the model.  Quality: Is this submission technically sound?: Not at all, borderline at least. Although it is clever to use Probabilistic Matrix Factorization, this task is already solved and only having taken into account the accuracy and not other features is a limitation. By adding these features that other approaches do not take into account, the soundness of the paper would be higher. Are claims well supported by theoretical analysis or experimental results?: No theoretical analysis is provided and experiments support well the results talking about quantity but I am not so sure about having set some of the variables of the experiments to a fixed value (Q=20, LR = 10-7, batch-size = 50). Is this a complete piece of work or work in progress?: Work in progress. Further work, really well explained, in the conclusions section must be presented in order for this paper to be accepted. Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?: I miss more analysis and explanations about the strenghts and weaknesses of the approach. Some are given, but it is not enough.  Clarity: Is the submission clearly written?: Not at all. This paper is hard to read for newcomers or even researchers that are not experts. Some areas are given too much detail (for example the explanation of the EI acquisition function, I would just cite it). With that gained space, I will clearly explain what does \mathbf{X}, \mathbf{x}_i and \mathbf{x}_j mean. There are also some quantities that are not explained, as const in \mathbf{NLL}, the implications of \eta, the learning rate, but above all, in the experiments section, an interpretation of Q, \eta and batch size values chosen must be provided. Is it well organized?: I would change some things to improve organization. Regarding the abstract, the first paragraph, until "learning community" is the standard literature of Bayesian Optimization that applies to all BO papers. In this case, all this information that does not describe this particular paper represents more than half of the abstract. Also, the other paragraph lacks cohesion with the first one. Regarding introduction, the third paragraph "Bayesian optimization techniques" should be a continuation of the first one, for coherence. Other critical problem is that, based on the title, I expected a complete different thing in the paper until I read from line 44 on advance. This information should be stated earlier or the title should be changed. I encourage to rewrite the abstract to make clear the pipeline topic. Another issue, section 3.1 is not expected when you finished reading section 3. I expected what is explained in section 3.2. I would change the last paragraph of section 3, introducing section 3.1, to add cohesion. Does it adequately inform the reader?: By solving the mentioned issues, I think so.  Originality: Are the tasks or methods new?: No. The methodology is built from well-known techniques. Is the work a novel combination of well-known techniques?: Yes, Probabilistic Matrix Factorization, Gaussian Processes and ML algorithms. Is it clear how this work differs from previous contributions?: Yes, I have not read nothing similar nor the related work presents nothing like the contribution of the authors. Is related work adequately cited?: Yes, the authors did a good job here.  Significance: Are the results important?: Currently not. But, if more features are added of dataset, it could be interesting to analyze that results. Are others likely to use the ideas or build on them?: Same answer as before. Does the submission address a difficult task in a better way than previous work?: Not in my opinion. I prefer the SMAC methodology. But I admit that this opinion can be subjective. I prefer that the model can handle all and here the addition of Probabilistic Matrix Factorization adds more complexity. Does it advance the state of the art in a demonstrable way?: Currently not. I am not sure about the fixed values of the experiments and the comparison with others. Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?: It does provide some unique conclusions, but further work must be implemented. Arguments for acceptance:  -> New methodology based on GPs and Probabilistic Matrix Factorization for Automatic Machine Learning.  Arguments against acceptance:  -> It is just a combination of well-known techniques that does not solve a new problem and its potential is not fully developed (further work). -> I would add more experiments. -> Title is misleading. -> Clarity and Quality are poor. Issues must be corrected. Line 138. Y -> \mathbf{Y}  -> Abstract, line 14: Name the state of the art techniques. -> Abstract, line 12: Define pipeline. -> Introduction, line 22: That is an assumption that you are considering. Define that fact as an assumption, parameter space should be formalized. -> Equation of line 27: Put a number and \mathbf{x} is a matrix, so capital letter. -> The instatiations of the pipelines that you are considering are a restriction for the experiments wrt the theory. State that this is a constraint. -> Line 57: Demonstrate empirically. -> Line 119: That "idea" is a hypothesis that you are considering. Define that idea as a hypothesis. -> Line 153: Why do you choose the Squared Exponential Kernel over the Matern kernel if the Matern kernel is preferred? -> Complexity analysis: How does this methodology scale wrt pipelines/datasets?  To sum up, I would consider giving a borderline to this paper if all these demands are satisfied.  If there are not satisfied I can not recommend this paper although I liked the idea of using Probabilistic Matrix Factorization.


Generalized Inverse Optimization through Online Learning
Inverse optimization is a powerful paradigm for learning preferences and restrictions that explain the behavior of a decision maker, based on a set of external signal and the corresponding decision pairs. However, most inverse optimization algorithms are designed specifically in batch setting, where all the data is available in advance. As a consequence, there has been rare use of these methods in an online setting suitable for real-time applications. In this paper, we propose a general framework for inverse optimization through online learning. Specifically, we develop an online learning algorithm that uses an implicit update rule which can handle noisy data. Moreover, under additional regularity assumptions in terms of the data and the model, we prove that our algorithm converges at a rate of O(1/ √ T ) and is statistically consistent. In our experiments, we show the online learning approach can learn the parameters with great accuracy and is very robust to noises, and achieves a dramatic improvement in computational efficacy over the batch learning approach.

Inverse optimization deals with finding the underlying objective function of an optimization problem (mainly in a parametric setup) from observation of inputs and correspoding optimal solutions as the data.  This can be done in a batch or in an online fashion from the data. The authors provide an online approach for doing this and prove that it can achieve a regret O(1/sqrt(T)) as a function of data size T.  1. I believe the paper is not written very well. For example, the underlying idea of inverse optimization is not clearly explained as in main references such as [1]. Moreover, the problem has been posed in a quite general setup but at the end the authors have focused on the convex and strongly convex setup where they have used the result already in the literature with some minor modifications.  2. the definition of the function h(x,thet_1,theta_2) at end of page 4 is not clear at all. What is the role of the variable "u" and where does it appear?   3. There are some typoes in the text that should be corrected: 3.1. batching learning --> batch learning 3.2. Also. estimation --> Also, estimation 3.3. rational making decison --> rational decision makers  4. the definition of the function h(x,thet_1,theta_2) at end of page 4 is not clear at all. What is the role of the variable "u" and where does it appear?  5. since the authors have focused on the convex case, there was no need to consider S(u, theta) since the optimal solution would be unique thus S(u, \theta) would be a single point.  6. As far as I see, the main novelty of the paper lies in the fact that the authors have notice that the inverse optimization in an online formulation can be posed as an online learning problem with a cost function that relates to the objective of the optimization problem and have posed it in the context of learning with regret. Also, the proved results seem to be a simple extension of the techniques in "Brian Kulis and Peter L. Bartlett. Implicit online learning".   However, this is not clearly written and connected in the paper. So, the paper needs to be modified to clearify this connection.   7. In the "Problem Statement" in Section 2, it is necessary to explain that the inverse optimization is considered in a parametric setup where \theta is the parameter of the objective function to be learned. This is not clearly epxlained in the paper.


Hybrid Knowledge Routed Modules for Large-scale Object Detection
The dominant object detection approaches treat the recognition of each region separately and overlook crucial semantic correlations between objects in one scene. This paradigm leads to substantial performance drop when facing heavy long-tail problems, where very few samples are available for rare classes and plenty of confusing categories exists. We exploit diverse human commonsense knowledge for reasoning over large-scale object categories and reaching semantic coherency within one image. Particularly, we present Hybrid Knowledge Routed Modules (HKRM) that incorporates the reasoning routed by two kinds of knowledge forms: an explicit knowledge module for structured constraints that are summarized with linguistic knowledge (e.g. shared attributes, relationships) about concepts; and an implicit knowledge module that depicts some implicit constraints (e.g. common spatial layouts). By functioning over a region-to-region graph, both modules can be individualized and adapted to coordinate with visual patterns in each image, guided by specific knowledge forms. HKRM are light-weight, general-purpose and extensible by easily incorporating multiple knowledge to endow any detection networks the ability of global semantic reasoning. Experiments on large-scale object detection benchmarks show HKRM obtains around 34.5% improvement on VisualGenome (1000 categories) and 30.4% on ADE in terms of mAP. Codes and trained model can be found in https://github.com/chanyn/HKRM.

This paper is about an object detection method that exploits both implicit and explicit knowledge to increase the semantic consistency of detections in a scene. Starting from one of the current state of art works in object detection (Faster-RCNN) this approach extract object proposal from its RPN, and produce an augmented feature where explicit and implicit knowledge, in the form of feature graphs, is integrated before producing the final classification and bounding box regression.  While exploiting the knowledge of spatial, similarity, attributes and context has a long history in the literature (the work cite several of them), this work has the merit of producing adaptive region-to-region graphs and put its foundation on a modern object detection framework that is known to be more reliable than in the past. Moreover the use of Visual Genome and the scene parsing dataset ADE enable the possibility of seeing a marked improvement that was probably not possible in the past.  Strengths: + Shows a framework able to integrate explicit and implicit knowledge in a modern framework for object detection such as Faster-RCNN. + Very well written and easy to read. + Enables further studies among this important line of research.  Weakness: - The method is tested on ADE and Visual Genome while all previous work on object detection typically use COCO, Pascal VOC 2007 or ILSRVC detection challenge datasets. While they may be considered less challenging than ADE and Visual Genome, it would have been interesting to observe if there is any advantage in using the encoded knowledge in such more simplified dataset. Moreover, it would enable comparison with other state of the art works. - As a continuation of the previous point, ADE is a dataset for Image parsing, which may be biased among different scenes and favour the performance of the proposed method. - The approach is claimed to be "light-weight, general-purpose and extensible by easily incorporating multiple knowledge to endow any detection networks the ability of global semantic reasoning.". Beside Faster-RCNN, since it is an uncommon dataset for object detection, it would be interesting to see more comparisons with other state of the art methods that have better performance than Faster-RCNN. For instance, YOLOv3 and SSD. Such other methods may already have better performance than the proposed method without using any knowledge. Even in such a case, it would not decrease the significance of the paper, but would put more in focus the actual performance that is obtainable by exploiting the knowledge by the proposed method. - The novelty of integrating knowledge is not new.  Suggestion: - I would like to suggest the authors to release the source code and/or the trained models.  Very minor issues: - row 215: Faser-RCNN -> Faster-RCNN - Faster-rcnn is written differently among the paper.  —————————- Thanks to the authors for addressing my concerns. They are all well addressed in the rebuttal.


Unorganized Malicious Attacks Detection
Recommender systems have attracted much attention during the past decade. Many attack detection algorithms have been developed for better recommendations, mostly focusing on shilling attacks, where an attack organizer produces a large number of user profiles by the same strategy to promote or demote an item. This work considers another different attack style: unorganized malicious attacks, where attackers individually utilize a small number of user profiles to attack different items without organizer. This attack style occurs in many real applications, yet relevant study remains open. We formulate the unorganized malicious attacks detection as a matrix completion problem, and propose the Unorganized Malicious Attacks detection (UMA) algorithm, based on the alternating splitting augmented Lagrangian method. We verify, both theoretically and empirically, the effectiveness of the proposed approach.

This paper proposes a new kind of attack on recommender systems that the authors call unorganized malicious attacks. In these attacks, each item can only attacked by a limited number of users. The authors propose an algorithm for solving it and show global convergence of it. Finally, the authors evaluate their algorithm on two (semi-)synthetic tasks and one real-world one.  My main issues with this paper are that (i) assumptions are not well motivated and justified, (ii) the theoretical results are not well aligned with the task of finding malicious users and (iii) the empirical evaluation does not have any recent baselines. With all the major revisions needed, I vote to reject this submission. Below are my detailed comments.  Significance. The paper lacks proper motivation and justification for the scenario of malicious unorganized attacks that is studied in it. Why is this scenario more important or realistic than shilling attacks? To what degree does it generalize those? I think answering those questions right away in the introduction would strengthen the paper tremendously. Moreover, especially the formulation of the optimization problem (lines 148-159) needs proper justification. For example, why do we care about the rank function? Moving on to the theoretical results, I don't really feel there are connected well to the problem of finding malicious users, e.g., users with |Y_ij| != 0. The results give us convergence in terms of the Frobenius norm on Y, but I would like to see convergence in terms of the L_1 norm here.   Quality. Regarding the empirical evaluation, I am not convinced that the baselines offer a fair comparison. Especially the N-P and k-means based methods are very simple. The most recent baseline is 5 years old which would need some justification. I would also expect to see Maximum-margin matrix factorization as a baseline here.   Originality. This work is based on standard approaches for matrix completion and convex analysis. The main novelty is the introduction of the attack matrix Y. Hence, I would consider the work in this paper to be incremental overall.  Clarity. I had a hard time reading through the paper because relevant information was scattered throughout and it also lacked a clear focus. I would rewrite the paper so that it is clear from the very begining that the problem you are trying to solve is to find users that were part of an unorganized malicious attack. This will make the theory and introduction easier to write. Right now, it's not clear what exactly the problem is that the authors seek out to solve. For example, in the introduction the paper says "in this paper, we investigate a new attack model" -- yes, but what is the goal? There are also various typos ("recommender system has" -> "recommender systems have", "there are many works" -> "there is much work", "is a (or partial) matrix" -> "is a partial matrix" etc.) that should be fixed. Moreover, the first paragraph in the introduction is   unnecessarily broad and should be more focused, then the last paragraph can be omitted since it is already clear from the section headings what follows. The problem formulation should follow right after the introduction, I would move related work to the end. I would also start with the new unorganized attack scenario and then later talk about the other attack strategies. Section 3 should also be written to make the assumptions and model more prominent.  Author rebuttal: Thank you for clarifying my concerns. However, the revisions needed / proposed would be substantial, and the paper would need another round of reviewing. Hence, I still vote to reject this paper.


Visual Object Networks: Image Generation with Disentangled 3D Representation
Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel our image formation process into three conditionally independent factors-shape, viewpoint, and texture-and present an end-to-end adversarial learning framework that jointly models 3D shapes and 2D images. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e., silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic texture to these 2.5D sketches to generate natural images. The VON not only generates images that are more realistic than state-of-the-art 2D image synthesis methods, but also enables many 3D operations such as changing the viewpoint of a generated image, editing of shape and texture, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.

This paper describes a generative model for image formation, with disentangled latent parameters for shape, viewpoint and texture. This is in keeping with the vision as an inverse graphics problem, where image generation is formulated as a parameter search in model space, that when rendered, produces the given image. The difference between the rendered image and the original image is used to train the model. Using inverse graphics as inspiration, this paper learns the following models: 1. An voxel generator that can map the latent 3D shape code to a voxellized 3D shape 2.A differentiable projection module that converts the output of 1 to a 2.5D sketch (depth map) and a silhouette mask, conditional on a latent representation of the required viewpoint 3.A texture generator, that can map the output of 2 to a realistic textured image, conditional on a latent representation of the required texture 4.A 2.5D sketch encoder that can map a 2D image to a 2.5D sketch 5.A texture encoder that maps the texture of the object in a 2D image into a texture latent code The models are learnt adversarially using GANs, and without the need for paired image and shape data using the now common cycle consistency constraints. The training of the GANs are improved using the recently described WGAN that enforces the Lipshitz constraint with a gradient normalizing term, which leads to a better optimization landscape and prevents mode collapse. Results on car and chair datasets indicate qualitative and quantitative (Frechet Inception distance) improvements in image generation by varying pose, texture and shape, compared to competing techniques.  The trained models and disentangled latent parameters of shape, texture and viewpoint allow a mix and match between the same.   Quality This work demonstrates a generative model of image formation with disentangled factors that can be trained without paired training data between factors. The work is well-reasoned and represents an advance in tying together the factors responsible for image formation, and how to train generative models to train each of those, end-to-end.  Clarity The paper is well written, with the motivation GAN losses well-explained.  The following points require the writers’ attention: Line 26 - you mean virtual reality and not visual reality Line 103 - do you classify individual voxels, and not the whole voxel shape as real/fake? More explanation required. Section 3.2 - More details required on the projection module, perhaps a diagram. And how is it differentiable? Figure 5 (b) Texture transfer does not seem to work as well in the first row, where the white car’s texture is transferred. The two cars to whom the texture has been transferred look more grey/silver than white. This could be a consequence of lighting change. Perhaps lighting is another factor that requires consideration as a factor in a disentangled generative model of image formation. Figure 5 (c) It would clarify things if you show the 2 objects you are interpolating between. Results on the 3rd row are not very impressive - the front of the car seems mashed up, and not a very smooth transition between shapes. An explanation in the text is required. Why do you not use the W-GAN loss in the loss functions for 2.5D GAN and texture GAN, when you use it for the shape GAN? In addition to the applications listed in 4.3, it would be useful to see if you can apply this to objects for which the 3D model does not exist during training. Can you go from 2D images (from 1 or 2 viewpoints) of a new object (unseen during training), and go from the 2D images to the 3D shape, and then generate 2D images of the object from a new viewpoint? This would be a test of how well your model is able to generalize over shapes.   Originality  This paper describes an incremental advance in the development of generative models of image formation. It has enough originality to warrant publication at NIPS.  Significance The inverse graphics style generative model with disentangled factors for image formation, that can be trained without paired training data between the factors is a fundamental development in the field of Computer Vision, and has the potential for significant advances in applications like virtual reality, augmented reality and robotics. This paper presents a well written and comprehensive treatment of an incremental improvement in this direction.


The Cluster Description Problem -Complexity Results, Formulations and Approximations
Consider the situation where you are given an existing k-way clustering π. A challenge for explainable AI is to find a compact and distinct explanation of each cluster which in this paper is assumed to use instance-level descriptors/tags from a common dictionary. Since the descriptors/tags were not given to the clustering method, this is not a semi-supervised learning situation. We show that the feasibility problem of testing whether any distinct description (not necessarily the most compact) exists is generally intractable for just two clusters. This means that unless P = NP, there cannot exist an efficient algorithm for the cluster description problem. Hence, we explore ILP formulations for smaller problems and a relaxed but restricted setting that leads to a polynomial time algorithm for larger problems. We explore several extensions to the basic setting such as the ability to ignore some instances and composition constraints on the descriptions of the clusters. We show our formulation's usefulness on Twitter data where the communities were found using social connectivity (i.e. follower relation) but the explanation of the communities is based on behavioral properties of the nodes (i.e. hashtag usage) not available to the clustering method.

The main contributions of the paper are: - Formulating a cluster description problem as a combinatorial optimisation problem to find descriptions for clusters that are descriptive and non-overlapping - Showing NP-completeness of the problem, even if there are only two clusters - Providing  ILP encodings modelling (also extended) variants of the problem  - Providing a tractable subclass of the problem, and together with a polynomial time  algorithm for this subclass  Pros: - A novel problem that is relevant by providing explanations for previously computed clusterings - Well-written paper with a clear focus and clear contributions  Cons:  - In lines 102–103, it is stated that some version of DTDF can be reduced to SAT. Should not this be 2-SAT? Since DTDF is shown to be NP-complete, all versions of DTDF can be reduced to SAT, right? The NP-completeness should hold in general (if not, it should not be possible to devise and ILP encoding for the problem).  - The experimental section is missing some detail (see minor comments below)  Quality: The paper is technically sound and the proofs for complexity parts are provided in the supplementary material. While I did not check all the details in the supplementary, the constructions for reductions seem sound. The  cluster description problem is solved in general using a ILP formulation and also for a tractable subclass. The experimental evaluation supports the theoretical results.  Clarity: The paper is well written and I found it a pleasure to read. Being a reader with a strong background in combinatorial methods I found the level of details sufficient and discussion very concise, and I think that non-experts on combinatorial optimisation should be able to follow the paper.   Originality: The problem definition is novel to my knowledge. The work is placed well in context with other (not too similar) works.   Significance: The results and the new problem formulation have potential for significance. It is relevant for us to find new methods to understand and explain the solutions provided by machine learning algorithms.   Minor:  -line 61: descriptive -> description - the Twitter example with hashtags related to Trump-Clinton election could use some explanation about the meanings of hashtags etc. While currently these should be clear to many if not all, in couple of years it might not be as clear why #MakeAmericaGreatAgain should not go together with #ImWithHer. - It was not really clear to me, whether the IPL encoding or Algorithm 1 was used in the experimental section.  - Regarding Figure 3: how many tags were there in the left table? How many nodes were there in the right table? What is the meaning on *? Is the number of nodes forgotten correct for 25 tags and k=60?  AFTER AUTHOR RESPONSE: The response answered questions raised in the review in a satisfactory manner.


Sharp Bounds for Generalized Uniformity Testing
We study the problem of generalized uniformity testing of a discrete probability distribution: Given samples from a probability distribution p over an unknown size discrete domain Ω, we want to distinguish, with probability at least 2/3, between the case that p is uniform on some subset of Ω versus -far, in total variation distance, from any such uniform distribution. We establish tight bounds on the sample complexity of generalized uniformity testing. In more detail, we present a computationally efficient tester whose sample complexity is optimal, within constant factors, and a matching worst-case information-theoretic lower bound. Specifically, we show that the sample complexity of generalized uniformity testing is Θ 1/( 4/3 p 3 ) + 1/( 2 p 2 ) .
ori structure of the underlying distributions to obtain significantly improved samples complexities [BKR04, DDS + 13, DKN15b, DKN15a, CDKS17, DP17, DDK16, DKN17].
Perhaps surprisingly, the natural setting where the distribution is arbitrary on a discrete but unknown domain (of unknown size) does not seem to have been explicitly studied before the recent work of Batu and Canonne [BC17]. Returning to the specific problem studied here, at first glance it might seem that generalized uniformity testing and uniformity testing are essentially the same task. Naively, one might attempt to apply the existing uniformity testers directly without explicit knowledge of the domain. This nearly works, as standard testers do not need to make use of any particular information about the names of domain elements. However, these algorithms do make use of the domain size in a critical way. This difficulty is not so easy to overcome. In fact, as was shown in [BC17], the sample complexity with an unknown domain size is significantly different. Specifically, [BC17] gave a generalized uniformity tester with expected sample complexity O(1/( 6 p 3 )) and showed a lower bound of Ω(1/ p 3 ). This should be compared to the O(n 1/2 / 2 )-sample tester for distributions on domains of size n. Of particular interest here is that distributions p with support size n can have 1/ p 3 as large as n 2/3 , making the problem with unknown domain substantially harder in the worst case.

This paper considers the recently introduced problem of generalized uniformity testing where the goal is to test if a distribution is uniform on a support of unknown size or is it \epsilon-far in total variation distance from all uniform distributions.  The main result of the paper is a complete characterization of sample complexity of this problem. It is shown that this sample complexity is   \Theta( 1/(||p||_3 \epsilon^{4/3}) +1/(||p||_2\epsilon^{2}) ),  when the unknown distribution is p. Note that when p is uniform on n, this sample complexity becomes   \Theta( n^{2/3}/\epsilon^{4/3}) +n^{1/2}/\epsilon^{2})),  which exceeds \sqrt{n}/\epsilon^{2} (the sample complexity of the problem when the support of the distribution is known to be n) if n^{1/4}>1/\epsilon.   The algorithm achieving this bound, first uses estimates for ||p||_2 and ||p||_3 to ensure that they satisfy ||p||_2^4 = ||p||_3^3 (which must be the case if the distribution p is uniform) and define n= 1/||p||_2^2. Then, it proceeds to accomplish uniformity testing assuming that the support is n. When p is indeed uniform, it seems that any standard uniformity test for support-size n can be used and its analysis will go through. However, the technical difficulty arises from the alternative where we might be off in our estimate of support. The main contribution of this paper is to identify appropriate statistic that is robust to this uncertainty.  Specifically, the paper presents a different algorithm for the small \epsilon (n^{1/4}<1/\epsilon) and large \epsilon (n^{1/4}>1/\epsilon) regimes. For large epsilon, it evaluates ||p||_3^3-||p||_2^4 and compares it with \epsilon^2/n^2. An interesting component of analysis of this case shows how to control the variance, which depends on  ||p||_4, ||p||_5. Note that it is not possible to estimate these quantities with the available sample budget. Yet, the proof shows that we can ensure the desired bound on the quantities within our sample budget; an extra check is included in the algorithm to ensure from estimates of  ||p||_4, ||p||_5 that the variance is small as desired.  For small epsilon, we have the usual sample budget of O(n^{1/2}/\epsilon^2). But it is not clear if the usual collision based test can be used here since we don’t have a handle on its variance when p is not uniform. Instead, the authors propose a different test which first takes a sample of length n and ensure that all symbols appear at least log n times. Note that in this regime we can draw n samples since n
Reviewer 2
The authors consider the problem of generalized uniformity testing. This problem -- which is a natural extension to the standard uniformity testing problem -- was recently been proposed and studied by Batu and Canonne (2017). In standard uniformity testing problem, the task is to distinguish the case where the samples are coming from (the given) uniform density, or are being generated from a probability measure that is epsilon-far from the mentioned uniform density (in terms of total variation distance). In generalized uniformity testing, however, the tester should still accept if the distribution is uniform over a "subset" of the domain.   As observed and proved by Batu and Canonne (2017), the sample-complexity bounds for generalized uniformity testing exhibit a somewhat surprising divergence from those of uniformity testing. In this submission, the authors extend the results of Batu and Canonne (2017) by improving both the lower bound and the upper bound (for the sample complexity), essentially characterizing the sample complexity of generalized uniformity testing. Furthermore, the provided algorithm is computationally efficient.   The improvement in the lower bound is achieved using the standard lower bound for uniformity testing, as well as the use of the poissonization technique which is known for proving lower bounds (still, using the poissonization technique requires a non-trivial construction).  The improvement in the upper bound is achieved using multiple delicate improvements over the work of Batu and Canonne (2017), which is on a high level based on comparing the second and the third moments.   Overall, I think this a solid theoretical result about an interesting and important problem. The ideas are also well described. I think for this conference, it would make sense to add some more motivations about why generalized uniformity testing is important (e.g., use cases)
Reviewer 3
This works studies the problem of generalized uniformity testing that was recently introduced by Batu and Canonne (FOCS'17). In this problem, one is given sample access to a distribution over a discrete domain and is asked to determine if the distribution is uniform over some subset of the domain or far in total variation distance from any such distribution. Batu and Canonne showed that O(1)/(|p|_3 eps^6) samples suffice to test up to total variation distance eps, where |p|_3 is the L3 norm of the underlying distribution which is n^(2/3) for a uniform distribution on n elements.  The main result of this paper is pinning down the exact sample complexity of the problem. In particular, it is shown that  1/(|p|_3 eps^(4/3)) + 1/(|p|_2 eps^2) samples are both necessary and sufficient for this problem.  The upper bound works by examining two cases based on the magnitude of eps. It tests which of the terms 1/(|p|_3 eps^(4/3)) and 1/(|p|_2 eps^2) is larger and uses different strategies for each of the cases. To distinguish between them, it uses a method to (roughly) approximate within constant factors the L2 and L3 norms of the distributions by counting collisions.   In the first case where eps is large, the algorithm works by checking whether |p|_3^3 - |p|_2^4 is 0 or not by computing more fine-grained estimations of the norms. A key lemma that enables this algorithm is that the difference is 0 for a uniform distribution and sufficiently far from 0 when the distribution is far from uniform.   The second case where eps is small is easier, as one can afford to recover the support of the distribution. In that case, a standard uniformity tester is performed on the recovered support.  In addition to the upper bounds, the paper shows a lower bound of 1/(|p|_3 eps^(4/3)) + 1/(|p|_2 eps^2). For  the term 1/(|p|_2 eps^2), the standard uniformity testing lower-bound by Paninski applies here. In that lower bound, the distribution is either uniform or (1+eps)/n over a random half of the support and (1-eps)/n on the other half. To get the lower bound for the term 1/(|p|_3 eps^(4/3)), this work uses a similar construction over a random subset of the whole domain. To establish the lower-bound, a Poissonization argument is used together with the method of shared information of Diakonikolas and Kane (FOCS'16).  Overall, I think that this is a very interesting work settling the sample complexity of a very well motivated and natural problem. It improves both upper and lower bounds significantly over the previous work, giving very simple and natural algorithms for testing. The paper is well written and self contained with very intuitive explanations and presents many techniques in simple arguments behind the proofs.  I enjoyed reading the paper and support acceptance.


Universal Growth in Production Economies
We study a simple variant of the von Neumann model of an expanding economy, in which multiple producers make goods according to their endowed production function. The players trade their goods at the market and then use the bundles acquired as inputs for the production in the next round. The decision that players have to make is how to invest their money (i.e. bids) in each round.
We show that a simple decentralized dynamic, where players update their bids proportionally to how useful the investments were in the past round, leads to growth of the economy in the long term (whenever growth is possible) but also creates unbounded inequality, i.e. very rich and very poor players emerge. We analyze several other phenomena, such as how the relation of a player with others influences its development and the Gini index of the system.
We show that a simple decentralized dynamic, where players update their bids proportionally to how useful the investments were in the past round, leads to growth of the economy in the long term (whenever growth is possible) but also creates unbounded inequality, i.e. very rich and very poor players emerge. We analyze several other phenomena, such as how the relation of a player with others influences its development and the Gini index of the system.

This paper studies the evolution of a growing production economy under a certain dynamic model of participant behavior.  The model is a minor variation on a classic model of von Neumann.  There are n firms, each producing their own type of good, and each with a budget of money.  Each firm i also has the ability to convert units of any other good j into units of their own good, at some given conversion rate a_{ij}.  Each round, the firms bid on the goods produced by other firms.  This results in exchanges of goods for money (more on this below).  After this, the firms convert any acquired goods into their own type of good, and the process repeats.  The economy is said to expand if the total quantity of goods increases without bound.  Rather than study the optimal equilibrium growth, the paper studies the following dynamics.  In each round, firms bid on goods in proportion to how much the goods contributed to their production last round.  Also, the bidding is resolved by proportional share: each firm's output is divided in proportion to the bids received, and every firm pays their bids.  The main result of the paper is that if there is any production path such that the economy expands, then this proportional dynamics will also expand.  Furthermore, the rate of expansion converges to the optimal rate of expansion.  This result is complemented with a variety of simulations and structural observations.  E.g., growth can oscillate, and some firms will grow more quickly than others in the limit.  The authors relate this last observation to notions of income inequality, and simulate the evolution of the GINI index in simple economies.  Comments  I like the idea of studying growth in pure production economies.  This is a classic topic in market theory that has not received much (or any) attention in the TCS literature.  The model studied here is a simple and natural starting point.  I also like the agenda of understanding natural dynamics by which firms learn to bid and bring the economy toward equilibrium.  This has been a successful line of inquiry for markets with consumption, so it makes sense to apply it here as well.  The main result --- that the proposed dynamics leads to optimal growth --- is clean and appealing.  The proof is short and slick, following a potential-function argument.  While similar arguments have been used for other market types, the setting here is different enough that new ideas were needed.    The strength of the main result depends on how much the reader likes the "trading-post proportional-bidding" dynamics being studied.  Personally I think the model is reasonable, as it blends together two ideas (proportional sharing of outcomes and proportional bidding) that have been well-studied in related contexts.  I do find myself wondering if there is evidence that this dynamics is predictive.  Do we expect that markets actually behave this way?  Or is the point of the paper that a market-maker should encourage production firms to behave this way?  More motivation would be welcome.  I found the other content in the paper less exciting.  For instance:  - I find it natural that some firms grow more quickly than others, since there is no consumption, taxation, or limited labor pool.  Since those forces are missing here, it's unsurprising that firms on the "efficient" production path will extract much of the gains.  Because of this, I don't view the paper as saying too much about inequality in practical modern economies.   - The presence of phase transitions in growth vs shrinkage of a firm is not particularly exciting, as this essentially amounts to a certain coefficient being larger or smaller than 1, which naturally occurs at a transition point.  - The presence of oscillation patterns is to be expected, as the discrete nature of time coupled with the fact that an efficient cycle might contract at some points and expand at others leads directly to non-monotonicities.  One thing I would have liked to see is more comparison between the dynamics and the equilibrium (i.e., optimal) growth.  E.g., rate of convergence, and whether the set of growing vs vanishing firms differs between the dynamic and equilibrium outcomes in the long run.  A more complete analysis of equilibria in this particular variant of von Neumann's model would help to better appreciate the outcome of the learning dynamics.  Overall, I think the model being studied is natural, and the main result provides a crisp result for a natural set of local dynamics.  The fact that one very specific set of dynamics is studied does limit the significance of the work, but I believe there is a reasonable chance that this model could generate follow-up work.  On the other hand, the paper is padded out with a significant amount of extra content that I think detracts from the overall message, and the clarity of the paper suffers as a result.  The topic of "firms learning to bid in an economy" fits within the scope of NIPS, if perhaps tangentially.  Comments for the authors  Abstract, line 7: the comment about "unbounded inequality" makes it sound as though this is an outcome specific to the dynamics.  Would a social planner designing an optimal rate of growth not also generate unbounded inequality?  I'd clarify and/or rephrase both here and in the introduction. Page 3, line 96: Should we take these oscillation patterns as evidence that the model is not predictive?  Or should we expect these outcomes in practice?  Consider commenting on this here or elsewhere. Page 7, line 270: please comment on the strength of this assumption of a unique best cycle Page 7, line 274: broken reference to an Appendix  Response to Author Comments:  The authors make a good point about adversarial initial states, and the challenge that they pose. I still feel that many of the "extra" results are unsurprising at a qualitative/conceptual level, but I certainly don't mean to imply that they are obvious or trivial on a technical level.  I still feel very positively about the "main" (in my opinion) contribution, which is the convergence and optimal growth properties of the studied dynamics.


Reinforced Continual Learning
Most artificial intelligence models are limited in their ability to solve new tasks faster, without forgetting previously acquired knowledge. The recently emerging paradigm of continual learning aims to solve this issue, in which the model learns various tasks in a sequential fashion. In this work, a novel approach for continual learning is proposed, which searches for the best neural architecture for each coming task via sophisticatedly designed reinforcement learning strategies. We name it as Reinforced Continual Learning. Our method not only has good performance on preventing catastrophic forgetting but also fits new tasks well. The experiments on sequential classification tasks for variants of MNIST and CIFAR-100 datasets demonstrate that the proposed approach outperforms existing continual learning alternatives for deep networks.

This paper presents a simple but effective way to dynamically grow a neural network in a continual learning setup. The framework uses reinforcement learning to learn a policy that outputs decisions in terms of number of nodes/filters to add, and those additional filters are trained specifically for the new task (while the previous layers are held constant to prevent reduction in performance for old tasks). This is progressively done layer by layer where previous decisions for earlier layers serve as input to an LSTM in order to make decisions for the subsequent layer. Results show similar performance to approaches such as DEM, but have some better results for CIFAR-100 especially and the method results in networks with lower complexity.   Strengths  - Good overview of two groups of approaches on this topic   - The idea of using RL for continual learning is a good one, and one that makes sense given recent advancements in RL for architecture search.   - The results seem positive, especially for CIFAR-100 and for lowering the overall complexity of the solutions.    - The method is simple, which means that it should be easy to implement, barring issues in RL training stability. Releasing source code would be very helpful in this regard for reproducing the results in the paper.  Weaknesses  - An argument against DEN, a competitor, is hyper-parameter sensitivity. First, this isn't really shown, but second (and more importantly) reinforcement learning is well-known to be extremely unstable and require a great deal of tuning. For example, even random seed changes are known to change the behavior of the same algorithm, and different implementation of the same algorithm can get very different results (this has been heavily discussed in the community; see keynote ICLR talk by Joelle Pineau as an example). This is not to say the proposed method doesn't have an advantage, but the argument that other methods require more tuning is not shown or consistent with known characteristics of RL.    * Related to this, I am not sure I understand experiments for Figure 3. The authors say they vary the hyper-parameters but then show results with respect to # of parameters. Is that # of parameters of the final models at each timestep? Isn't that just varying one hyperparameter? I am not sure how this shows that RCL is more stable.   - Newer approaches such as FearNet [1] should be compared to, as they demonstrated significant improvement in performance (although they did not compare to all of the methods compared to here).  [1] FearNet: Brain-Inspired Model for Incremental Learning, Ronald Kemker, Christopher Kanan, ICLR 2018.   - There is a deeper tie to meta-learning, which has several approaches as well. While these works don't target continual learning directly, they should be cited and the authors should try to distinguish those approaches. The work on RL for architecture search and/or as optimizers for learning (which are already cited) should be more heavily linked to this work, as it seems to directly follow as an application to continual learning.   - It seems to me that continuously adding capacity while not fine-tuning the underlying features (which training of task 1 will determine) is extremely limiting. If the task is too different and the underlying feature space in the early layers are not appropriate to new tasks, then the method will never be able to overcome the performance gap. Perhaps the authors can comment on this.  - Please review the language in the paper and fix typos/grammatical issues; a few examples:   * [1] "have limitation to solve" => "are limited in their ability to solve"   * [18] "In deep learning community" => "In THE deep learning community"   * [24] "incrementally matche" => "incrementally MATCH"   * [118] "we have already known" => "we already know"   * and so on  Some more specific comments/questions:   - This sentence is confusing [93-95] "After we have trained the model for task t, we memorize each newly added filter by the shape of every layer to prevent the caused semantic drift." I believe I understood it after re-reading it and the subsequent sentences but it is not immediately obvious what is meant.    - [218] Please use more objective terms than remarkable: "and remarkable accuracy improvement with same size of networks". Looking at the axes, which are rather squished, the improvement is definitely there but it would be difficult to characterize it as remarkable.    - The symbols in the graphs across the conditions/algorithms is sometimes hard to distinguish (e.g. + vs *). Please make the graphs more readable in that regard.  Overall, the idea of using reinforcement learning for continual learning is an interesting one, and one that makes sense considering recent advances in architecture search using RL. However, this paper could be strengthened by 1) Strengthening the analysis in terms of the claims made, especially with respect to not requiring as much hyper-parameter tuning, which requires more evidence given that RL often does require significant tuning, and 2) comparison to more recent methods and demonstration of more challenging continual learning setups where tasks can differ more widely. It would be good to have more in-depth analysis of the trade-offs between three approaches (regularization of large-capacity networks, growing networks, and meta-learning).   ==============================================  Update after rebuttal: Thank you for the rebuttal. However, there wasn't much new information in the rebuttal to change the overall conclusions. In terms of hyper-parameters, there are actually more hyper-parameters for reinforcement learning that you are not mentioning (gamma, learning rate, etc.) which your algorithm might still be sensitive to. You cannot consider only the hyper-parameter related to the continual learning part. Given this and the other limitations mentioned, overall this paper is marginally above acceptance so the score has been kept the same.


A 2 -Nets: Double Attention Networks
Learning to capture long-range relations is fundamental to image/video recognition. Existing CNN models generally rely on increasing depth to model such relations which is highly inefficient. In this work, we propose the "double attention block", a novel component that aggregates and propagates informative global features from the entire spatio-temporal space of input images/videos, enabling subsequent convolution layers to access features from the entire space efficiently. The component is designed with a double attention mechanism in two steps, where the first step gathers features from the entire space into a compact set through second-order attention pooling and the second step adaptively selects and distributes features to each location via another attention. The proposed double attention block is easy to adopt and can be plugged into existing deep neural networks conveniently. We conduct extensive ablation studies and experiments on both image and video recognition tasks for evaluating its performance. On the image recognition task, a ResNet-50 equipped with our double attention blocks outperforms a much larger ResNet-152 architecture on ImageNet-1k dataset with over 40% less the number of parameters and less FLOPs. On the action recognition task, our proposed model achieves the state-of-the-art results on the Kinetics and UCF-101 datasets with significantly higher efficiency than recent works.

This paper proposes the “double attention block” to aggregate and propagate informative global features from the entire spatio/spatio-temporal space of input. Specifically, the model first generates a set of attention distributions over the input and obtains a set of global feature vectors based on the attention. Then, for each input position, it generates another attention distribution over the set of global feature vectors and uses this to aggregate those global feature vectors into a position-specific feature vector. The proposed component can be easily plugged into existing architectures. Experiments on image recognition (ImageNet-1k) and video classification (Kinetics, UCF-101) show that the proposed model outperforms the baselines and is more efficient.  Strength 1.    The proposed module is efficient and easy to adapt to modern frameworks. 2.    Extensive empirical experiments show that the proposed method outperforms consistently outperform other baselines.  Weakness 1.    I am concerned about the originality of the proposed model. The idea of applying an attention over the global feature vectors is conceptually similar to the Transformer architecture (Vaswani et al.). 2.    Lack of theoretical analysis of the proposed model. 3.    The improvement over the baselines is not significant. For example, in the ablation studies (Table 2 and 3), the relative error reduction over “Non-local neural networks” is less than 2%.


Stacked Semantics-Guided Attention Model for Fine-Grained Zero-Shot Learning
Zero-Shot Learning (ZSL) is generally achieved via aligning the semantic relationships between the visual features and the corresponding class semantic descriptions. However, using the global features to represent fine-grained images may lead to sub-optimal results since they neglect the discriminative differences of local regions. Besides, different regions contain distinct discriminative information. The important regions should contribute more to the prediction. To this end, we propose a novel stacked semantics-guided attention (S 2 GA) model to obtain semantic relevant features by using individual class semantic features to progressively guide the visual features to generate an attention map for weighting the importance of different local regions. Feeding both the integrated visual features and the class semantic features into a multi-class classification architecture, the proposed framework can be trained end-to-end. Extensive experimental results on CUB and NABird datasets show that the proposed approach has a consistent improvement on both fine-grained zero-shot classification and retrieval tasks.

This paper aims at using attention on image regions to improve fine grained zero shot learning task. The proposed method is build up on existing zero shot learning frame work with a novel stacked attention branch, which matches global image and text representation and use it to weight regional features. The proposed method shows better performance in two bird datasets.  Strengths: - The idea to put attention into zero shot learning task, especially first regresses global image feature to textual representation, then use it to guide attention, is new.  - Proposed method shows better performance in two datasets by a significant margin, especially in zero-shot retrieval task.  Weaknesses:  - This paper misses a few details in model design and experiments: A major issue is the "GTA" / "DET" feature representation in Table 1. As stated in section 4.1, image regions are extracted from ground-truth / detection methods. But what is the feature extractor used on top of those image regions? Comparing resnet / densenet extracted features with vgg / googlenet feature is not fair.  - The presentation of this paper can be further improved. E.g. paragraph 2 in intro section is a bit verbose. Also breaking down overly-long sentences into shorter but concise ones will improve fluency.  Some additional comments: - Figure 3: class semantic feature should be labeled as "s" instead of "c"?  - equation 1: how v_G is fused from V_I? please specify.  - equation 5: s is coming from textual representations (attribute / word to vec / PCA'ed TFIDF). It might have positive / negative values? However the first term h(W_{G,S}, v_G) is post ReLU and can only be non-negative?  - line 157: the refined region vector is basically u_i = (1 + attention_weight)  * v_i. since attention weight is in [0, 1] and sums up to 1 for all image regions. this refined vector would only scales most important regions by a factor of two before global pooling? Would having a scaling variable before attention weight help?  -  line 170: class semantic information is [not directly] embedded into the network?   - Equation 11: v_s and u_G are both outputs from trained-network, and they are not normalized? So minimize L-2 loss could be simply reducing the magnitude of both vectors?  - Line 201: the dimensionality of each region is 512: using which feature extractor?   - Section 4.2.2: comparing number of attention layers is a good experiment. Another baseline could be not using Loss_G? So attention is only guided by global feature vector.   - Table 4: what are the visual / textual representations used in each method? otherwise it is unclear whether the end-to-end performance gain is due to proposed attention model.


Convergence of Cubic Regularization for Nonconvex Optimization under KŁ Property
Cubic-regularized Newton's method (CR) is a popular algorithm that guarantees to produce a second-order stationary solution for solving nonconvex optimization problems. However, existing understandings of the convergence rate of CR are conditioned on special types of geometrical properties of the objective function. In this paper, we explore the asymptotic convergence rate of CR by exploiting the ubiquitous Kurdyka-Łojasiewicz (KŁ) property of nonconvex objective functions. In specific, we characterize the asymptotic convergence rate of various types of optimality measures for CR including function value gap, variable distance gap, gradient norm and least eigenvalue of the Hessian matrix. Our results fully characterize the diverse convergence behaviors of these optimality measures in the full parameter regime of the KŁ property. Moreover, we show that the obtained asymptotic convergence rates of CR are order-wise faster than those of first-order gradient descent algorithms under the KŁ property.

The paper investigates the convergence of different measurement of cubic regularization method for non-convex optimization under KL property. It consists with a list of work on CR methods based on the analysis of Nesterove el.s' work. Since the type of methods can guarantee the convergence to the second-order stationary point, it is quite popular also considering the raising of training neural networks.   The paper is well-written, clear-organized and the theorems and proofs are easy to follow. Note that this is a pure theoretical work i.e., without new algorithms and/or numerical experiments. I have the following concerns  1. The results of the paper is based on KL property, and show convergence result in terms of the critical parameter \theta of KL. As a generalized function property, it is not easy to have \theta for a given function and the paper does not provide examples where this \theta can be values other than  \theta=1/2 for some machine learning problems.  2. To apply the algorithm, the Lipschitz-Hessian constant and an exact solution for subproblem is needed to derive the second-order stationary point convergence, which makes the algorithm unpractical for large problems. 3. There is no discussion over finite sum problems, which is typical in machine learning field, which I believe NIPS has more focus on such problems.  Considering the above, the paper is good but might not suitable for NIPS. I would therefore only vote the paper a weak accept but I'm also open to others suggestions.   minors: 1. \bar{x} in Definition 1 is not used and make the definition confusing.


Simple, Distributed, and Accelerated Probabilistic Programming
We describe a simple, low-level approach for embedding probabilistic programming in a deep learning ecosystem. In particular, we distill probabilistic programming down to a single abstraction-the random variable. Our lightweight implementation in TensorFlow enables numerous applications: a model-parallel variational auto-encoder (VAE) with 2nd-generation tensor processing units (TPUv2s); a data-parallel autoregressive model (Image Transformer) with TPUv2s; and multi-GPU No-U-Turn Sampler (NUTS). For both a state-of-the-art VAE on 64x64 ImageNet and Image Transformer on 256x256 CelebA-HQ, our approach achieves an optimal linear speedup from 1 to 256 TPUv2 chips. With NUTS, we see a 100x speedup on GPUs over Stan and 37x over PyMC3.
def model(): p = ed.Beta(1., 1., name="p") x = ed.Bernoulli(probs=p, sample_shape=50, name="x") return x Figure 2: Variational program [35], available in eager mode. Python control flow is applicable to generative processes: given a coin flip, the program generates from one of two neural nets. Their outputs can have differing shape (and structure).
system: this makes them difficult to compose with arbitrary computation, and with the broader machine learning ecosystem, such as production platforms [5].
In this paper, we describe a simple approach for embedding probabilistic programming in a deep learning ecosystem; our implementation is in TensorFlow and Python, named Edward2. This lightweight approach offers a low-level modality for flexible modeling-one which deep learners benefit from flexible prototyping with probabilistic primitives, and one which probabilistic modelers benefit from tighter integration with familiar numerical ecosystems.
Contributions. We distill the core of probabilistic programming down to a single abstraction-the random variable. Unlike existing languages, there is no abstraction for learning: algorithms may for example be functions taking a model as input (another function) and returning tensors.
This low-level design has two important implications. First, it enables research flexibility: a researcher has freedom to manipulate model computation for training and testing. Second, it enables bigger models using accelerators such as tensor processing units (TPUs) [22]: TPUs require specialized ops in order to distribute computation and memory across a physical network topology.
We illustrate three applications: a model-parallel variational auto-encoder (VAE) [24] with TPUs; a data-parallel autoregressive model (Image Transformer [31]) with TPUs; and multi-GPU No-U-Turn Sampler (NUTS) [21]. For both a state-of-the-art VAE on 64x64 ImageNet and Image Transformer on 256x256 CelebA-HQ, our approach achieves an optimal linear speedup from 1 to 256 TPUv2 chips. With NUTS, we see a 100x speedup on GPUs over Stan [8] and 37x over PyMC3 [39].

The paper describes Edward2, a probabilistic programming language on top of TensorFlow and supports the TPU, GPU and CPU accelerators. It is the successor of the Edward-1 package.  Edward-2's main USP is that it has a single abstraction (random variable), and offers no separate abstractions to describe learning itself. The learning procedures are left as functions that are user-defined, making Edward-2 low-level (and powerful -- claim).  The paper describes Edward2's tracing mechanism, which allows one to do complex inference procedures, doing interventions on outcomes, etc.  The paper does experiments on No-U-Turn sampler, comparing to other packages in the ecosystem. Additionally it does experiments on VAEs, also comparing to other packages in the ecosystem.  The results showcase that Edward's usage of GPUs (in NUTS) vs Stan and PyMC3 give it considerable speedups (attributing to the use of GPUs). Additionally for VAEs, they show that their use of TPUs give it a 5x speedup over using GPUs, attributing to the TPU being faster.  ------------------------------  The paper is very incomprehensible. I am familiar with TensorFlow and a little bit of probabilistic programs, but I could barely comprehend the paper. I read it 2 and a half times, and I was often short of context. I think the authors seem to have struggled with the page limits and compressed content + removed context. Sections such as 2.2 and 3, 3.1, 3.2 are clearly shortened beyond comprehension for a reader that is not familiar with Edward-1 or whatever context that the authors assume that the reader has. While I have a superficial "tl;dr" of such sections, the point that was being made in the section didn't actually come across. I have very little understanding of what makes Edward2 better. All the code samples are not discussed in sufficient detail either, often "assuming" that the reader "gets" the superiority of what is being presented.  While the project seems to be an impressive effort: "PPL with CPU, GPU, TPU support and considerable integration with popular framework such as TensorFlow", the paper should be expanded and probably sent to a venue or journal where one does not have to fight the page limits and can express themselves better. I dont have confidence that as a poster, the authors can do a good job of explaining the framework well. This is my primary reason for rejection of the paper, and I will let the Area Chair make a decision on whether this is a valid reasoning line or not.  Lastly, something that seems to be a dishonest "wording" is their sentences around speedups against other packages. Starting from the abstract, and continuing into many places in the paper (8, 44 Table-{1, 2, 3} captions, 235, 243, 9, Table-1 caption, 219, 244) they keep repeating "Edward2 on TPU vs Pyro on GPU" or "Edward2 on GPU vs STAN on CPU". This is not only disingenous, it's dangerous. If you see Table-2, it is clear that Edward2 on TPU vs Edward2 on GPU is a 5x speed gain. So the main speed gain is actually TPU vs GPU, yet they clearly try to mix up the language so that "Edward2" is associated with the speedup rather than "TPU". There's more problematic issues with the benchmarking of TPUs vs GPUs. They compare the TPU to a 1080Ti GPU. They are comparing the latest generation TPU hardware to an older generation GPU hardware. This is not acceptable, especially if they want to make their claim that TPUs are faster. If they compare a TPU against Volta GPUs, will their speedup reduce by a significant amount? Definitely, as Volta's fp32 is a 2x number of flops and has more memory bandwidth. This is my added reason for rejecting the paper in it's current form.   Nit: 55, Fig.4, 73 refer Edward2 as "Edward", making it confusing.


Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple interrelated objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structured prediction, which optimizes over complex labels, while modeling within-label interactions. However, it is unclear what principles should guide the design of a structured prediction model that utilizes the power of deep learning components. Here we propose a design principle for such architectures that follows from a natural requirement of permutation invariance. We prove a necessary and sufficient characterization for architectures that follow this invariance, and discuss its implication on model design. Finally, we show that the resulting model achieves new state-of-the-art results on the Visual Genome scene-graph labeling benchmark, outperforming all recent approaches.

This paper studies the property of permutation invariance in the context of structured prediction. The paper argues that in many applications permutation invariance is a desirable property of a solution and it makes sense to design the model such that it is satisfied by construction rather than to rely on learning to get this property. The paper proposes a model to represent permutation invariant functions and claims that this model is a universal approximator within this family. The proposed method is evaluated on a synthetic and a real task (labelling of scene graphs).  1) Most importantly, I think that in the current form the proof of the main theoretical result (Theorem 1) is wrong. The problem is with the reverse direction proving that any permutation invariant function can be represented in the form of Theorem 1. Specifically, Lines 142-159 construct matrix M which aggregates information about the graph edges. However, the actual function \rho is defined (line 160) without taking the matrix M into account. Instead, it looks at y, which is the output of the function. As is, the statement in fact says "define the function output with y, now set it equals to y", which is a logical error and does not prove anything.  2) I also think that the assumptions that z_k uniquely identifies the node (line 138) and that F is a function of only pairwise features (line 140) contradict each other, because if there are no unary features they cannot identify the nodes.  3) The form of the function F defined in Theorem 1 is very similar to one round of neural message passing [9] and thus will not allow to propagate information across the graph. For example, imaging the toy situation: one node has a very strong preference for one label through the unary potentials, the others have no preferences. The pairwise potentials defined a long chain of equalities saying that in the end of the day all the variables should take the same value. However, to infer this we need to propagate information across the whole graph, which will need many rounds.  4) The experimental evaluation is somewhat interesting, but does not have sufficient description to be reproduced and does not show significant improvements, thus I find it not sufficient for acceptance.  Minor: Line 147 probably should have the sum over j and not over pairs i, j   ==== after the response 1) I agree with the authors my point 1 was not valid. 2) I agree with the authors that this particular place has no contradiction. However, I'm still not quite buying the whole argument based on the perfect hash function. The theorem is proving a statement about continuous inputs, which means that for any hash function with a finite hash table there exist a pair of inputs z_i, z_j, for which there will be a collision. The construction will break for this input. I think we cannot use a different hash function for each input, but need it to defined before looking at the inputs, which looks like a fundamental flaw in the proof technique. 3) I'm not getting why the presented method cannot be viewed as a message passing algorithm. Functions phi can be viewed as messages and alpha as the message aggregation. The big difference is probably the fact that the messages are not passed w.r.t. the initial graph, but w.r.t. the complete graph on the nodes. 4) I still think that the description of the experiments does not allow reproducibility and the improvements are somewhat incremental (the response claims otherwise, but does not give any extra data, e.g. some upper bounds).  Having said that, I'm ready to raise my score to 5, because the theorem might be correct, but the paper still requires a major revision (proof of the theorem and the description of the experimental setup).


The Pessimistic Limits and Possibilities of Margin-based Losses in Semi-supervised Learning
Consider a classification problem where we have both labeled and unlabeled data available. We show that for linear classifiers defined by convex margin-based surrogate losses that are decreasing, it is impossible to construct any semi-supervised approach that is able to guarantee an improvement over the supervised classifier measured by this surrogate loss on the labeled and unlabeled data. For convex margin-based loss functions that also increase, we demonstrate safe improvements are possible.

Overview and Recommendation: Many popular binary classifiers are defined by convex margin-based surrogate losses such as SVMs and Logistic regression. Designing a semi-supervised learning algorithm for these classifiers, that is guaranteed to improve upon the "lazy" approach of throwing away the unlabeled data and just using the labeled data while training, is of considerable interest, because of the time-consuming experimentation that the use of SSL currently requires. This paper analyzes this problem and the results presented in the paper are primarily of theoretical interest.  I had great difficulty in rating the significance of this work, therefore my own confidence rating is only 3. The proofs of the theorems use elementary steps. I checked them in detail and they are correct, but, the significance of the theorems themselves was hard to measure. There is definitely some importance for theorem 3 as a good counter-example for common intuition and previous conjectures, therefore I have given the paper a rating of 6.   Summary: The paper gives a new definition of a  "safe" semi-supervised binary classifier as a classifier that is guaranteed to attain a lower loss on the combined labeled-unlabeled dataset  than the loss achived by a supervised classifier learnt from just the labeled data. Under this definition of safety, theorems 1 and 2 in the paper prove that a safe semi-supervised classifier/hypothesis does not exist for  monotonically decreasing convex margin-based, e.g. SVMs, and therefore safe SSL is impossible for such classifiers. Next, a more surprising result is shown in theorem 3, that it is possible to do safe SSL for some special convex surrogate loss functions that are not monotonically decreasing, e.g. the quadratic loss.   Pros: - This gives a very general proof for the common intuition that it is not possible to do semi-supervised learning for discriminative classifiers without making assumptions about the joint distribution of inputs and labels. - One important strength of our analysis is that we also consider the minimization of loss functions that may not induce a correct probability.  - There are no assumptions on the input features so the inputs can be the features extracted from the last layer of a neural network and the results will continue to hold.  Cons: -  In Theorems 1 and 2, the paper shows that "safe" semi-supervised learning, under an extremely stringent definition of safety, is impossible to perform for "reasonable" convex surrogate loss functions such as hinge loss and logistic loss, which seems unsurprising. And on the other side the paper proves that "safe" semi-supervised learning is possible for some "unreasonable" classification surrogate loss functions such as the squared-loss. My main concern is that it is not clear how useful "safe" semi-supervised learning on squared surrogate loss function is since it is not a common surrogate loss anyway. The authors do anticipate this objection in line 304-307 but they really need to give many more examples for their claim that "surrogate losses like the quadratic loss generally give decent performance".  - The definition of safety in this work is very stringent and fairly non-standard. For example, (Sakai 2017) present estimation error bounds for PNU learning and they present sufficient condition when learning from positive-negative and unlabeled data is likely to outperform supervised learning with high probability. The authors should give more motivation for why their definition of safety is preferable compared to Sakai et al.  [Sakai 2017] "Semi-Supervised Classification Based on Classification from Positive and Unlabeled Data" by Sakai et al. ICML, (2017)  === After Author Response ===   Based on the additional references for the use of squared loss provided by authors in their response, I feel better about the use of squared error loss. I will encourage the authors to make space for discussing these references, even if briefly, in the main body of the paper since it can make the audience more accepting of the ideas in the paper. Note that the other reviewers had similar doubt about the usefulness of the results.


Approximation algorithms for stochastic clustering *
We consider stochastic settings for clustering, and develop provably-good (approximation) algorithms for a number of these notions. These algorithms allow one to obtain better approximation ratios compared to the usual deterministic clustering setting. Additionally, they offer a number of advantages including providing fairer clustering and clustering which has better long-term behavior for each user. In particular, they ensure that every user is guaranteed to get good service (on average). We also complement some of these with impossibility results.

SUMMARY: The paper studies stochastic versions of classical center-based clustering procedures (e.g. k-center, k-supplier, k-means etc). Such algorithms end up searching for k centers that minimize some objective depending on R_y for each datapoint y where R_y is the distance from y to the closest of the k centers. The stochastic modification allows a distribution over possible clusterings (i.e. distribution over sets of k centers) thus giving a randomized solution from some optimized distribution rather than a fixed solution.  The approximation guarantees for the classical clustering procedures are typically stated in terms of the worst-case sub-optimality amongst the R_y's. For example, for k-centers, we get for R_y <= 2 T for all y where T is the optimal solution. This paper gives stochastic clustering procedures which simulatenously attain these classical guarantees but also gives a per-point guarantee in expectation which are tighter (i.e. E[R_y] <= 1.592 T for stochastic k-centers).  This per-point guarantee forms the basis for the notion of fairness as the idea is to guarantee that in expectation, each datapoint's distance to its closest center is bounded while simulatenously with probability 1, the original (but weaker) bound still holds.  The paper then presents the following results: 1. Chance k-covering guarantee: given demand parameters p_y, r_y, the goal is to find a clustering that satisfies P[R_y <= r_y] >= p_y. The paper proposes a procedure which attains P[R_y <= r_y] >= (1 - 1/e) p_y.  2. The paper then gives a iterative randomized rounding based procedure that attains a guarantee of P[R_y <= 9 r_y] >= p_y 3. Then the paper shows that if p_y are all equal or r_y are all equal, then we obtain better approximation guarantees.   4. Then the paper tackles a special case of the k-covering problem which is essentially better known as k-centers or k-supplier depending on the setting.  They provide guarantees of form E[R_y] <= c * r_y where c depends on the setting. The paper address the general case, the SCC setting (where the cluster centers is a subset of the datapoints), and the homogenous SCC setting (where it is SCC and the r_y's are all equal). 5. The paper then provides lower bounds to complement the previous guarantees. 6. The paper then gives approximation algorithms for finding distributions of clusterings that aim at satisfying conditions of form E[R_y] <= t_y where t_y is any feasible vector. They then give a procedure with approximation guarantees for this problem with sparsity guarantees using a multiplicative-weights based procedure and then a shrinking procedure which reduces it down to a linear size.  7. At last, the paper studies whether a stochastic clustering can be used to obtain a deterministic clustering with some positive results and lower bounds.   COMMENTS: 1. I am not entirely convinced of the fairness angle of this paper. While it is true that we get tighter per-datapoint guarantees, we already had some (weaker) per-datapoint guarantees. In my opinion, this result should be presented as such: an improved approximation guarantee by allowing stochasticity rather than a fairness guarantee. My impression of standard notions of fairness are based on so-called protected groups where we give guarantees of equality/parity amongst the protected groups. Here, we are giving guarantees about each individual. I guess there are also individual notions of fairness but here, each individual is treated the same regardless of feature values.  2. The iterative-randomized rounding technique is interesting, has such a  similar technique been observed before?   OVERALL: To my knowledge, this paper provides many new ideas and contributions and is a considerable step forward in our understanding of approximation guarantees for these classical clustering procedures. This is very exciting work; well done!


Neural Interaction Transparency (NIT): Disentangling Learned Interactions for Improved Interpretability
Neural networks are known to model statistical interactions, but they entangle the interactions at intermediate hidden layers for shared representation learning. We propose a framework, Neural Interaction Transparency (NIT), that disentangles the shared learning across different interactions to obtain their intrinsic lower-order and interpretable structure. This is done through a novel regularizer that directly penalizes interaction order. We show that disentangling interactions reduces a feedforward neural network to a generalized additive model with interactions, which can lead to transparent models that perform comparably to the state-of-theart models. NIT is also flexible and efficient; it can learn generalized additive models with maximum K-order interactions by training only O(1) models.

The paper proposes a variation of feedforward neural networks that does not learn (potentially unwanted) non-additive interactions between variables above a certain order.  The variation includes a block-based architecture with a sparsifying layer and a regularizer that caps the interaction order within blocks.  If the max interaction order is <= 2, then all interactions can be easily visualized as independent plots.  The authors draw an analogy between this model and GAMs with interactions.  Pros:  - The paper is relevant to NIPS and reasonably well written.  - Combining blocking and soft-capping the interaction order is an interesting idea and a natural extension to GAMs.  - Some results are nice: the proposed model can reach the same performance but better visualizability than the competitors.  Cons:  - The definition of interactions in this paper is not equivalent to the one given    in [28], which affects both the main message and the results.  It is also    confusing.  - Evaluation is lacking, see below.   Major remarks:   - The paper focuses on non-additive interactions.  However, definition 1 captures additive interactions as well.     Consider for instance the function f(x) = x1 + x2 + x3 + x4, which has no non-additive interactions at all.  Also let all variables be non-negative.  One way to write f(x) as a feedforward is:        f(x) = 1*h1(x) + 1*h2(x) + 0       h1(x) = max{1*x1 + 1*x2 + 0, 0}       h2(x) = max{1*x3 + 1*x4 + 0, 0}     Although this network has no non-additive interactions at all, definition 1 would classify all variables as entangled.     In other words, the presence of a path of positive weights does not imply that there is a non-additive interaction.  That is, the definition is incorrect.  Indeed, the authors do not show that it is equivalent to the actual definition of non-additive interactions given in [28].     Now, both the regularizer and the results depend on this definition.  This implies that:     - the regularizer controls a *surrogate* (actually, an upper bound) of the # of non-additive interactions.     - the # of interactions reported in the results may overestimate the real # of non-additive interactions.     To summarize, unless I'm missing something, the method does not do what the authors claim it does, and the same goes for the experiments.  The appropriate claims should be toned down accordingly.    - In order to see if DI really works, two things should be checked: (i) the first sparsifying layer works, (ii) the cap on maximum interaction order works.     Point (ii) has been validated empirically with real-world datasets.     The same is not true for point (i). The synthetic dataset is extremely simple (it is sampled from an order 2 homogeneous polynomial).     This is not enough to show that the sparsifying layer, especially when coupled with the cap on the interaction order, works as intended.  It should be easy to show that it does by testing it on *several* of order K polynomials (with varying K), and checking that the first layer learns the intended independencies.     The experiments with realistic datasets do not help in this case, because the gold standard is unknown.   - I also have a few other (neither major nor minor) questions about the experiments:    - The output layer of the proposed method is linear.  Is this also the case for binary classification tasks?    - I am a bit puzzled that there is no comparison to kernel methods (where the interaction order can be capped very easily, e.g. with polynomial kernels). Why were they left out of the comparison?    - It seems that training these models may be possibly complicated, potentially hindering reproducibility.  Will the code/data be made available?    - What about run-time comparisons?   Minor remarks:   - Figure 1 is very confusing.  I suggest to use colors to identify different models/regularizers and linestyles for the different layers.  The legend should also be sorted accordingly.   - l 110: a function the in the form   - l 152: to learn such form of model -> to learn such a model   - l 194: "since neural networks are known to learn smooth functions".  This may be the case in general, but it is definitely *not* the case for the ReLU networks used in this paper.  It is well known that ReLU networks with linear outputs learn piecewise linear functions (see [Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution, Chu et al., 2018] for a nice visualization), which are definitely not smooth.  The learned function may rensamble a smooth function, but this is only the case if the target function itself is smooth.  So the claim that the proposed method encourages smooth plots should be either removed or qualified appropriately.   - l 217: set to a small value [and] then increased   - l 216: "if K was not given ... model".  This is very confusing.  Please just    say that you report the performance of models while varying K.  (Otherwise it    sounds like a single value of K was chosen unfairly.)  ==== After reading the rebuttal, I decided to increase my score.  The issues I found can be either solved by rewording the paper (namely clarifying that the method uses a surrogate upper-bound of the # of non-additive independencies), or are not too major (they concern "only" the first layer).